Switching stations paper in press.pdf (843.32 kB)
Does changing examiner stations during UK postgraduate surgery objective structured clinical examinations influence examination reliability and candidates' scores?
journal contribution
posted on 2023-06-09, 00:56 authored by Peter A Brennan, David T Croke, Malcolm ReedMalcolm Reed, Lee Smith, Euan Munro, John Foulkes, Richard ArnettObjective Objective structured clinical examinations (OSCE) are widely used for summative assessment in surgery. Despite standardizing these as much as possible, variation, including examiner scoring, can occur which may affect reliability. In study of a high-stakes UK postgraduate surgical OSCE, we investigated whether examiners changing stations once during a long examining day affected marking, reliability, and overall candidates’ scores compared with examiners who examined the same scenario all day. Design, Setting, and Participants An observational study of 18,262 examiner-candidate interactions from the UK Membership of the Royal College of Surgeons examination was carried at 3 Surgical Colleges across the United Kingdom. Scores between examiners were compared using analysis of variance. Examination reliability was assessed with Cronbach’s alpha, and the comparative distribution of total candidates’ scores for each day was evaluated using t-tests of unit-weighted z scores. Results A significant difference was found in absolute scores differences awarded in the morning and afternoon sessions between examiners who changed stations at lunchtime and those who did not (p < 0.001). No significant differences were found for the main effects of either broad content area (p = 0.290) or station content area (p = 0.450). The reliability of each day was not affected by examiner switching (p = 0.280). Overall, no difference was found in z-score distribution of total candidate scores and categories of examiner switching. Conclusions This large study has found that although the range of marks awarded varied when examiners change OSCE stations, examination reliability and the likely candidate outcome were not affected. These results may have implications for examination design and examiner experience in surgical OSCEs and beyond.
History
Publication status
- Published
File Version
- Accepted version
Journal
Journal of Surgical EducationISSN
1931-7204Publisher
ElsevierExternal DOI
Issue
4Volume
73Page range
616-623Department affiliated with
- BSMS Publications
Full text available
- Yes
Peer reviewed?
- Yes
Legacy Posted Date
2016-04-22First Open Access (FOA) Date
2017-07-01First Compliant Deposit (FCD) Date
2016-04-22Usage metrics
Categories
No categories selectedKeywords
Licence
Exports
RefWorks
BibTeX
Ref. manager
Endnote
DataCite
NLM
DC