University of Sussex
Browse
Switching stations paper in press.pdf (843.32 kB)

Does changing examiner stations during UK postgraduate surgery objective structured clinical examinations influence examination reliability and candidates' scores?

Download (843.32 kB)
journal contribution
posted on 2023-06-09, 00:56 authored by Peter A Brennan, David T Croke, Malcolm ReedMalcolm Reed, Lee Smith, Euan Munro, John Foulkes, Richard Arnett
Objective Objective structured clinical examinations (OSCE) are widely used for summative assessment in surgery. Despite standardizing these as much as possible, variation, including examiner scoring, can occur which may affect reliability. In study of a high-stakes UK postgraduate surgical OSCE, we investigated whether examiners changing stations once during a long examining day affected marking, reliability, and overall candidates’ scores compared with examiners who examined the same scenario all day. Design, Setting, and Participants An observational study of 18,262 examiner-candidate interactions from the UK Membership of the Royal College of Surgeons examination was carried at 3 Surgical Colleges across the United Kingdom. Scores between examiners were compared using analysis of variance. Examination reliability was assessed with Cronbach’s alpha, and the comparative distribution of total candidates’ scores for each day was evaluated using t-tests of unit-weighted z scores. Results A significant difference was found in absolute scores differences awarded in the morning and afternoon sessions between examiners who changed stations at lunchtime and those who did not (p < 0.001). No significant differences were found for the main effects of either broad content area (p = 0.290) or station content area (p = 0.450). The reliability of each day was not affected by examiner switching (p = 0.280). Overall, no difference was found in z-score distribution of total candidate scores and categories of examiner switching. Conclusions This large study has found that although the range of marks awarded varied when examiners change OSCE stations, examination reliability and the likely candidate outcome were not affected. These results may have implications for examination design and examiner experience in surgical OSCEs and beyond.

History

Publication status

  • Published

File Version

  • Accepted version

Journal

Journal of Surgical Education

ISSN

1931-7204

Publisher

Elsevier

Issue

4

Volume

73

Page range

616-623

Department affiliated with

  • BSMS Publications

Full text available

  • Yes

Peer reviewed?

  • Yes

Legacy Posted Date

2016-04-22

First Open Access (FOA) Date

2017-07-01

First Compliant Deposit (FCD) Date

2016-04-22

Usage metrics

    University of Sussex (Publications)

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC