Article Text

Download PDFPDF

An evaluation of demographic factors affecting performance in a paediatric membership multiple-choice examination
  1. Lara Menzies1,2,
  2. Susan Minson2,
  3. Alexandra Brightwell2,
  4. Anna Davies-Muir3,
  5. Andrew Long2,3,
  6. Caroline Fertleman2,4
  1. 1Institute of Cognitive Neuroscience, University College London, London, UK
  2. 2London Specialty School of Paediatrics, London, UK
  3. 3Royal College of Paediatrics and Child Health, London, UK
  4. 4Paediatric Department, Whittington Hospital, London, UK
  1. Correspondence to Dr Lara Menzies, Institute of Cognitive Neuroscience, University College London, Alexandra House, 17 Queen Square, London WC1N 3AR, UK; lacmenzies{at}gmail.com

Abstract

Objective To determine if demographic factors are associated with outcome in a multiple-choice, electronically marked paediatric postgraduate examination.

Method Retrospective analysis of pass rates of UK trainees sitting Membership of the Royal College of Paediatrics and Child Health (MRCPCH) part 1B from 2007 to 2011. Data collected by the RCPCH from examination candidates were analysed to assess the effects of gender, age, and country and university of medical qualification on examination outcome.

Results At first attempt at MRCPCH part 1B, the overall pass rate from 2007 to 2011 was 843/2056 (41.0%). In univariate analysis, passing the examination was associated with being a UK graduate (649/1376 (47.2%)) compared with being an international medical graduate (130/520 (25.0%)) (OR 2.68 (95% CI 2.14 to 3.36), p<0.001). There was strong evidence that the proportion of candidates passing the examination differed for graduates of the 19 different UK medical schools (Fisher's exact test p<0.001). In multivariate logistic regression analysis, after adjustment for age, sex and whether the part 1A examination was taken concurrently, being a UK graduate was still strongly associated with passing the examination (OR 3.17 (95% CI 2.41 to 4.17), p<0.001). UK graduates performed best at 26–27 years of age (52.4% pass rate), whereas overseas graduates performed best at ≥38 years of age (50.8% pass rate).

Conclusions MRCPCH part 1B outcome was related to place of primary medical qualification, with a significantly lower pass rate for international medical graduates compared with UK graduates, as well as significant variation in examination outcome between graduates from different UK medical schools. These data may be used to guide new initiatives to improve support and education for these trainees and to inform development of undergraduate curricula and help trainees prepare more successfully for postgraduate examinations.

  • MEDICAL EDUCATION & TRAINING
  • PAEDIATRICS
View Full Text

Statistics from Altmetric.com

Introduction

Postgraduate examinations are a significant hurdle for trainees across medical specialties and have been identified as a significant cause of stress.1 Examinations are, however, essential to ensure that professional clinical standards are maintained; they also play a key role in maintaining public confidence in doctors’ abilities. In the UK, paediatric trainees must pass four examinations to achieve Membership of the Royal College of Paediatrics and Child Health (MRCPCH) before progression to registrar level (ST4). The structure of the examination is described in detail by the RCPCH.2 Before 2013, the examination involved three parts: parts 1A and 1B, part 2, and the clinical examination. Parts 1A, 1B and 2 are all written examinations which are remotely assessed and electronically marked using ‘single best answer’ and ‘extended matching’ multiple-choice questions. The clinical examination is a face-to-face patient-based assessment.

We chose to study the MRCPCH part 1B examination, as it has been the most challenging of the written papers, with a pass rate of 31–46%.3 The MRCPCH part 1B paper pass mark is calculated using criterion referencing, a method commonly used in education to assess learning outcomes.4 In this method, every candidate is judged against a previously agreed reference standard (there is no predetermined pass rate or pass mark). Since 2013, the examination has been updated and is now called the MRCPCH theory and science (TAS) examination, which has a modified syllabus but is an otherwise similar assessment.5 The RCPCH examinations can be undertaken in many countries, but in this study we analysed examination outcome for trainees sitting MRCPCH at UK centres only.

UK paediatric trainees have a limited time in which to successfully complete their examinations. In order to progress from level 1 (senior house officer) to level 2 (registrar) training, trainees must complete all four parts of the membership examination. Repeated failure delays progression and in some cases results in the trainee losing their training number. Examination failure may also contribute to voluntary loss of trainees from the specialty by reducing confidence and job satisfaction. Attrition of junior doctors and challenges around workforce planning are both currently national concerns in the UK.6 ,7

Data from postgraduate medical examinations within other specialties reveal that examination outcomes vary for different demographic groups, such as country of primary qualification, university of primary qualification, gender and candidate age.8–16 In response to concerns about pass rates in the clinical skills assessment in general practice in the UK, the General Medical Council have recommended that demographic data on postgraduate examinations should be readily and transparently available to trainees.17–19 Concerns have also been raised that the variation that exists in UK medical school undergraduate curricula may also impact on postgraduate performance of doctors.20

Demographic data predicting outcome in membership examinations have been previously analysed for obstetrics and gynaecology,8 ,21 general medicine,10 ,11 ,12 ,22 ,23 general practice,15 ,16 psychiatry13 and anaesthetics,14 but not previously for paediatrics. UK graduates have been shown to perform consistently better than international medical graduates (IMGs) in general practice (MRCGP), general medical (MRCP) and psychiatry (MRCPsych) postgraduate examinations. The medical school attended by UK graduates has also been found to be strongly associated with performance differences for MRCP12 ,15 and in obstetrics and gynaecology (MRCOG)8 and anaesthetics (FRCA).14 The effect of gender is less consistent: for the MRCOG examination, there is some evidence to support higher pass rates for women than men,8 and in the MRCP examination, women performed better than men in the clinical face-to-face part of the examination (PACES) but not in the written examinations.11

The aim of this study was to determine if there was a link between MRCPCH part 1B examination outcome and demographic factors including gender, age, ethnicity, and university and country of primary qualification.

Methods

Candidate data collection

We obtained anonymised retrospective data for candidates sitting the MRCPCH part 1B examination from 2007 to 2011 at UK centres from the RCPCH database. We collected the following details, which are collected routinely by the RCPCH for all examination candidates: gender, country and medical school of primary qualification, age and ethnicity (categories based on those used by UK Office for National Statistics24). Only candidates’ first attempts at the examination were included, to prevent bias from including large numbers of unsuccessful attempts by single candidates.

Data analysis

Data were analysed using SPSS V.20. The aim of the analysis strategy was to identify whether demographic factors were related to examination outcome at first attempt at MRCPCH part 1B. We assessed whether the following were related to examination outcome: (i) place of medical qualification (UK graduate vs IMG); (ii) the medical school attended for UK graduates; (iii) gender; (iv) age at time of sitting the examination; (v) whether part 1A (the first written paper of MRCPCH) was concurrently undertaken at the same sitting.

Statistical methods

χ2 tests were used to determine whether there were significant differences between the proportions of IMGs and UK graduates and between men and women passing MRCPCH part 1B at first attempt. In addition, ORs for passing at first attempt and their associated 95% CIs were calculated for both variables. Secondly, for candidates graduating in the UK and who provided information on the medical school they attended, a χ2 test was applied to determine if the proportion of candidates passing MRCPCH part 1B differed significantly between medical schools. Pearson χ2 statistics were calculated unless otherwise stated. For analyses with small expected counts of <5, Fisher’s exact test was performed, using a Monte Carlo estimate when the size of the contingency table exceeded 2×2, such as in the analysis comparing UK medical schools. Thirdly, logistic regression was used to assess the contribution of the following variables to outcome at first attempt at part 1B: gender, place of primary medical qualification, and undertaking part 1B separately versus together with part 1A. Age was also incorporated into the regression model as a continuous variable. This methodological approach was chosen based on previously published methods.11

Results

Sample overview

Data for 2056 candidates were available for analysis (1491 women, 565 men); 1376 candidates were UK graduates and 520 were IMGs graduating overseas (report rate of 92%). All candidates declared their age when sitting the examination; mean age was 29 years (range 22–58). At first attempt at MRPCH part 1B, the overall pass rate from 2007 to 2011 was 41.0% (843/2056). Only a minority of candidates provided their ethnic group: 610 of 2056 candidates in total and 419 of 1376 UK graduates, which is a report rate of just 29.7% and 30.5%, respectively. For our sample, 386 candidates described themselves as white and 224 as black and ethnic minority candidates; the remaining 1446 candidates did not provide ethnicity data. Owing to this very low reporting rate for ethnicity, this variable is not considered further in the analyses below.

Place of primary qualification

UK graduates had a higher pass rate (649/1376 (47.2%)) than IMGs (130/520 (25.0%)) (OR 2.68 (95% CI 2.14 to 3.36), Pearson χ2 statistic 76.6 on one degree of freedom, p<0.001).

UK medical schools

Most UK graduates (1321/1376 (96%)) reported the medical school from which they obtained their medical degree. The pass rates for the 19 medical schools are shown in table 1. Graduates from London medical schools receive their qualification from the University of London, and we were therefore not able to assign these graduates to specific schools within this general category. There was strong evidence of a difference in pass rate by medical school (Fisher’s exact test p<0.001).

Table 1

Examination outcome for candidates attending different UK medical schools

Gender

We compared pass rates between male and female candidates across the whole dataset and also performed subset analyses between men and women within UK graduates only. Gender had no effect on performance at MRCPCH part 1B. Overall, 611/1491 (41.0%) women and 232/565 (41.1%) men passed the examination (OR women vs men 1.0 (95% CI 0.82 to 1.21), Pearson χ2 statistic 0.001 on one degree of freedom, p=0.97). Among UK graduates, only 485/1025 (47.3%) women and 164/351 (46.7%) men passed (OR 1.02 (95% CI 0.80 to 1.31), Pearson χ2 statistic 0.04, p=0.85).

Age and sitting part 1A concurrently

Figure 1 shows the detailed breakdown of pass rates by age for UK and international graduates. UK graduates had the highest pass rate between the ages of 26 and 27 years, equivalent to the first or second year of specialist training assuming a standard progression through UK postgraduate training. International graduates had the highest pass rate when aged 38 years an older. A formal assessment of the association, assuming that the relationship between age and pass rates was linear and that there was no interaction between being a UK or international graduate, gave a univariate OR for association for passing per year older of 0.96 (95% CI 0.94 to 0.99), p=0.002.

Figure 1

Relationship between age and examination outcome at first attempt at Membership of the Royal College of Paediatrics and Child Health (MRCPCH) part 1B for UK and international medical graduates. Proportions adjacent to each bar indicate number of candidates passing MRCPCH part 1B at first attempt compared with total number of candidates sitting MRCPCH part 1B for the first time.

There was no significant difference in examination outcome between candidates who concurrently sat the first written examination (part 1A) (pass rate 51/107, 47.7%) and those who did not (pass rate 792/1949, 40.6%) (OR sitting part 1a concurrently vs sitting part 1B alone 0.75 (95% CI 0.51 to 1.11), Pearson χ2 statistic=2.07 on one degree of freedom, p=0.15).

Multivariate regression model for association with passing the examination

The results of a multivariate logistic regression model for association with passing the examination is shown in table 2. The four potential predictors were UK versus international graduate, sex, age and concurrently taking the part 1A paper, and were available for 1896 candidates. Place of graduation was still strongly associated with passing the examination: after adjustment for place of qualification, sex and concurrent sitting of part 1A, increasing age was associated with a higher likelihood of passing the examination, with an increase in the odds of passing the exam of 3.4% per year (95% CI 0.3% to 6.6%). Gender and whether part 1A was taken at the same sitting were not related to examination performance.

Table 2

Logistic regression analysis assessing contribution of demographic factors to examination outcome

Discussion

This study identified a link between examination outcome for MRCPCH part 1B and country of primary qualification, UK medical school and age at time of sitting the examination.

Candidates who qualified in the UK were more likely to pass the MRCPCH part 1B at first attempt than IMGs. Under-performance of IMGs has been documented for other postgraduate examinations—for example, general practice (MRCGP),16 general medicine (MRCP)10 and psychiatry (MRCPsych).13 Contributing factors may include language use—for example, a high level of language ability may be necessary to fully appreciate the language subtleties and nuances of multiple-choice questions, and grasp of language complexity may be lower for IMGs than for UK-qualified doctors.25 In addition, familiarity with the UK health service and UK-based clinical protocols may also impact on examination success.

We also found considerable differences in pass rates between UK medical schools. Candidates who had qualified at the four highest performing medical schools had a 65.1% pass rate (138/212 candidates passing), a 2.7-fold higher pass rate than the graduates of the four medical schools with the lowest pass rates (61/257 candidates passing, 23.7%). Similar findings are reported for other specialties.8 ,12 ,14 ,15 Some medical schools perform well across several specialties26; it may be that these medical schools select for the brightest candidates. However, McManus et al12 demonstrated that 40% of the variance seen in MRCP postgraduate examination results remains unexplained when accounting for academic ability at medical school entry.

Little is known about the impact of variation in medical school curricula on initial career choice, postgraduate examination success, and career outcome. Future work might investigate whether the extent of paediatric material in undergraduate curricula is related to postgraduate examination outcomes, and whether implementation of nationally proposed undergraduate curricula impacts on examination outcome.20 A single examination for all medical graduates wishing to practise in the UK, such as that proposed by the General Medical Council,27 ,28 may also be a potential future driver to reducing some of the differences in medical school outcome. It is also interesting to note that the variation in the number of candidates from different UK medical schools undertaking the paediatric postgraduate examinations is not solely related to medical school size. This variation indicates differences in recruitment to paediatrics from different medical schools, which may be of interest to those involved in encouraging trainees to select paediatrics as their chosen specialty and in paediatric workforce planning.29

There was no effect of gender on examination outcome. Previous studies found consistent evidence for women performing better at undergraduate level,30 ,31 but did not identify significant differences at postgraduate level,11 ,32 for written papers, consistent with our data. Notable exceptions are the anaesthetic postgraduate written examination (FRCA), a negatively marked paper where men outperformed women,14 and the clinical parts of MRCP and MRCOG, where women performed significantly better,8 ,11 suggesting that there may be gender differences in performance for some examination techniques but not others.

We found that success at first attempt at part 1B was most likely for UK graduates aged 26–27 years. For IMGs, successful candidates were much older, probably reflecting experience gained elsewhere before coming to the UK and time spent becoming familiar with National Health Service practice. Whether part 1A and part 1B were taken simultaneously or separately did not affect part 1B success rates. There is little previous evidence regarding the effects of age on examination success in any specialty. The high pass rates for UK graduates in their early years of training supports the notion that candidates should ‘get on with’ their examinations early in their postgraduate experience.

Limitations of study

Demographic information was collected by the RCPCH on a voluntary basis, and missing data is the main limitation of our retrospective analysis. In particular, we were unable to analyse data on candidate ethnicity, since so few candidates (less than one-third) provided this information. Self-reporting of ethnicity has also been low in previous analyses.10 ,33 Given the increasing importance of monitoring high-stakes examination outcomes, it is vital that consideration be given to how ethnicity data collection can be improved. In addition, since studying at all five London medical schools (Barts and the London, University College London, Kings College London, St George's and Imperial College London) results in a primary medical qualification from the University of London, we were unable to differentiate between candidate performances for the five London medical schools, which accounted for 27% of our UK medical school sample; further information on these universities would be helpful in understanding performance variation.

Despite analysis by age, we do not know at which stage of their training candidates undertook the examination; this can only be inferred from the typical training pathway. In addition, the logistic regression analysis assumes a linear relationship between age and examination outcome, which may not be the case. Of further note, since this analysis was undertaken, MRCPCH part 1B has been modified to become the MRCPCH TAS examination with an altered syllabus. There are therefore limitations in extrapolating these results to the new modified examination.

Summary

Our findings demonstrate that country of primary medical qualification, UK medical school attended, and age are all significantly associated with likelihood of success in the MRCPCH part 1B examination. Similar associations have been reported for other specialties. Reasons for the difference in performance between IMG and UK graduates are complex, but issues regarding preparedness for practice and language use may be amenable to targeted interventions. The difference in performance by UK medical schools raises questions about differences in quality of undergraduate exposure to paediatrics and relevant associated teaching such as basic sciences and suggests that undergraduate curriculum design may be an important area for future research.

Main messages

  • Success in this written part of the paediatric postgraduate examination is significantly associated with country of primary medical qualification, UK medical school and age.

  • There is no evidence to support an effect of gender on likelihood of success.

Current research questions

  • What interventions could reduce the disparity between UK and international medical graduates in examination performance?

  • What factors influence discrepancies in postgraduate examination performance between candidates who attended different UK medical schools?

  • Would a national undergraduate curriculum in child health reduce differences in performance of UK medical schools seen at postgraduate level?

Key references

  • Report on MRCP(UK) and Specialty Certificate Examination Pass Rates by Gender and Ethnicity: Federation of Royal Colleges of Physicians of the UK, 2012.

  • Dewhurst NG, McManus C, Mollon J, et al. Performance in the MRCP(UK) Examination 2003–4: analysis of pass rates of UK graduates in relation to selfdeclared ethnicity and gender. BMC Med 2007;5:8.

  • McManus IC, Elder AT, de Champlain A, et al. Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations. BMC Med 2008;6:5.

  • Wakeford R. International medical graduates' relative under-performance in the MRCGP AKT and CSA examinations. Educ Prim Care 2012;23:148–52.

Acknowledgments

We wish to thank in particular Dr Simon Newell, RCPCH Vice President in Training and Assessment, for valuable comments and feedback on the manuscript. In addition, we thank Graeme Muir, RCPCH Education and Assessment Advisor, for help with obtaining the data analysed in this paper. Without the help and assistance of the College, this work would not have been possible.

References

View Abstract

Footnotes

  • LM and SM share first authorship and contributed equally to this work.

  • Contributors LM contributed to study design, data cleaning, data analysis, data interpretation, and drafting and critically revising the work for publication. She is the guarantor. SM contributed to study conception and design, data cleaning, data analysis, data interpretation, and drafting and critically revising the work for publication. AB contributed to study conception and design, data cleaning, data analysis, data interpretation, and drafting and critically revising the work for publication. AD-M contributed to data acquisition, data cleaning, data interpretation, and critical revision of the intellectual content of this work for publication. AL contributed to study conception and design, data interpretation, and critical revision of the intellectual content of this work for publication. CF contributed to study conception and design, data analysis and interpretation, and critical revision of the intellectual content of this work for publication. All authors have given final approval of the version to be published and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

  • Competing interests LM is a paediatric trainee within the London Deanery. She is an Academic Clinical Fellow at the Institute of Cognitive Neuroscience and Institute of Child Health, London; her post is funded by the National Institute for Health Research (NIHR). She is a member of the London Specialty School of Paediatrics Trainee Committee Assessment sub-group and undertook this study as part of her role in this organisation. AB is currently a paediatric trainee in the East of England Deanery; SM is a paediatric trainee in London. At the time this work was undertaken, both held the post of Fellow in Medical Education in the London Specialty School of Paediatrics. AD-M is part of the Education and Training Division at the Royal College of Paediatrics and Child Health. Andrew Long is the Head of the London Specialty School of Paediatrics and a Consultant Paediatrician at Great Ormond Street Hospital. CF is a Consultant Paediatrician at The Whittington Hospital, London, leads the Assessment subgroup at the London Specialty School of Paediatrics, and is convenor of the RCPCH Paediatric Education Special Interest Group (PEdSIG).

  • Ethics approval The administrative dataset analysed in this study is collected routinely and voluntarily from all MRCPCH candidates on the basis that it may be analysed anonymously as part of quality assurance. The data were transferred retrospectively to us in a fully anonymised format, in accordance with the RCPCH data protection policy. Ethics permission beyond that issued by the RCPCH was therefore not obtained, as this was not necessary.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement The data used to produce this manuscript were obtained in anonymous format from the Royal College of Paediatrics and Child Health, in accordance with a data handling agreement from the College; no additional data are available.

  • Data access All authors, external and internal, had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis. LM (lead author and guarantor) affirms that the manuscript is an honest, accurate and transparent account of the study being reported, that no important aspects of the study have been omitted, and that any discrepancies from the study as planned (and, if relevant, registered) have been explained. No specific funding was obtained for this study; hence all authors were independent of any funding body.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.