Article Text

Download PDFPDF

Identifying critical success factors for designing selection processes into postgraduate specialty training: the case of UK general practice
  1. Simon Plint1,
  2. Fiona Patterson2
  1. 1GP National Recruitment Office, and GP Dean NHS Education South Central, Oxford, UK
  2. 2City University, London and Work Psychology Group Ltd, UK
  1. Correspondence to Dr Simon Plint, NHS Education South Central, The Triangle Roosevelt Drive, Oxford PGMDE OX3 7XP, UK; splint{at}oxford-pgmde.co.uk

Abstract

The UK national recruitment process into general practice training has been developed over several years, with incremental introduction of stages which have been piloted and validated. Previously independent processes, which encouraged multiple applications and produced inconsistent outcomes, have been replaced by a robust national process which has high reliability and predictive validity, and is perceived to be fair by candidates and allocates applicants equitably across the country. Best selection practice involves a job analysis which identifies required competencies, then designs reliable assessment methods to measure them, and over the long term ensures that the process has predictive validity against future performance. The general practitioner recruitment process introduced machine markable short listing assessments for the first time in the UK postgraduate recruitment context, and also adopted selection centre workplace simulations. The key success factors have been identified as corporate commitment to the goal of a national process, with gradual convergence maintaining locus of control rather than the imposition of change without perceived legitimate authority.

  • Human resource management
  • medical education & training
View Full Text

Statistics from Altmetric.com

Introduction

For many occupational groups there exists a large body of international research investigating best practice selection.1 In medicine, there is a significant volume of research exploring medical school admissions procedures and the link to subsequent performance during medical school. However, there is relatively little research on developing selection methodology for entry to postgraduate training. This paper presents the case of selection for training in UK general practice, which has now successfully completed 3 years of national recruitment, to highlight critical success factors in the design of selection processes into postgraduate specialty training. We report on this case to encourage further debate and research activity in developing selection into specialty training in general.

Background

Before the development of the national selection system, the average applicant for GP training was assessed 2.5 times (one candidate was tracked making 12 applications), and research showed that different selection methods and different standards were applied across different deaneries (box 1). With variable standards and lack of equity, “the time honoured selection process had neither guidelines nor standards: recruitment based on patronage was considered the norm; and the potential for discrimination was rife”.2 Directors of postgraduate GP education were therefore given responsibility for developing a selection and training system, ensuring that “the process of recruitment demonstrates fair and open competition, and observes best practice in providing equal opportunities for all applicants”.3 A collective commitment was made to work towards standardisation of recruitment methods, which resulted in the national process. Currently, the GP selection process uses a national machine markable shortlisting test together with selection centre methodology, which has developed incrementally over 7 years. Indeed, the GP selection process survived the collapse of confidence in the Medical Training Application System (MTAS) introduced in the UK in 2007,4–6 with corporate confidence and commitment to the national process, deriving from convergence rather than imposition of process.

Box 1 Analysis of outcomes of multiple applications

Previously an applicant for GP training was assessed on average 2.5 times, and multiple applications ironically offered a unique opportunity to compare the outcomes of applications for the same candidate in different deaneries. Analysis showed only fair agreement for shortlisting outcome (‘shortlistable’ or ‘not shortlistable’) and worse than chance agreement (meaning deliberate disagreement) for interview outcome (‘appointable’ or ‘not appointable’).29 (The Kappa co-efficient compares agreement between paired outcomes where 1 signifies perfect agreement and 0 signifies chance agreement.)30

Analysis of outcomes of multiple applications

Number of paired outcomesKappa coefficient
Agreement on shortlisting outcome (‘shortlistable’ or ‘not shortlistable’)78420.3
Agreement on interview outcome (‘appointable’ or ‘not appointable’)396-0.3

This descriptive paper outlines the development of a robust selection system for UK general practice, identifying the success factors and the lessons for the future.

Developing a robust selection system

The key elements involved in designing and implementing any selection system are outlined in figure 1. The process starts by conducting a thorough analysis of the relevant knowledge, skills, abilities and attitudes associated with performance in the target role.7 This information is used to construct a person specification (and job description where appropriate), which in turn informs which selection instruments and methods can be designed to evaluate the candidate's capabilities against these attributes.8 A validation process should then be used to assess the extent to which the selection methods provide valid predictors of progress in training or job performance.9 Research shows that candidates prefer selection methods that are job relevant10 and offer them the opportunity to demonstrate their ability, as well as treating them sympathetically10; because of this, work sample and assessment centres are significantly less open to legal challenge.11

Figure 1

The selection process.

In UK general practice, a job analysis was followed by the agreement of a national person specification, together with the creation of a national recruitment office, and the introduction of a common timetable for recruitment rounds. Standardised application forms were developed, and eventually all applications were processed through the national recruitment office, with the potential for quality management. A competency model for selection into GP training was adopted because of the evidence base behind its development, in keeping with research into best practice selection.9 The focus is on demonstrated competency domains rather than reported competencies—the difference between a candidate reporting consultation performance in an interview and actually demonstrating competence in an assessed simulation of a consultation.12 General practice also developed an online recruitment system which was successfully piloted, demonstrating that a “central application portal with local selection”13 can work effectively in the UK, as in other countries, for recruitment into medical training (figure 2).

Figure 2

The UK national general practitioner selection process. MMT, machine markable test.

Development of the machine markable test for shortlisting

One key limitation of using standardised application forms is the developing industry of websites and organisations which provide model answers to questions and provide guidance for presenting curriculae vitae. As a consequence GP recruiters made the decision that the application forms would have to be completed under invigilated conditions, but only as an interim measure. The long term goal was to develop a machine markable test (MMT) that would assess applicants against cognitive (clinical problem solving) and non-cognitive (eg, empathy, integrity, resilience) criteria in the person specification. The cost of hiring testing centres and invigilating thousands of applicants, along with individually marking more than 100 000 questions, would be vastly outweighed by the costs saved having a machine markable assessment. The development of the MMT with clinical problem solving and situational judgement components, and its successful implementation, has been a significant innovation in UK postgraduate specialty selection.14

The further advantage of a validated shortlisting test is its reliability and potential to rank applicants, not just locally but nationally. By introducing a national shortlisting process, all applicants can be assessed on the same day across the UK; then ranked and allocated for interview to their highest preference deanery (a deanery is a regional office in the UK responsible for education and training of junior doctors) available in rank order. Although such a system restricts applicants to only one interview, it also guarantees that a much greater number of applicants get the chance of interview and the chance of appointment. It also avoids the potential scenario of the same small number of applicants being offered all the available interviews, as happened with MTAS in 2007.6 This results in an equitable distribution of applicants around deaneries, but of course cannot standardise the levels of competition in different locations (as has always been the case for more or less popular locations in the UK). With this system, candidates assessed as appointable have the opportunity for clearing across other deaneries (so-called ‘cascading score’ system) where vacancies remain. For GP recruitment in 2007 and 2008, all posts across the UK were filled with only one round and national clearing, and in 2008 the allocation algorithm resulted in 80% of appointments being offered in first choice location, and 16% in second choice location.

Validity and utility of the GP selection process

The MMT delivered standardisation of assessment, but the real challenge was to develop a selection methodology that would be perceived and demonstrated to be valid. Evaluation of the process has shown increased reliability of selection, and increased predictive validity of performance during GP training.14–16 Evaluation of the national GP selection system from the candidate's perspective (over N=6000 each year) has also consistently demonstrated that applicants have confidence in the relevance and fairness of the process. Not only are the MMTs for shortlisting well received in terms of fairness and relevance to the GP role, there has also been resounding confidence in the fairness (90%) and the professionalism (92%) of work relevant simulations comprising the selection centres, including a patient simulation exercise (table 1).

Table 1

Candidate evaluation of shortlisting machine markable tests

Resource efficiency is a critical consideration when designing selection processes for medical training. A cost comparison between the machine marked test and the previous hand marking of the competency based application questions for short listing has estimated that the MMT compares very favourably (£20 compared with £50 per applicant.) However, the selection centre methodology incorporating workplace simulations is more expensive than interviews (estimated £200 compared with £100 per applicant). The GP selection centre assesses candidates in three exercises over a period of 90 min, involving three separate assessors and an actor, compared with the previous structured interview process which typically would have comprised two interviews over a period of 40 min, involving at least four assessors.

However, the utility analysis is not simply about the costs of the selection process, but must balance them against the long term costs of appointing doctors with capability or suitability problems, who either do not complete training or do not follow a career in general practice. Historically, the NHS has not invested the resources in selection of doctors into training and substantive posts, which are seen in industry, with the often compared airline industry as the most obvious example.

Selection processes face unprecedented scrutiny over the next few years in the UK, and general practice is undertaking a long term validation of its competency based selection methods, as an integral part of the national selection process. The measures will include the predictive validity of the selection process performance against training progression and the end point licensing exam performance. Ideally, the utility analysis would also take into account subsequent career progression, but the challenge of measuring the ‘good doctor’ is well recognised in medicine across the world.

Success factors for national selection

Almost certainly a critical success factor is corporate commitment of recruiters—as witnessed in general practice over the last 7 years—to develop a national process, combined with the appropriate authority and locus of control. The process has been one of gradual convergence, with deaneries (and, indeed, devolved nations) opting in rather than having selection methods imposed. All deaneries have played their part in making incremental improvements to the selection method.

However, it is not just the selection methodology that has to be corporately ‘owned’ and validated; it is the system supporting the process that is also important. This requires an infrastructure, which, in general practice, has been provided by the creation of the national recruitment office (NRO) to manage recruitment, together with a national recruitment steering group, supported by three operational groups.17 A critical success factor is the perception that the NRO is working on behalf of the deaneries, and that there is a rational basis underlying its authority to lead the national process. The office has been empowered to lead from bottom up representation, rather than top down imposition. There has always been a commitment to professionalism, with administrators working together to produce manuals of procedure and best practice, and deaneries ensuring that all assessors and simulators have been properly trained for the best practice selection procedures. Each quality standard is proposed and discussed at national recruitment meetings before broader consultation and adoption. In terms of delivery, perhaps the most important factor for the successful introduction of an electronic platform is a well established stability of the recruitment processes. An electronic system is therefore a solution for an explicitly defined process, rather than a system that needs constant redesign.

Best practice selection

The GP selection process is entirely consistent with the research literature on selection in other professions18 (see box 2) and, indeed, with developments in medical school admissions. In the UK, other specialities are now introducing selection centres, and piloting the use of the machine marked test for shortlisting purposes.19–21

Box 2 Predictive validity of selection methods from (Robertson and Smith 200127)

  • Graphology (0.02)

  • Years job experience (0.18)

  • References (0.26)

  • Conscientiousness (0.31)

  • Biodata (0.35)

  • Assessment centers alone (0.37)

  • Unstructured interviews (0.38)

  • General mental ability alone (0.51)

  • Structured interviews alone (0.38)

  • Work sample tests alone (0.54)

  • General mental ability + work sample tests (0.60)

  • General mental ability + structured interviews (0.63)

  • General mental ability + integrity tests results (0.65)

Undergraduate medical schools are increasingly supplementing the baseline of academic achievement with the BMAT aptitude test,22 and supplementing structured interviews with authentic scenarios23 or even patient simulations24 assessing competencies.

Is there a place for biographical data (biodata), such as medical employment history, in competency based selection? One of the major criticisms of MTAS, and indirectly the PMETB,25 was that short listing was undertaken blind to previous employment history and verifiable achievements, with the long term concern that medical students and doctors would lose the incentive to work towards them.26 In other occupations, biodata have been shown to have predictive validity for training progress and job performance,27 and the maxim is well known that past performance is the best predictor of future performance.28 However, there remains the problem that doctors with longer employment history will accumulate more lines in their CVs, and shortlisters now wrestle with the challenge of distinguishing career progression. It could be argued that actual performance, demonstrated in a competency based assessment, as opposed to reported past performance, is an even better predictor of future performance.

Significantly, the GP selection process does not assess biodata except to determine relevant experience for eligibility. Indeed, the original validation work showed neither further medical qualifications nor medical employment were significantly associated with selection centre performance.15 This may be singular to general practice, where previous hospital employment and qualifications will not necessarily confer any advantage for exercises in the context of general practice; however, the straightforward explanation may be that the best candidates perform the best in a competency based assessment system, without any other variables.

Conclusions and lessons for the future

The case of GP national recruitment provides an opportunity to inform development of selection systems for entry into postgraduate training in general. Best practice recruitment processes are developed incrementally through rigorous piloting and methods must be validated; crucially they must also be perceived as appropriate and fair assessments by candidates. Best practice selection systems for medicine tend to comprise multiple assessments reflecting the complexity of the job role. For entry into postgraduate training future methods will probably include biographical data evidencing academic excellence and conscientiousness, tests of applied clinical knowledge, and specialty relevant non-cognitive domains, together with interviews and workplace simulations. The person specification and relative weighting of each method should be informed by longitudinal validations studies.

For the immediate future in the UK, specialties are now working collectively and corporately to develop equivalent recruitment processes for the future, as demonstrated by current initiatives led by UK Royal Colleges. This work has important implications for other nations in developing selection methodologies and recruitment systems for specialty training.

Summary of success factors

  1. Corporate commitment to national process

  2. Legitimate authority and locus of control

  3. Process of incremental convergence, rather than imposition

  4. Development and adoption of validated selection method

  5. Representative infrastructure operating the process

  6. Electronic recruitment solution only possible when selection process established

Future challenges

  1. Long term validation studies of selection process

  2. Evolution of selection methods

  3. Utility analysis in managed NHS economy

Acknowledgments

Pat Lane acknowledged as the inspiration behind the development of a national recruitment process, Gai Evans as the Manager of the National Recruitment Office, and all the recruitment teams around all the deaneries who have played their corporate part in making it all happen. Lara Zibarras is acknowledged for her support in preparing a final version of the paper.

References

View Abstract

Footnotes

  • Authorship: SP conceived of the original idea, interpreted the evaluation data and wrote the paper. FP conceived of the evaluation, interpreted the evaluation data and wrote final versions of the paper.

  • Guarantor: SP is guarantor for the paper.

  • Competing interests Both authors have been involved in the design and development of the GP recruitment process. SP is seconded to the Department of Health MMC team, but the views expressed are personal, derived from the experience of leading the national recruitment process on behalf of the Committee of General Practice Educational Directors. FP is employed by the Work Psychology Group which advises the Department of Health on selection methodology.

  • Provenance and peer review Not commissioned; not externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.