Abstract
Context: Competition for postdoctoral training positions is at an all-time high, and residency program directors continue to have little direction when it comes to structuring an effective interview process.
Objectives: To examine whether a relationship existed between interview methods used and program director satisfaction with resident selection decisions and whether programs that used methods designed to assess candidate personal characteristics were more satisfied with their decisions.
Methods: Residency directors from the Statewide Campus System at the Michigan State University College of Osteopathic Medicine were invited to complete a 20-item survey regarding their recent interview methods and proportion of resident selections later regretted. Data analyses examined relationships between interview methods used, frequency of personal characteristics evaluated, and subsequent satisfaction with selected residents.
Results: Of the 186 program director surveys distributed, 83 (44.6%) were returned, representing 11 clinical specialty areas. In total, 69 responses (83.1%) were from programs accredited by the American Osteopathic Association only, and 14 (16.9%) were from programs accredited dually by the American Osteopathic Association and Accreditation Council for Graduate Medical Education. The most frequent interview method reported was faculty or peer resident interview. No statistically significant correlational relationships were found between type of interview methods used and subsequent satisfaction with selected residents, either within or across clinical specialties. Although program directors rated ethical behavior/honesty as the most highly prioritized characteristic in residents, 27 (32.5%) reported using a specific interview method to assess this trait. Program directors reported later regrets concerning nearly 1 of every 12 resident selection decisions.
Conclusion: The perceived success of an osteopathic residency program’s interview process does not appear to be related to methods used and is not distinctively different from that of programs dually accredited. The findings suggest that it may not be realistic to aim for standardization of a common set of best interview methods or ideal personal characteristics for all programs. Each residency program’s optimal interview process is likely unique, more dependent on analyzing why some resident selections are regretted and developing an interview process designed to assess for specific desirable and unwanted characteristics.
The resident selection process identifies candidates who are most likely to perform well in the residency training program.
1-3 The first step of this process is to screen candidates and identify interviewees. Application materials, board scores, academic performance, dean’s letters, performance on clerkship rotations, and any other information available at the time of application are reviewed. The next step is the interview process, with the ultimate goal of selecting which candidates to rank. Candidate interviewing and selection has been shown to be daunting, sometimes confusing, and frequently stressful for program directors and faculty members on selection committees.
1-5
Graduate medical education (GME) literature has analyzed the correlation between academic performance on standardized tests with future performance on residency board and in-training examinations.
6-9 However, the correlation between medical school academic performance and overall success in residency as determined by faculty member ratings remains fairly weak,
1,8,9 and the interview process with a highly competitive pool of candidates is even more critical. Selecting the right candidate can also prevent the need for remediation.
Residency program directors should look beyond traditional objective criteria for better predictors of future success and consider candidates’ personal characteristics.
1-3,7,10 In addition to demonstrating strong academic achievement, the best-performing residents are often those who fit well culturally and organizationally into a residency program and who share the same overall values, attitudes, and behaviors as the existing program staff and faculty.
11-14
The annual National Residency Match Program Update,
15 based on a national survey of program directors, consistently lists personal characteristics among the 5 most important factors in determining rank order lists. Some GME programs have attempted to identify 1 or more specific personal characteristics most favorably correlating with postmatch success during residency, largely defined by faculty ratings of overall resident performance.
12-14,16,17
In the selection process, the candidate interview is used to assess a candidate’s academic performance and organizational fit based on personal and professional characteristics.
1,3,18-20 The structure of residency interviews can vary widely by program and may comprise a singular in-person interview or an interview with multiple people, methods, or assessment types. The interview process is time-consuming and often unstructured and cannot guarantee selection of residency candidates who will ultimately be successful and fit well organizationally.
3,10,15,18 Interviews that include the use of additional screening methods to assess personal characteristics such as psychomotor skills assessments, team collaboration exercises, and personality inventories have demonstrated some success in selecting ideal candidates.
3,21,22
The standardized multiple mini interview is a method that has gained popularity in undergraduate medical applications and GME screening.
16,23,24 Other factors likely to affect the success of a program’s resident interview process may include whether interview questions are strictly standardized
14,17,25,26 or whether multiple interviewers participate and provide multiple data points to improve the reliability of the ratings based on clinical specialty or program-specific criteria.
3,5,13 More residency programs are incorporating a wider variety of additional assessment methods into their interview processes to improve their capability of selecting the most desirable residents for their programs.
4,5,10,13,17,26
Much of the existing literature about resident interview methods has come from surgical subspecialties (ie, non–primary care) programs.
2,3,7,10-12,17,21,25,27 To our knowledge, no recent studies have compared the variation of interview methods used within different specialty programs and perceived success in selecting resident candidates or compared interview methods between allopathic and traditional osteopathic programs. Regardless, an improved understanding of which methods are typically used and perceived to be most effective in selecting successful candidates is acutely needed to inform osteopathic GME officials who make selection decisions.
4,13,19,26
In this descriptive correlational pilot study, we investigated differences in the type and quantity of interview methods used by residency programs across multiple specialty disciplines and whether selecting certain personal characteristics influenced which interview methods were used. We also sought to examine what proportion of residency program selection decisions were determined by the program directors to be satisfactory or problematic during a 5-year period and whether any correlation existed between satisfaction rates and interview methods used.
Following institutional review board approval as an exempt study, program directors from 35 community-based training institutions representing 20 different medical specialties were recruited in the fall of 2011. All residency training programs were accredited by the American Osteopathic Association (AOA), the Accreditation Council for Graduate Medical Education (ACGME), or both (AOA/ACGME). All programs were members of the Statewide Campus System educational consortium at the Michigan State University College of Osteopathic Medicine.
Directors received e-mail invitations to complete a 20-item online survey regarding their interview methods (eg, faculty interview, ethical problem-solving exercise, multiple mini-interview technique, team collaboration exercise) and criteria for choosing residents for their programs. Survey questions focused on whether applicant personal characteristics factored into designing the interview process, which traits were the most highly prioritized by GME faculty and officials, what additional interview method(s) were used to assess for these characteristics, and how successful the process was later perceived to be in terms of accurately identifying candidates who would be successful in the training program. Directors were also asked to report the type of residency or fellowship program and accreditation type. Last, directors were asked to rank personal characteristics on a scale of 1 (highest priority) to 10 (least essential personal characteristic).
For our study, the overall success of a program’s interview process was measured based on program directors’ responses regarding the proportion of residents selected over a 5-year period who required formal remediation, were removed from the program, or were otherwise concluded to be an especially “poor fit” for the program.
Two additional follow-up e-mail survey requests were sent, each 1 week apart, comprising a total survey period of 3 to 4 weeks. We did not seek to identify whether specific personal characteristics were possessed by a majority of successful residents and assumed that candidates who were invited to an interview had already met the minimum academic screening standards of the program. Descriptive analyses were performed, as well as subsequent correlational analyses and analysis of variance procedures with α=.05. Statistical analyses were performed using SPSS statistical software version 22 (IBM).
Of 186 residency and fellowship program directors, 83 (44.6%) responded to the 20-item survey. Not all respondents answered each question. Of the 83 respondents, 29 (34.9%) were in a primary care specialty area (eg, family medicine, internal medicine, pediatrics). Of the remaining respondents, 28 (33.7%) were in a surgical subspecialty; 12 (14.5%) were in emergency medicine; and 14 (16.9%) were in cardiology, neuromusculoskeletal medicine, obstetrics and gynecology, psychiatry, radiology, and sports medicine programs. The majority of respondents (69 [83.1%]) represented AOA-only programs, and 14 (16.9%) were from AOA/ACGME programs.
The characteristics rated as most important by respondents were ethical behavior/honesty; being a team player and fitting in with existing team dynamics; and self-motivation (
Figure 1).
Of 83 respondents, 13 (15.7%) reported using specific method(s) to assess ethical behavior/honesty, 20 (24.1%) reported using tools or method(s) to assess team dynamics, and 2 (2.4%) reported using a method to evaluate self-motivation. Initial inferential data analysis indicated no significant correlation between quantity of interview screening methods aimed at assessing candidates’ personal characteristics and subsequent satisfaction with selected residents’ program performance (Pearson r=0.026, P=.83).
No notable differences were found in the specific methods used by AOA-only programs and AOA/ACGME programs. No substantial differences were found in interview methods used by surgical, nonsurgical, and primary care residency programs, or the respondents’ later satisfaction levels with residents who had been admitted into their programs.
The 3 most frequently reported candidate assessment methods used during the interview stage were one-on-one faculty interviews (52 [62.7%]), faculty panel interviews (40 [48.8%]), and interviews with current residents (73 [89.0%]) (
Figure 2).
A majority of programs (79 [95.2%]) reported using multiple methods as part of the interview process. Between 1 and 8 different methods were incorporated into the total interview screening process of individual residency programs, including methods such as ethical problem-solving exercises, medical knowledge tests, and team collaboration exercises. A 1-way analysis of variance demonstrated a nonsignificant correlation between the number of screening interview methods used by respondents beyond some form of interview and later satisfaction with selected residents (F23,71=0.296, P=.83). Respondents reported later regretting nearly 1 in every 12 resident selections (8.3%) they had made during the previous 5 academic years.
Several studies concerning resident selection and interview methods have focused on the evaluation of factors available at the time of application to identify predictors of future performance in residency, including academic performance indicators, reference letters, board examination scores, and clinical rotation evaluations.
2,4,6,7,10,12,17,25 The current study sought to determine if methods used exclusively during the interview process could influence selection of best-fit residents. Replicating the current study with a larger sample may enable GME leaders to better determine the specific components of an interview process that could be identified as critical elements for successful selection of residents from multiple candidate interviews. An in-depth review of interview methods and criteria between osteopathic and allopathic programs may also provide insight into best practices for structuring the residency interview process.
Some residents require remediation efforts, and program directors should select residents who are a good fit culturally as well as technically with the training program.
11-14,18 More systematically examining the reasons program directors are later dissatisfied with some of their resident selections would provide better information about the deficiency in critical success factors and whether these deficiencies are attributable to the applicant screening phase or interview phase (eg, academic performance, personal characteristics, or other factors that contributed most to eventual dissatisfaction).
Studies have sought to pinpoint the reasons some residents eventually require remediation.
8,18,28 Other researchers have looked for better ways to prevent the need for remediation through careful structuring of the interview process.
12,15,29 The present study was unique in evaluating whether the quantity and quality of interview methods influenced program directors’ subsequent satisfaction with resident selection decisions and whether any notable differences existed between the interview processes of osteopathic and allopathic programs or surgical and nonsurgical programs.
The identification of specific personal characteristics as critical to success factors in selecting residents, and optimal methods to screen for these variables during interviews is a focus of research.
1,3,30,31 The present study offers an alternative perspective by examining whether program directors who intentionally interviewed for specific characteristics were more satisfied with the resulting selections. Although being ethical, honest, a team player, and self-motivated were the most highly ranked traits by respondents in the present study, few programs specifically attempted to assess these traits in their applicants during the interview phase, possibly because of the lack of tools to do so.
Programs are likely able to gain valuable information regarding the salient personal characteristics of potential residents during previous clerkship rotation experiences. Although the present study examined specific methods used by programs at the interview stage rather than elements evaluated earlier in the applicant phase, we acknowledge that this is a limitation and could provide additional insight into why programs have not assessed specific personal characteristics during interviews.
The results of the current study suggest that program directors choosing a multifaceted interview process that incorporates multiple methods may not be more effective in identifying candidates who are the best overall fitfor their programs. The inferential data analytic procedures we used did not indicate a correlation between type or quantity of interview methods used and subsequent satisfaction with selected residents’ performance in the residency program, and they did not identify notable differences between interview methods of AOA-only programs and those of AOA/ACGME programs. These results may be caused by difficulties in quantifying the degree of satisfaction or the fit of a particular resident in a program. Although our data suggest it still may be possible to choose the best-fit candidates using an interview-only process, the objectives of each program’s overall selection process should be better defined, and an interview strategy designed to satisfy these objectives should be developed.
Future studies should examine how recall bias on the part of program directors could affect the number of resident selections directors have regretted over the past 5 academic years, especially in larger programs. Future studies should also consider other factors that are difficult to measure that contribute to why GME programs select residents who do not perform adequately during their residency. An additional complicating element in most clinical specialty areas is that residency programs are unable to control whether their top-ranked candidates ultimately make the choice to join their program, indicating that program director dissatisfaction with some resident choices may be due at least in part to matching with a lower-ranked (less desirable) candidate.
27,32,33 Future studies might compare programs using interview methods based on objectives that include screening for desired personal characteristics and programs using interview methods not tied to assessment of desired personal characteristics to determine if overall satisfaction rates are markedly different.
The perceived success of a program’s recruitment efforts did not appear to be related to the type of interview methods used, and no distinct differences existed between methods used and satisfaction rates of AOA-only programs and AOA/ACGME programs. Developing a universal protocol of best interview methods or identifying a standard set of personal characteristics ideal for all programs may not be feasible. The best interview process for many GME residency programs regardless of specialty may not be dependent on which interview methods are used or how complex the interview process is. The ideal selection process is likely more dependent on identifying and intentionally screening out candidates possessing characteristics shared by past poorer performers. The findings of this study indicate that no specific interview methods have emerged as potential solutions, which is a clear indication of the need for better tools to discover candidates’ personal characteristics, such as behavioral interview questionnaires or personality testing during the interview process. Future larger-scale projects that examine this vitally important area of GME program administration are needed.
We thank William Corser, PhD, research specialist at the Statewide Campus System at the Michigan State University College of Osteopathic Medicine for editorial assistance; Jenn Sysak, MA, for developing the figures; and Robert Crawley for data collection support.