Abstract
Context:
Medical schools use a variety of preadmission indices to select potential students. These indices generally include undergraduate grade point average (GPA), Medical College Admission Test (MCAT) scores, and preadmission interviews.
Objective:
To investigate whether the admission indices used by Midwestern University Arizona College of Osteopathic Medicine are associated with the academic and clinical performance of their students.
Methods:
Associations between the prematriculation variables of undergraduate science GPA, undergraduate total GPA, MCAT component scores, and interview scores and the academic and clinical variables of the first- and second-year medical school GPA, Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 1 and Level 2-Cognitive Evaluation (CE) total and discipline scores, scores in clinical rotations for osteopathic competencies, COMLEX-USA Level 2-Performance Evaluation passage, and match status were evaluated. Two-tailed Pearson product-moment correlations with a Bonferroni adjustment were used to examine these relationships.
Results:
The traditional predictors of science and total undergraduate GPA as well as total and component MCAT scores had small to moderate associations with first- and second-year GPA, as well as COMLEX-USA Level 1 and Level 2-CE total scores. Of all predictors, only the MCAT biological sciences score had a statistically significant correlation with failure of the COMLEX-USA Level 2-Performance Evaluation examination (P=.009). Average interview scores were associated only with the osteopathic competency of medical knowledge (r=0.233; n=209; P=.001), as assessed by clerkship preceptors. No predictors were associated with scores in objective structured clinical encounters or with failing to match to a residency position.
Conclusion:
The data indicate that traditional predictors of academic performance (undergraduate GPA, undergraduate science GPA, and MCAT scores) have small to moderate association with medical school grades and performance on COMLEX-USA Level 1 and Level 2-CE. This finding requires additional research into the value of the interview in the medical school admissions process and the availability of alternatives that allow better prediction and assessment of applicant performance.
Selection of appropriate candidates for medical school is an important first step in shaping the future of medical care and the reputation of the medical school. The process developed by colleges of osteopathic medicine (COMs) to determine how applicants are selected should, in some ways, also predict the likelihood of the student's success in medical school, matching to the specialties of their choice, and pursuing a successful career in medicine.
Considerable research
1-5 has examined the value of quantitative prematriculation predictors of clinical and professional performance
, such as undergraduate grade point averages (GPAs) and Medical College Admission Test (MCAT) scores for allopathic and osteopathic medical students, but less information is available regarding the value of the medical school admissions interview. As more focus is paid to the development of clinical and professional skills and assessment of the entrustability of medical school graduates,
6-8 determining a medical student's professionalism and “soft skills” (eg, communication, empathy) is linked to both residency performance and future professional standing.
9
The aim of our study is to examine the associations between prematriculation variables (undergraduate science and total GPAs, component and total MCAT scores, and interview scores) and measures of academic, clinical, and professional performance (first- and second-year medical school GPAs, Comprehensive Osteopathic Medical Licensing Examination-USA [COMLEX-USA] Level 1 and Level 2 total and discipline scores, preceptor evaluations of students in the 7 osteopathic competencies, objective structured clinical encounter [OSCE] scores, and residency match outcomes) in students at the Midwestern University Arizona College of Osteopathic Medicine (MWU/AZCOM).
The Midwestern University Institutional Review Board reviewed and approved this study. Prematriculation undergraduate science GPA; total GPA; total and component MCAT sciences, physical sciences, verbal reasoning, and combined total numeric scores; and average admissions interview scores were retrieved for all members of the incoming MWU/AZCOM class of 2016. The semi-blinded interview process for MWU/AZCOM consists of 1 basic science faculty, 1 clinical science faculty, and 1 third- or fourth-year medical student. Interviewees were rated on a scale of 1 to 6 (1 being the worst score and 6 being the best) in the areas of professional motivation and preparedness, problem solving and critical thinking, personal qualities, and overall performance. These 4 ratings are combined to determine the total interview score.
In addition, first- and second-year GPA, COMLEX-USA Levels 1, 2-Cognitive Evaluation (CE), and 2-Performance Evaluation (PE) scores, competency domain scores from clinical clerkship rotations in the third year, and final scores on the OSCE at the end of third year were obtained for the same individuals. The third-year core rotations include 4-week rotations in cardiology, obstetrics and gynecology, pediatrics, psychiatry, and surgery, with 2 rotations each in family medicine and internal medicine. One internal medicine rotation is required to be done with a residency team; other rotations may be inpatient or outpatient and with an individual preceptor or a residency team.
In June 2016, residency match information was obtained for the MWU/AZCOM class of 2016—specifically, whether the student matched to a residency program accredited by the American Osteopathic Association or the Accreditation Council for Graduate Medical Education, whether the student found a position in the post-match Scramble (for programs accredited by the American Osteopathic Association) or the Supplemental Offer and Acceptance Program (for programs accredited by the Accreditation Council for Graduate Medical Education), or whether the student failed to find a postgraduate position. If a student matched during the National Matching Services or the National Resident Matching Program, that student was considered matched. If a student found a position during either of the postmatch programs or failed to find a position, we counted that student as failing to find a position in the match.
Measures of association between the prematriculation variables and each set of performance variables were calculated using 2-tailed Pearson product-moment correlation coefficients or Fisher exact test. Because of the number of correlations, Bonferroni adjustments were made to the significance tests, requiring P≤.002 for statistical significance. Because of the low number of students who failed the COMLEX-USA Level 2-PE and to match in the residency match, a 2-sided Fisher exact test was performed to test the association between the pre-matriculation variables and the proportion of students who failed the COMLEX-USA Level 2-PE and who failed to match.
Information from 206 of 219 graduating students in the class of 2016 at MWU/AZCOM was included in the analysis.
Table 1 shows the 2-tailed Pearson product-moment correlation coefficients for each of the prematriculation variables and the outcome variables of first- and second-year GPA and discipline and total scores on COMLEX-USA Level 1 and 2-CE. Undergraduate science GPA had small but statistically significant positive correlations with first-year (
r=0.285, n=208,
P<.001) and second-year (
r=0.285, n=206,
P<.001) medical school GPA. Undergraduate total GPA had moderate, statistically significant correlations with first-year (
r=0.326, n=208,
P<.001) and second-year (
r=0.331, n=206,
P<.001) medical school GPA and COMLEX-USA Level 2-CE total score (
r=0.226, n=210,
P=.001).
Of the MCAT component scores, verbal reasoning had small but statistically significant correlations with COMLEX-USA Level 2-CE total score (r=0.228, n=201, P=.001) and the COMLEX-USA Level 2-CE disciplines of family medicine (r=0.228, n=201, P=.001) and emergency medicine (r=0.220, n=201, P=.002). The MCAT biological sciences score was correlated with first-year (r=0.344, n=210, P<.001) and second-year (r=0.322, n=210, P<.001) medical school GPA; COMLEX-USA Level 1 total score (r=0.301, n=210, P<.001) and scores in the disciplines of physiology (r=0.334, n=210, P<.001), pharmacology (r=0.216, n=210, P=.002), and osteopathic principles and practice (r=0.273, n=210, P<.001); and COMLEX-USA Level 2-CE total score (r=0.253, n=201, P<0.001) and internal medicine discipline score (r=0.238, n=201, P=.001). Likewise, total MCAT score correlated with first-year (r=0.272, n=210, P<.001) and second-year (r=0.250, n=210, P<.001) medical school GPA; COMLEX-USA Level 1 total (r=0.296, n=210, P<.001), physiology (r=0.349, n=210, P<.001), pharmacology (r=0.214, n=210, P=.002), and osteopathic principles and practice (r=0.239, n=210, P<.001) scores; and COMLEX-USA Level 2-CE total (r=0.266, n=201, P<.001) and internal medicine (r=0.249, n=201, P<.001) scores. The MCAT physical sciences score and the average interview score did not have significant correlations with medical school GPA or COMLEX-USA Level 1 or Level 2-CE scores.
Results of the 2-tailed Pearson product-moment correlations comparing the prematriculation variables and each of the 7 osteopathic clinical competencies, as measured by the average score for each student across all third-year core rotations, are presented in
Table 2. If more than 1 preceptor, attending physician, or resident evaluated the student, the evaluation scores were averaged for that rotation. Average interview score had a small but statistically significant correlation with average medical knowledge competency (
r=0.233, n=209,
P=.01) score, as assessed on the student's core third-year rotations. No other variables compared in this analysis had significant correlations.
Results of the 2-tailed Pearson product-moment correlations for each of the prematriculation variables with the scored performance on the final OSCEs in the domains of history and physical examination skills, interpersonal skills, and the SOAP (subjective, objective, assessment, plan) note writing are presented in
Table 3. Each student performed at least 2 patient encounters, and the score in each area for each student was averaged across the patient encounters. No significant correlations with any of the prematriculation variables were found.
The association between the prematriculation variables and the proportion of students who failed the COMLEX-USA Level 2-PE and failed to match are presented in
Table 4. Only one significant relationship, between the biological sciences MCAT score and failing the COMLEX-USA Level 2-PE (
P=.009), was found.
Admission requirements for applicants to nearly all COMs include prerequisite courses and minimum undergraduate science or total GPA, MCAT scores, and interview scores. Some schools have adopted more holistic admission requirements that consider characteristics more often associated with humanities,
1,2 in the belief that these students are likely to have qualities that are desired by both medical schools and patients, including empathy, altruism, duty, and the ability to effectively use language to explain complex topics.
The current study examined whether the traditional admissions process used by MWU/AZCOM is associated with both quantitative and qualitative measures of medical student success. As with all single-site studies, our study is limited by the use of a single institution's admissions process to discover findings that can be generalizable to all schools. Another limitation is that, by nature, the process of selecting applicants for medical school cannot be randomized, as the process of selecting and educating medical students relies heavily on judgment, which may differ by institution. We have tried to increase the statistical rigor and generalizability of our results by using the Bonferroni adjustment to the significance tests of the correlation coefficients.
Several studies have found associations between a high level of performance on the United States Medical Licensing Examination (USMLE)—a 3-step, 4-examination series required for licensure of all allopathic medical students—and higher scores on the MCAT.
3,4 Scores on the MCAT have a direct, positive predictive value for students’ academic performance in allopathic medical schools, as measured by scores on the USMLE.
3,10-12 Other studies have found that low MCAT scores are associated with academic difficulties during the first 2 years of medical school,
3,5,13 which reflect the findings of the current study. However, when examining the relationship between traditional preadmission measures of MCAT and undergraduate GPA and clinical skill performance on the USMLE Step 2 Clinical Skills, Roberts et al
9 showed that preadmission measures of academic ability are not related to later clinical performance. Likewise, Baker et al
14 studied the MCAT and its predictive value for performance on the COMLEX-USA, but they found no significant correlation between students’ MCAT scores and subsequent performance on COMLEX-USA Level 1.
A review of the medical school admissions process conducted by Stratton et al
15 examined medical student academic performance during the first year and identified significantly increased academic risks among students who had lower undergraduate GPA, were aged 31 years or older, and were selected by nonunanimous admission committee votes. The authors concluded that a variety of factors underlie students’ underperformance during the first year of medical school, underscoring the complexity of predicting which students were at risk.
15 Interestingly, no aspect of the MCAT was noted to be significantly associated with students’ underperformance, though undergraduate science GPA showed a moderate and consistent negative relationship, with a lower science GPA increasing the likelihood of a student's appearance before a promotional committee.
Daugherty et al
16 found that higher levels of clerkship performance were associated with higher MCAT scores in allopathic medical students
, and similar findings were noted by Hojat et al
17 for osteopathic medical students. However, the results of the current study did not find this association. Swanson et al
12 demonstrated that the MCAT sore was more predictive of USMLE Step 1 performance than undergraduate science GPA. Although the data in the current study indicated more associations between MCAT discipline and total scores and COMLEX-USA discipline and total scores, we found that undergraduate total GPA was a significant predictor of COMLEX-USA Level 2-CE total score as well.
Similar to the findings of other studies,
1-4 our analysis showed that MCAT (especially total score and biological sciences score) and undergraduate GPA have small to moderate correlations with COMLEX-USA performance, and that neither are strong predictors of global academic performance. While Kulatunga-Moruzi and Norman
18 suggest that MCAT score is a better predictor of academic performance than undergraduate GPA, we did not find that to be the case. Like Dunleavy et al,
6 we conclude that the combination of undergraduate GPA and MCAT total scores are moderate predictors of academic performance in medical school. Similarly, although Julian
4 reported that MCAT scores were far better than undergraduate GPA in predicting performance on USMLE Step 1, 2, and 3, we found that undergraduate science GPA and undergraduate total GPA are weak predictors of COMLEX-USA performance, but so are MCAT and its component scores.
There has been limited research regarding the COM admissions interview process and its predictive value for student performance. To compare communication skills between students admitted to medical school with or without considering interview performance, Dahlin et al
7conducted a retrospective cohort study and concluded that selection through interview reduced the number of students who demonstrated inferior communication skills in the fourth year of medical school. Eva et al
8 found that interviews and letters of recommendation are less valid and reliable criteria for admission decisions than other prematriculation variables. Similarly, Basco et al
19 demonstrated that interview scores have limited predictive validity for performance on a fourth-year clinical examination, as well as limited, but statistically significant, correlation with interpersonal skills.
Some medical schools have developed a multiple mini-interview (MMI) process for admissions, in which candidates participate in a number of short, timed assessments of soft skills, such as teamwork, empathy, and communication. Jerant et al
20 concluded that MMI-based admissions did not disfavor underrepresented minority applicants, and they found that applicants with lower socioeconomic status were more likely to be invited to an MMI and recommended for acceptance than under the previous interview process. However, these studies did not relate MMI performance to academic or clinical performance. The value of the admission interviews is controversial, and the association between interview performance and medical school performance likely depends on the type of interview and the consistency and level of experience of the interviewers.
The lack of association between interview scores and any other measure of medical school performance was surprising and may reflect a desire on the part of the admissions committee to accept students whose skills were not limited to standard measures of academic performance. Alternatively, it may represent an overcompensation of conversational and interpersonal skills on the part of applicants, who may know their academic performance could be lacking. The lack of predictive ability of the interview may also represent limitations of the overall admissions process, in which individuals must quickly assess global applicant suitability for medical school and medical practice.
In reviewing the results of this study, members of the admissions committee at MWU/AZCOM suggested several changes to the application process, including fewer interviewers on each panel, because the value of interviews may not justify such a large investment of resources. The interview day was changed to include more interactive, hands-on activities with groups of current students and faculty. Interviewers were also trained to ask more clinically based interview questions. It is the hope that including more measures of soft skills will increase the correlation between interview performance and clinical and professional performance. Other changes in the structure of the admissions process may follow.
Our findings regarding traditional predictors of academic performance, undergraduate GPA, undergraduate science GPA, and MCAT scores as having small to moderate association with medical school grades and performance on COMLEX-USA Level 1 and Level 2-CE echo other studies. The lack of association between interview scores and any performance measure, other than the medical knowledge competency, also reflects the findings of other studies and requires additional research into the value of the interview and the availability of alternative admissions processes that allow better prediction and assessment of applicant performance. Finally, the weak overall relationship between standard prematriculation variables and academic and clinical performance in medical school as identified both in our study and studies regarding MMI-based admissions
8,19,20 is a call to COMs to develop and assess alternative methods to predict and assess the performance of applicants as medical students and as future physicians.
Dr Agahi provided contributions to the research concept and performed the literature review; Dr Speicher provided the analysis and interpretation of data and drafted the article; Dr Cisek acquired and processed the data for analysis; all authors developed the research design; all authors gave final approval of this version of the article to be published; and all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy and the integrity of any part of the work are appropriately investigated and resolved.