Association Between Undergraduate Performance Predictors and Academic and Clinical Performance of Osteopathic Medical Students

Farshad Agahi; Mark R. Speicher; Grace Cisek

doi:10.7556/jaoa.2018.025

Abstract

Context: Medical schools use a variety of preadmission indices to select potential students. These indices generally include undergraduate grade point average (GPA), Medical College Admission Test (MCAT) scores, and preadmission interviews.

Objective: To investigate whether the admission indices used by Midwestern University Arizona College of Osteopathic Medicine are associated with the academic and clinical performance of their students.

Methods: Associations between the prematriculation variables of undergraduate science GPA, undergraduate total GPA, MCAT component scores, and interview scores and the academic and clinical variables of the first- and second-year medical school GPA, Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 1 and Level 2-Cognitive Evaluation (CE) total and discipline scores, scores in clinical rotations for osteopathic competencies, COMLEX-USA Level 2-Performance Evaluation passage, and match status were evaluated. Two-tailed Pearson product-moment correlations with a Bonferroni adjustment were used to examine these relationships.

Results: The traditional predictors of science and total undergraduate GPA as well as total and component MCAT scores had small to moderate associations with first- and second-year GPA, as well as COMLEX-USA Level 1 and Level 2-CE total scores. Of all predictors, only the MCAT biological sciences score had a statistically significant correlation with failure of the COMLEX-USA Level 2-Performance Evaluation examination (P=.009). Average interview scores were associated only with the osteopathic competency of medical knowledge (r=0.233; n=209; P=.001), as assessed by clerkship preceptors. No predictors were associated with scores in objective structured clinical encounters or with failing to match to a residency position.

Conclusion: The data indicate that traditional predictors of academic performance (undergraduate GPA, undergraduate science GPA, and MCAT scores) have small to moderate association with medical school grades and performance on COMLEX-USA Level 1 and Level 2-CE. This finding requires additional research into the value of the interview in the medical school admissions process and the availability of alternatives that allow better prediction and assessment of applicant performance.

Selection of appropriate candidates for medical school is an important first step in shaping the future of medical care and the reputation of the medical school. The process developed by colleges of osteopathic medicine (COMs) to determine how applicants are selected should, in some ways, also predict the likelihood of the student's success in medical school, matching to the specialties of their choice, and pursuing a successful career in medicine.

Considerable research^1-5 has examined the value of quantitative prematriculation predictors of clinical and professional performance, such as undergraduate grade point averages (GPAs) and Medical College Admission Test (MCAT) scores for allopathic and osteopathic medical students, but less information is available regarding the value of the medical school admissions interview. As more focus is paid to the development of clinical and professional skills and assessment of the entrustability of medical school graduates,^6-8 determining a medical student's professionalism and “soft skills” (eg, communication, empathy) is linked to both residency performance and future professional standing.⁹

The aim of our study is to examine the associations between prematriculation variables (undergraduate science and total GPAs, component and total MCAT scores, and interview scores) and measures of academic, clinical, and professional performance (first- and second-year medical school GPAs, Comprehensive Osteopathic Medical Licensing Examination-USA [COMLEX-USA] Level 1 and Level 2 total and discipline scores, preceptor evaluations of students in the 7 osteopathic competencies, objective structured clinical encounter [OSCE] scores, and residency match outcomes) in students at the Midwestern University Arizona College of Osteopathic Medicine (MWU/AZCOM).

Methods

The Midwestern University Institutional Review Board reviewed and approved this study. Prematriculation undergraduate science GPA; total GPA; total and component MCAT sciences, physical sciences, verbal reasoning, and combined total numeric scores; and average admissions interview scores were retrieved for all members of the incoming MWU/AZCOM class of 2016. The semi-blinded interview process for MWU/AZCOM consists of 1 basic science faculty, 1 clinical science faculty, and 1 third- or fourth-year medical student. Interviewees were rated on a scale of 1 to 6 (1 being the worst score and 6 being the best) in the areas of professional motivation and preparedness, problem solving and critical thinking, personal qualities, and overall performance. These 4 ratings are combined to determine the total interview score.

In addition, first- and second-year GPA, COMLEX-USA Levels 1, 2-Cognitive Evaluation (CE), and 2-Performance Evaluation (PE) scores, competency domain scores from clinical clerkship rotations in the third year, and final scores on the OSCE at the end of third year were obtained for the same individuals. The third-year core rotations include 4-week rotations in cardiology, obstetrics and gynecology, pediatrics, psychiatry, and surgery, with 2 rotations each in family medicine and internal medicine. One internal medicine rotation is required to be done with a residency team; other rotations may be inpatient or outpatient and with an individual preceptor or a residency team.

In June 2016, residency match information was obtained for the MWU/AZCOM class of 2016—specifically, whether the student matched to a residency program accredited by the American Osteopathic Association or the Accreditation Council for Graduate Medical Education, whether the student found a position in the post-match Scramble (for programs accredited by the American Osteopathic Association) or the Supplemental Offer and Acceptance Program (for programs accredited by the Accreditation Council for Graduate Medical Education), or whether the student failed to find a postgraduate position. If a student matched during the National Matching Services or the National Resident Matching Program, that student was considered matched. If a student found a position during either of the postmatch programs or failed to find a position, we counted that student as failing to find a position in the match.

Measures of association between the prematriculation variables and each set of performance variables were calculated using 2-tailed Pearson product-moment correlation coefficients or Fisher exact test. Because of the number of correlations, Bonferroni adjustments were made to the significance tests, requiring P≤.002 for statistical significance. Because of the low number of students who failed the COMLEX-USA Level 2-PE and to match in the residency match, a 2-sided Fisher exact test was performed to test the association between the pre-matriculation variables and the proportion of students who failed the COMLEX-USA Level 2-PE and who failed to match.

Results

Information from 206 of 219 graduating students in the class of 2016 at MWU/AZCOM was included in the analysis. Table 1 shows the 2-tailed Pearson product-moment correlation coefficients for each of the prematriculation variables and the outcome variables of first- and second-year GPA and discipline and total scores on COMLEX-USA Level 1 and 2-CE. Undergraduate science GPA had small but statistically significant positive correlations with first-year (r=0.285, n=208, P<.001) and second-year (r=0.285, n=206, P<.001) medical school GPA. Undergraduate total GPA had moderate, statistically significant correlations with first-year (r=0.326, n=208, P<.001) and second-year (r=0.331, n=206, P<.001) medical school GPA and COMLEX-USA Level 2-CE total score (r=0.226, n=210, P=.001).

Table 1.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With First- and Second-Year GPA

	Undergraduate GPA^a		MCAT Score^a
Variable	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score^a
Mean (SD)	3.40 (0.278)	3.48 (0.242)	9.08 (1.60)	9.43 (1.66)	10.0 (1.33)	28.5 (3.04)	19.8 (2.32)
Medical School GPA
First-year GPA	0.285^b	0.326^b	0.071	0.160	0.344^b	0.272^b	−0.155
Second-year GPA	0.285^b	0.331^b	0.077	0.127	0.322^b	0.250^b	−0.138
COMLEX-USA Level 1
Total score	0.107	0.556	0.134	0.175	0.301^b	0.296^b	0.016
Anatomy	0.133	0.447	−0.005	0.163	0.188	0.168	−0.40
Physiology	0.038	0.367	0.188	0.200	0.334^b	0.349^b	−0.063
Biochemistry	0.044	0.347	0.024	0.146	0.170	0.166	−0.043
Pharmacology	0.107	0.395	0.099	0.127	0.216^c	0.214^c	0.047
Pathology	0.013	0.360	0.115	0.098	0.198	0.204	0.008
Microbiology	0.103	0.401	0.077	0.125	0.137	0.164	0.008
Behavioral sciences	0.135	0.162	0.110	0.048	0.022	0.101	0.168
OPP	0.084	0.484	0.133	0.091	0.273^b	0.239^b	0.009
COMLEX-USA Level 2-CE
Total score	0.170	0.226^b	0.228^b	0.069	0.253^b	0.266^b	0.058
Surgery	−0.002	0.290	0.150	0.021	0.120	0.137	0.007
OB/GYN	0.088	0.217	0.097	0.048	0.147	0.146	0.126
Psychiatry	−0.065	0.095	0.160	0.009	0.089	0.127	0.179
Family medicine	0.109	0.338	0.228^b	0.005	0.195	0.206	0.001
Pediatrics	0.134	0.292	0.013	0.013	0.140	0.078	−0.045
Internal medicine	0.162	0.433	0.192	0.090	0.238^b	0.249^b	0.084
Emergency medicine	0.057	0.240	0.220^c	0.019	0.060	0.157	0.004
OPP	0.180	0.338	0.140	0.080	0.181	0.192	0.071

^a The maximum grade point average (GPA) is 4.0; the maximum Medical College Admission Test (MCAT) score is 45 (maximum of 15 for each component); and the maximum interview score is 24.

^b Statistically significant at P≤.001.

^c Statistically significant at P≤.002.

Abbreviations: CE, Cognitive Evaluation; COMLEX-USA, Comprehensive Osteopathic Medical Licensure Examination-USA; OB/GYN, obstetrics and gynecology; OPP, osteopathic principles and practice.

View Large

Table 1.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With First- and Second-Year GPA

	Undergraduate GPA^a		MCAT Score^a
Variable	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score^a
Mean (SD)	3.40 (0.278)	3.48 (0.242)	9.08 (1.60)	9.43 (1.66)	10.0 (1.33)	28.5 (3.04)	19.8 (2.32)
Medical School GPA
First-year GPA	0.285^b	0.326^b	0.071	0.160	0.344^b	0.272^b	−0.155
Second-year GPA	0.285^b	0.331^b	0.077	0.127	0.322^b	0.250^b	−0.138
COMLEX-USA Level 1
Total score	0.107	0.556	0.134	0.175	0.301^b	0.296^b	0.016
Anatomy	0.133	0.447	−0.005	0.163	0.188	0.168	−0.40
Physiology	0.038	0.367	0.188	0.200	0.334^b	0.349^b	−0.063
Biochemistry	0.044	0.347	0.024	0.146	0.170	0.166	−0.043
Pharmacology	0.107	0.395	0.099	0.127	0.216^c	0.214^c	0.047
Pathology	0.013	0.360	0.115	0.098	0.198	0.204	0.008
Microbiology	0.103	0.401	0.077	0.125	0.137	0.164	0.008
Behavioral sciences	0.135	0.162	0.110	0.048	0.022	0.101	0.168
OPP	0.084	0.484	0.133	0.091	0.273^b	0.239^b	0.009
COMLEX-USA Level 2-CE
Total score	0.170	0.226^b	0.228^b	0.069	0.253^b	0.266^b	0.058
Surgery	−0.002	0.290	0.150	0.021	0.120	0.137	0.007
OB/GYN	0.088	0.217	0.097	0.048	0.147	0.146	0.126
Psychiatry	−0.065	0.095	0.160	0.009	0.089	0.127	0.179
Family medicine	0.109	0.338	0.228^b	0.005	0.195	0.206	0.001
Pediatrics	0.134	0.292	0.013	0.013	0.140	0.078	−0.045
Internal medicine	0.162	0.433	0.192	0.090	0.238^b	0.249^b	0.084
Emergency medicine	0.057	0.240	0.220^c	0.019	0.060	0.157	0.004
OPP	0.180	0.338	0.140	0.080	0.181	0.192	0.071

^a The maximum grade point average (GPA) is 4.0; the maximum Medical College Admission Test (MCAT) score is 45 (maximum of 15 for each component); and the maximum interview score is 24.

^b Statistically significant at P≤.001.

^c Statistically significant at P≤.002.

Abbreviations: CE, Cognitive Evaluation; COMLEX-USA, Comprehensive Osteopathic Medical Licensure Examination-USA; OB/GYN, obstetrics and gynecology; OPP, osteopathic principles and practice.

Of the MCAT component scores, verbal reasoning had small but statistically significant correlations with COMLEX-USA Level 2-CE total score (r=0.228, n=201, P=.001) and the COMLEX-USA Level 2-CE disciplines of family medicine (r=0.228, n=201, P=.001) and emergency medicine (r=0.220, n=201, P=.002). The MCAT biological sciences score was correlated with first-year (r=0.344, n=210, P<.001) and second-year (r=0.322, n=210, P<.001) medical school GPA; COMLEX-USA Level 1 total score (r=0.301, n=210, P<.001) and scores in the disciplines of physiology (r=0.334, n=210, P<.001), pharmacology (r=0.216, n=210, P=.002), and osteopathic principles and practice (r=0.273, n=210, P<.001); and COMLEX-USA Level 2-CE total score (r=0.253, n=201, P<0.001) and internal medicine discipline score (r=0.238, n=201, P=.001). Likewise, total MCAT score correlated with first-year (r=0.272, n=210, P<.001) and second-year (r=0.250, n=210, P<.001) medical school GPA; COMLEX-USA Level 1 total (r=0.296, n=210, P<.001), physiology (r=0.349, n=210, P<.001), pharmacology (r=0.214, n=210, P=.002), and osteopathic principles and practice (r=0.239, n=210, P<.001) scores; and COMLEX-USA Level 2-CE total (r=0.266, n=201, P<.001) and internal medicine (r=0.249, n=201, P<.001) scores. The MCAT physical sciences score and the average interview score did not have significant correlations with medical school GPA or COMLEX-USA Level 1 or Level 2-CE scores.

Results of the 2-tailed Pearson product-moment correlations comparing the prematriculation variables and each of the 7 osteopathic clinical competencies, as measured by the average score for each student across all third-year core rotations, are presented in Table 2. If more than 1 preceptor, attending physician, or resident evaluated the student, the evaluation scores were averaged for that rotation. Average interview score had a small but statistically significant correlation with average medical knowledge competency (r=0.233, n=209, P=.01) score, as assessed on the student's core third-year rotations. No other variables compared in this analysis had significant correlations.

Table 2.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With Third-Year Core Rotation Competencies as Evaluated by Preceptors

	Undergraduate GPA		MCAT Score
Core Rotation Competency	Science	Total	Verbal Reasoning	Physical Reasoning	Biological Sciences	Total	Interview Score
Professionalism	0.005	0.055	0.056	-0.009	0.055	0.048	0.05
Patient care	−0.115	−0.026	0.007	−0.033	−0.025	−0.027	0.175
Medical knowledge	−0.027	0.032	0.009	0.044	0.162	0.095	0.233^a
Interpersonal and communication skills	−0.056	−0.005	−0.028	−0.032	0.096	0.014	0.131
Practice-based learning and improvement	0.029	0.087	−0.023	0.082	0.204	0.121	0.136
Systems-based practice	0.023	0.068	−0.06	−0.036	0.08	−0.016	0.165
Osteopathic medicine	−0.112	−0.045	0.023	−0.124	−0.053	−0.079	0.039
Overall preceptor evaluation score	−0.039	0.037	−0.007	−0.005	0.111	0.041	0.190

^a Statistically significant at P≤.001.

Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test.

View Large

Table 2.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With Third-Year Core Rotation Competencies as Evaluated by Preceptors

	Undergraduate GPA		MCAT Score
Core Rotation Competency	Science	Total	Verbal Reasoning	Physical Reasoning	Biological Sciences	Total	Interview Score
Professionalism	0.005	0.055	0.056	-0.009	0.055	0.048	0.05
Patient care	−0.115	−0.026	0.007	−0.033	−0.025	−0.027	0.175
Medical knowledge	−0.027	0.032	0.009	0.044	0.162	0.095	0.233^a
Interpersonal and communication skills	−0.056	−0.005	−0.028	−0.032	0.096	0.014	0.131
Practice-based learning and improvement	0.029	0.087	−0.023	0.082	0.204	0.121	0.136
Systems-based practice	0.023	0.068	−0.06	−0.036	0.08	−0.016	0.165
Osteopathic medicine	−0.112	−0.045	0.023	−0.124	−0.053	−0.079	0.039
Overall preceptor evaluation score	−0.039	0.037	−0.007	−0.005	0.111	0.041	0.190

^a Statistically significant at P≤.001.

Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test.

Results of the 2-tailed Pearson product-moment correlations for each of the prematriculation variables with the scored performance on the final OSCEs in the domains of history and physical examination skills, interpersonal skills, and the SOAP (subjective, objective, assessment, plan) note writing are presented in Table 3. Each student performed at least 2 patient encounters, and the score in each area for each student was averaged across the patient encounters. No significant correlations with any of the prematriculation variables were found.

Table 3.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With Average Third-Year Final OSCE Scores

	Undergraduate GPA		MCAT Score
OSCE Component	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score
History and physical examination skills	0.057	0.117	0.074	0.032	0.067	0.094	0.018
Interpersonal skills	0.033	0.054	0.014	−0.035	−0.040	−0.027	0.044
Subjective, objective, assessment, plan note	0.050	0.108	0.080	−0.039	0.063	0.048	0.065

Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test; OSCE, objective structured clinical encounter.

View Large

Table 3.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With Average Third-Year Final OSCE Scores

	Undergraduate GPA		MCAT Score
OSCE Component	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score
History and physical examination skills	0.057	0.117	0.074	0.032	0.067	0.094	0.018
Interpersonal skills	0.033	0.054	0.014	−0.035	−0.040	−0.027	0.044
Subjective, objective, assessment, plan note	0.050	0.108	0.080	−0.039	0.063	0.048	0.065

Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test; OSCE, objective structured clinical encounter.

The association between the prematriculation variables and the proportion of students who failed the COMLEX-USA Level 2-PE and failed to match are presented in Table 4. Only one significant relationship, between the biological sciences MCAT score and failing the COMLEX-USA Level 2-PE (P=.009), was found.

Table 4.

Fisher Exact Test of Relationships Between Prematriculation Variables and Failure to Pass the COMLEX-USA Level 2-PE and Failure to Match

	Undergraduate GPA		MCAT Score
Failure to…	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score
Pass COMLEX-USA Level 2-PE	50.991	91.069	10.691	2.793	20.336^a	23.439	22.260
Match	214.669	164.023	10.232	19.450	17.728	27.179	55.797

^a Statistically significant at P≤.05.

Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensure Examination-USA; GPA, grade point average; MCAT, Medical College Admission Test; PE, Performance Evaluation.

View Large

Table 4.

Fisher Exact Test of Relationships Between Prematriculation Variables and Failure to Pass the COMLEX-USA Level 2-PE and Failure to Match

	Undergraduate GPA		MCAT Score
Failure to…	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score
Pass COMLEX-USA Level 2-PE	50.991	91.069	10.691	2.793	20.336^a	23.439	22.260
Match	214.669	164.023	10.232	19.450	17.728	27.179	55.797

^a Statistically significant at P≤.05.

Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensure Examination-USA; GPA, grade point average; MCAT, Medical College Admission Test; PE, Performance Evaluation.

Discussion

Admission requirements for applicants to nearly all COMs include prerequisite courses and minimum undergraduate science or total GPA, MCAT scores, and interview scores. Some schools have adopted more holistic admission requirements that consider characteristics more often associated with humanities,^1,2 in the belief that these students are likely to have qualities that are desired by both medical schools and patients, including empathy, altruism, duty, and the ability to effectively use language to explain complex topics.

The current study examined whether the traditional admissions process used by MWU/AZCOM is associated with both quantitative and qualitative measures of medical student success. As with all single-site studies, our study is limited by the use of a single institution's admissions process to discover findings that can be generalizable to all schools. Another limitation is that, by nature, the process of selecting applicants for medical school cannot be randomized, as the process of selecting and educating medical students relies heavily on judgment, which may differ by institution. We have tried to increase the statistical rigor and generalizability of our results by using the Bonferroni adjustment to the significance tests of the correlation coefficients.

Several studies have found associations between a high level of performance on the United States Medical Licensing Examination (USMLE)—a 3-step, 4-examination series required for licensure of all allopathic medical students—and higher scores on the MCAT.^3,4 Scores on the MCAT have a direct, positive predictive value for students’ academic performance in allopathic medical schools, as measured by scores on the USMLE.^3,10-12 Other studies have found that low MCAT scores are associated with academic difficulties during the first 2 years of medical school,^3,5,13 which reflect the findings of the current study. However, when examining the relationship between traditional preadmission measures of MCAT and undergraduate GPA and clinical skill performance on the USMLE Step 2 Clinical Skills, Roberts et al⁹ showed that preadmission measures of academic ability are not related to later clinical performance. Likewise, Baker et al¹⁴ studied the MCAT and its predictive value for performance on the COMLEX-USA, but they found no significant correlation between students’ MCAT scores and subsequent performance on COMLEX-USA Level 1.

A review of the medical school admissions process conducted by Stratton et al¹⁵ examined medical student academic performance during the first year and identified significantly increased academic risks among students who had lower undergraduate GPA, were aged 31 years or older, and were selected by nonunanimous admission committee votes. The authors concluded that a variety of factors underlie students’ underperformance during the first year of medical school, underscoring the complexity of predicting which students were at risk.¹⁵ Interestingly, no aspect of the MCAT was noted to be significantly associated with students’ underperformance, though undergraduate science GPA showed a moderate and consistent negative relationship, with a lower science GPA increasing the likelihood of a student's appearance before a promotional committee.

Daugherty et al¹⁶ found that higher levels of clerkship performance were associated with higher MCAT scores in allopathic medical students, and similar findings were noted by Hojat et al¹⁷ for osteopathic medical students. However, the results of the current study did not find this association. Swanson et al¹² demonstrated that the MCAT sore was more predictive of USMLE Step 1 performance than undergraduate science GPA. Although the data in the current study indicated more associations between MCAT discipline and total scores and COMLEX-USA discipline and total scores, we found that undergraduate total GPA was a significant predictor of COMLEX-USA Level 2-CE total score as well.

Similar to the findings of other studies,^1-4 our analysis showed that MCAT (especially total score and biological sciences score) and undergraduate GPA have small to moderate correlations with COMLEX-USA performance, and that neither are strong predictors of global academic performance. While Kulatunga-Moruzi and Norman¹⁸ suggest that MCAT score is a better predictor of academic performance than undergraduate GPA, we did not find that to be the case. Like Dunleavy et al,⁶ we conclude that the combination of undergraduate GPA and MCAT total scores are moderate predictors of academic performance in medical school. Similarly, although Julian⁴ reported that MCAT scores were far better than undergraduate GPA in predicting performance on USMLE Step 1, 2, and 3, we found that undergraduate science GPA and undergraduate total GPA are weak predictors of COMLEX-USA performance, but so are MCAT and its component scores.

There has been limited research regarding the COM admissions interview process and its predictive value for student performance. To compare communication skills between students admitted to medical school with or without considering interview performance, Dahlin et al⁷conducted a retrospective cohort study and concluded that selection through interview reduced the number of students who demonstrated inferior communication skills in the fourth year of medical school. Eva et al⁸ found that interviews and letters of recommendation are less valid and reliable criteria for admission decisions than other prematriculation variables. Similarly, Basco et al¹⁹ demonstrated that interview scores have limited predictive validity for performance on a fourth-year clinical examination, as well as limited, but statistically significant, correlation with interpersonal skills.

Some medical schools have developed a multiple mini-interview (MMI) process for admissions, in which candidates participate in a number of short, timed assessments of soft skills, such as teamwork, empathy, and communication. Jerant et al²⁰ concluded that MMI-based admissions did not disfavor underrepresented minority applicants, and they found that applicants with lower socioeconomic status were more likely to be invited to an MMI and recommended for acceptance than under the previous interview process. However, these studies did not relate MMI performance to academic or clinical performance. The value of the admission interviews is controversial, and the association between interview performance and medical school performance likely depends on the type of interview and the consistency and level of experience of the interviewers.

The lack of association between interview scores and any other measure of medical school performance was surprising and may reflect a desire on the part of the admissions committee to accept students whose skills were not limited to standard measures of academic performance. Alternatively, it may represent an overcompensation of conversational and interpersonal skills on the part of applicants, who may know their academic performance could be lacking. The lack of predictive ability of the interview may also represent limitations of the overall admissions process, in which individuals must quickly assess global applicant suitability for medical school and medical practice.

Modifications at MWU/AZCOM

In reviewing the results of this study, members of the admissions committee at MWU/AZCOM suggested several changes to the application process, including fewer interviewers on each panel, because the value of interviews may not justify such a large investment of resources. The interview day was changed to include more interactive, hands-on activities with groups of current students and faculty. Interviewers were also trained to ask more clinically based interview questions. It is the hope that including more measures of soft skills will increase the correlation between interview performance and clinical and professional performance. Other changes in the structure of the admissions process may follow.

Conclusion

Our findings regarding traditional predictors of academic performance, undergraduate GPA, undergraduate science GPA, and MCAT scores as having small to moderate association with medical school grades and performance on COMLEX-USA Level 1 and Level 2-CE echo other studies. The lack of association between interview scores and any performance measure, other than the medical knowledge competency, also reflects the findings of other studies and requires additional research into the value of the interview and the availability of alternative admissions processes that allow better prediction and assessment of applicant performance. Finally, the weak overall relationship between standard prematriculation variables and academic and clinical performance in medical school as identified both in our study and studies regarding MMI-based admissions^8,19,20 is a call to COMs to develop and assess alternative methods to predict and assess the performance of applicants as medical students and as future physicians.

Author Contributions

Dr Agahi provided contributions to the research concept and performed the literature review; Dr Speicher provided the analysis and interpretation of data and drafted the article; Dr Cisek acquired and processed the data for analysis; all authors developed the research design; all authors gave final approval of this version of the article to be published; and all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy and the integrity of any part of the work are appropriately investigated and resolved.

References

1.

Muller D, Kase N. Challenging traditional premedical requirements as predictors of success in medical school: the Mount Sinai School of Medicine Humanities and Medicine Program. Acad Med. 2010;85:1378-1383. doi: 10.1097/ACM.0b013e3181dbf22a [CrossRef] [PubMed]

2.

Hall JN, Woods N, Hanson MD. Is social sciences and humanities (SSH) premedical education marginalized in the medical school admission process? a review and contextualization of the literature. Acad Med. 2014;89(7):1075-1086. doi: 10.1097/ACM.0000000000000284 [CrossRef] [PubMed]

3.

Wiley A, Koenig JA. The validity of the Medical College Admission Test for predicting performance in the first two years of medical school. Acad Med. 1996;71(10 suppl):S83-S85. [CrossRef] [PubMed]

4.

Julian ER. Validity of the Medical College Admission Test for predicting medical school performance. Acad Med. 2005;80:910-917. [CrossRef] [PubMed]

5.

Huff KL, Fang D. When are students most at risk of encountering academic difficulty? a study of 1992 matriculates to U.S. medical schools. Acad Med. 1999;74:454-460. [CrossRef]

6.

Dunleavy DM, Kroopnick MH, Dowd KW, Searcy CA, Zhao X. The predictive validity of the MCAT exam in relation to academic performance through medical school: a national cohort study of 2001-2004 matriculates. Acad Med. 2013;88(5):666-671. doi: 10.1097/ACM.0b013e3182864299 [CrossRef] [PubMed]

7.

Dahlin M, Söderberg S, Holm U, Nilsson I, Farnebo LO. Comparison of communication skills between medical students admitted after interviews or on academic merits. BMC Med Educ. 2012;12:46. doi: 10.1186/1472-6920-12-46 [CrossRef] [PubMed]

8.

Eva KW, Reiter HI, Rosenfeld J, Norman GR. The relationship between interviewers: characteristics and ratings assigned during a mini-interview. J Assoc Am Med Coll. 2004;79(6):602-609. [CrossRef]

9.

Roberts WL, Pugliano G, Langenau E, Boulet JR. Modeling relationships between traditional preadmission measures and clinical skills performance on a medical licensure examination. Adv Health Sci Educ Theory Pract. 2012;17(3):403-417. doi: 10.1007/s10459-011-9321-4 [CrossRef] [PubMed]

10.

Basco WTJr, Way DP, Gilbert GE, Hudson A. Undergraduate institutional MCAT scores as predictors of USMLE step 1 performance. Acad Med. 2002;77(10 suppl):S13-S16. [CrossRef] [PubMed]

11.

Kumar K, Nguyen H, Indurkhya A. Relationship between pathology curriculum approaches and performance in the United States Medical Licensing Examination (USMLE), Step 1: a national cross-sectional study. Hum Pathol. 2003;34:417-422. [CrossRef] [PubMed]

12.

Swanson DB, Case SM, Koenig J, Killian CD. Preliminary study of the accuracies of the old and new medical college admissions tests for predicting performance on USMLE Step 1. Acad Med. 1996;71(1 suppl):S25-S27. [CrossRef] [PubMed]

13.

Blue AV, Gilbert GE, Elam CL, Basco WTJr. Does institutional selectivity aid in the prediction of medical school performance? Acad Med. 2000;75(10 suppl):S31-S33. [CrossRef] [PubMed]

14.

Baker HH, Cope MK, Fisk R, Gorby JN, Foster RW. Relationship of preadmission variables and first- and second-year course performance on the National Board of Osteopathic Medical Examiners’ COMLEX-USA Level 1 examination. J Am Osteopath Assoc. 2000;100(3):153-161. [PubMed]

15.

Stratton TD, Elam CL. A holistic review of the medical school admission process: examining correlates of academic underperformance. Med Educ Online. 2014;19:2922919. doi: 10.3402/meo.v19.22919 [CrossRef]

16.

Daugherty SR, Nora LM, Schmidt JL, Goodman LJ. Identifying poor preclinical performers who do well in clerkships. Acad Med. 1992;67(10 suppl):S72-S74. [CrossRef] [PubMed]

17.

Hojat M, Erdmann JB, Veloski JJ, et al. A validity study of the writing sample section of the medical college admission test. Acad Med. 2000;75(10 suppl):S25-S27. [CrossRef] [PubMed]

18.

Kulatunga-Moruzi C, Norman GR. Validity of admissions measures in predicting performance outcomes: the contribution of cognitive and non-cognitive dimensions. Teach Learn Med. 2002;14(1):34-42. [CrossRef] [PubMed]

19.

Basco WTJr, Lancaster CJ, Gilbert GE, Carey ME, Blur AV. Medical school application interview score has limited predictive validity for performance on a fourth year clinical practice examination. Adv Health Sci Educ. 2008;13(2):151-162. [CrossRef]

20.

Jerant A, Fancher T, Fenton JJ, et al. How medical school applicant, race, ethnicity, and socioeconomic status relate to multiple mini-interview-based admission outcomes: findings from one medical school. Acad Med. 2015;90(12):1667-1674. doi: 10.1097/ACM.0000000000000766 [CrossRef] [PubMed]

Table 1.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With First- and Second-Year GPA

	Undergraduate GPA^a		MCAT Score^a
Variable	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score^a
Mean (SD)	3.40 (0.278)	3.48 (0.242)	9.08 (1.60)	9.43 (1.66)	10.0 (1.33)	28.5 (3.04)	19.8 (2.32)
Medical School GPA
First-year GPA	0.285^b	0.326^b	0.071	0.160	0.344^b	0.272^b	−0.155
Second-year GPA	0.285^b	0.331^b	0.077	0.127	0.322^b	0.250^b	−0.138
COMLEX-USA Level 1
Total score	0.107	0.556	0.134	0.175	0.301^b	0.296^b	0.016
Anatomy	0.133	0.447	−0.005	0.163	0.188	0.168	−0.40
Physiology	0.038	0.367	0.188	0.200	0.334^b	0.349^b	−0.063
Biochemistry	0.044	0.347	0.024	0.146	0.170	0.166	−0.043
Pharmacology	0.107	0.395	0.099	0.127	0.216^c	0.214^c	0.047
Pathology	0.013	0.360	0.115	0.098	0.198	0.204	0.008
Microbiology	0.103	0.401	0.077	0.125	0.137	0.164	0.008
Behavioral sciences	0.135	0.162	0.110	0.048	0.022	0.101	0.168
OPP	0.084	0.484	0.133	0.091	0.273^b	0.239^b	0.009
COMLEX-USA Level 2-CE
Total score	0.170	0.226^b	0.228^b	0.069	0.253^b	0.266^b	0.058
Surgery	−0.002	0.290	0.150	0.021	0.120	0.137	0.007
OB/GYN	0.088	0.217	0.097	0.048	0.147	0.146	0.126
Psychiatry	−0.065	0.095	0.160	0.009	0.089	0.127	0.179
Family medicine	0.109	0.338	0.228^b	0.005	0.195	0.206	0.001
Pediatrics	0.134	0.292	0.013	0.013	0.140	0.078	−0.045
Internal medicine	0.162	0.433	0.192	0.090	0.238^b	0.249^b	0.084
Emergency medicine	0.057	0.240	0.220^c	0.019	0.060	0.157	0.004
OPP	0.180	0.338	0.140	0.080	0.181	0.192	0.071

^a The maximum grade point average (GPA) is 4.0; the maximum Medical College Admission Test (MCAT) score is 45 (maximum of 15 for each component); and the maximum interview score is 24.

^b Statistically significant at P≤.001.

^c Statistically significant at P≤.002.

Abbreviations: CE, Cognitive Evaluation; COMLEX-USA, Comprehensive Osteopathic Medical Licensure Examination-USA; OB/GYN, obstetrics and gynecology; OPP, osteopathic principles and practice.

View Large

Table 1.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With First- and Second-Year GPA

	Undergraduate GPA^a		MCAT Score^a
Variable	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score^a
Mean (SD)	3.40 (0.278)	3.48 (0.242)	9.08 (1.60)	9.43 (1.66)	10.0 (1.33)	28.5 (3.04)	19.8 (2.32)
Medical School GPA
First-year GPA	0.285^b	0.326^b	0.071	0.160	0.344^b	0.272^b	−0.155
Second-year GPA	0.285^b	0.331^b	0.077	0.127	0.322^b	0.250^b	−0.138
COMLEX-USA Level 1
Total score	0.107	0.556	0.134	0.175	0.301^b	0.296^b	0.016
Anatomy	0.133	0.447	−0.005	0.163	0.188	0.168	−0.40
Physiology	0.038	0.367	0.188	0.200	0.334^b	0.349^b	−0.063
Biochemistry	0.044	0.347	0.024	0.146	0.170	0.166	−0.043
Pharmacology	0.107	0.395	0.099	0.127	0.216^c	0.214^c	0.047
Pathology	0.013	0.360	0.115	0.098	0.198	0.204	0.008
Microbiology	0.103	0.401	0.077	0.125	0.137	0.164	0.008
Behavioral sciences	0.135	0.162	0.110	0.048	0.022	0.101	0.168
OPP	0.084	0.484	0.133	0.091	0.273^b	0.239^b	0.009
COMLEX-USA Level 2-CE
Total score	0.170	0.226^b	0.228^b	0.069	0.253^b	0.266^b	0.058
Surgery	−0.002	0.290	0.150	0.021	0.120	0.137	0.007
OB/GYN	0.088	0.217	0.097	0.048	0.147	0.146	0.126
Psychiatry	−0.065	0.095	0.160	0.009	0.089	0.127	0.179
Family medicine	0.109	0.338	0.228^b	0.005	0.195	0.206	0.001
Pediatrics	0.134	0.292	0.013	0.013	0.140	0.078	−0.045
Internal medicine	0.162	0.433	0.192	0.090	0.238^b	0.249^b	0.084
Emergency medicine	0.057	0.240	0.220^c	0.019	0.060	0.157	0.004
OPP	0.180	0.338	0.140	0.080	0.181	0.192	0.071

^a The maximum grade point average (GPA) is 4.0; the maximum Medical College Admission Test (MCAT) score is 45 (maximum of 15 for each component); and the maximum interview score is 24.

^b Statistically significant at P≤.001.

^c Statistically significant at P≤.002.

Abbreviations: CE, Cognitive Evaluation; COMLEX-USA, Comprehensive Osteopathic Medical Licensure Examination-USA; OB/GYN, obstetrics and gynecology; OPP, osteopathic principles and practice.

Table 2.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With Third-Year Core Rotation Competencies as Evaluated by Preceptors

	Undergraduate GPA		MCAT Score
Core Rotation Competency	Science	Total	Verbal Reasoning	Physical Reasoning	Biological Sciences	Total	Interview Score
Professionalism	0.005	0.055	0.056	-0.009	0.055	0.048	0.05
Patient care	−0.115	−0.026	0.007	−0.033	−0.025	−0.027	0.175
Medical knowledge	−0.027	0.032	0.009	0.044	0.162	0.095	0.233^a
Interpersonal and communication skills	−0.056	−0.005	−0.028	−0.032	0.096	0.014	0.131
Practice-based learning and improvement	0.029	0.087	−0.023	0.082	0.204	0.121	0.136
Systems-based practice	0.023	0.068	−0.06	−0.036	0.08	−0.016	0.165
Osteopathic medicine	−0.112	−0.045	0.023	−0.124	−0.053	−0.079	0.039
Overall preceptor evaluation score	−0.039	0.037	−0.007	−0.005	0.111	0.041	0.190

^a Statistically significant at P≤.001.

Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test.

View Large

Table 2.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With Third-Year Core Rotation Competencies as Evaluated by Preceptors

	Undergraduate GPA		MCAT Score
Core Rotation Competency	Science	Total	Verbal Reasoning	Physical Reasoning	Biological Sciences	Total	Interview Score
Professionalism	0.005	0.055	0.056	-0.009	0.055	0.048	0.05
Patient care	−0.115	−0.026	0.007	−0.033	−0.025	−0.027	0.175
Medical knowledge	−0.027	0.032	0.009	0.044	0.162	0.095	0.233^a
Interpersonal and communication skills	−0.056	−0.005	−0.028	−0.032	0.096	0.014	0.131
Practice-based learning and improvement	0.029	0.087	−0.023	0.082	0.204	0.121	0.136
Systems-based practice	0.023	0.068	−0.06	−0.036	0.08	−0.016	0.165
Osteopathic medicine	−0.112	−0.045	0.023	−0.124	−0.053	−0.079	0.039
Overall preceptor evaluation score	−0.039	0.037	−0.007	−0.005	0.111	0.041	0.190

^a Statistically significant at P≤.001.

Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test.

Table 3.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With Average Third-Year Final OSCE Scores

	Undergraduate GPA		MCAT Score
OSCE Component	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score
History and physical examination skills	0.057	0.117	0.074	0.032	0.067	0.094	0.018
Interpersonal skills	0.033	0.054	0.014	−0.035	−0.040	−0.027	0.044
Subjective, objective, assessment, plan note	0.050	0.108	0.080	−0.039	0.063	0.048	0.065

Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test; OSCE, objective structured clinical encounter.

View Large

Table 3.

2-Tailed Pearson Product-Moment Correlation Coefficients for Prematriculation Variables Correlated With Average Third-Year Final OSCE Scores

	Undergraduate GPA		MCAT Score
OSCE Component	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score
History and physical examination skills	0.057	0.117	0.074	0.032	0.067	0.094	0.018
Interpersonal skills	0.033	0.054	0.014	−0.035	−0.040	−0.027	0.044
Subjective, objective, assessment, plan note	0.050	0.108	0.080	−0.039	0.063	0.048	0.065

Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test; OSCE, objective structured clinical encounter.

Table 4.

Fisher Exact Test of Relationships Between Prematriculation Variables and Failure to Pass the COMLEX-USA Level 2-PE and Failure to Match

	Undergraduate GPA		MCAT Score
Failure to…	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score
Pass COMLEX-USA Level 2-PE	50.991	91.069	10.691	2.793	20.336^a	23.439	22.260
Match	214.669	164.023	10.232	19.450	17.728	27.179	55.797

^a Statistically significant at P≤.05.

Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensure Examination-USA; GPA, grade point average; MCAT, Medical College Admission Test; PE, Performance Evaluation.

View Large

Table 4.

Fisher Exact Test of Relationships Between Prematriculation Variables and Failure to Pass the COMLEX-USA Level 2-PE and Failure to Match

	Undergraduate GPA		MCAT Score
Failure to…	Science	Total	Verbal Reasoning	Physical Sciences	Biological Sciences	Total	Interview Score
Pass COMLEX-USA Level 2-PE	50.991	91.069	10.691	2.793	20.336^a	23.439	22.260
Match	214.669	164.023	10.232	19.450	17.728	27.179	55.797

^a Statistically significant at P≤.05.

Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensure Examination-USA; GPA, grade point average; MCAT, Medical College Admission Test; PE, Performance Evaluation.

Association Between Undergraduate Performance Predictors and Academic and Clinical Performance of Osteopathic Medical Students

Related content

Related Topics