Abstract
Context: New accreditation standards require that all US colleges of osteopathic medicine (COMs) publically report the first-time pass rates of graduates on the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 3. Little is known about the extent to which admissions variables or COM performance measures relate to Level 3 performance.
Objective: To examine the relationship of admissions variables and COM performance to scores on Level 3 and to assess whether a relationship existed between Level 3 scores and sex, curriculum track, year of graduation, and residency specialty in the first postgraduate year.
Methods: Data were analyzed from 4 graduating classes (2008-2011) of the West Virginia School of Osteopathic Medicine in Lewisburg. Relationships were examined between first-attempt scores on COMLEX-USA Level 3 and Medical College Admission Test (MCAT) scores; undergraduate grade point averages (GPAs); GPAs in COM year 1, year 2, and clinical rotation years (years 3 and 4); and first-attempt scores on COMLEX-USA Level 1, Level 2-Cognitive Evaluation, and Level 2-Performance Evaluation.
Results: Of the 556 graduates during this 4-year period, COMLEX-USA Level 3 scores were available for 552 graduates (99.3%). No statistically significant differences were found in Level 3 scores based on sex, curriculum track, graduating class, or residency specialty. The strongest relationship between Level 3 scores and any admissions variable was with total MCAT score, which accounted for 4.2% of the variation in Level 3 scores. The strongest relationship between Level 3 scores and COM year performance measures was with year 2 GPA, which accounted for 35.4% of the variation in Level 3 scores. Level 1 scores accounted for 38.5% of the variation in Level 3 scores, and Level 2-Cognitive Evaluation scores accounted for the greatest percentage of variation (45.7%). The correlation of Level 3 scores with passing the Level 2-Performance Evaluation on the first attempt was not statistically significant.
Conclusion: A weak relationship was found between admissions variables and performance on COMLEX-USA Level 3, suggesting that graduates with lower MCAT scores and undergraduate GPAs may have overcome their early disadvantage. Strong relationships were found between Level 3 scores and year 2 GPAs, as well as scores on COMLEX-USA Level 1 and Level 2-Cognitive Evaluation.
Colleges of osteopathic medicine (COMs) accredited by the American Osteopathic Association Commission on Osteopathic College Accreditation (AOA COCA) have long been required to maintain “a longitudinal record marking the career tracks, choices, and achievements of their graduates” (Standard 6.6), with instructions that this database should be used “as part of a process to determine how well students accomplish the COM's educational goals” (Standard 6.8.3).
1 As part of this assessment, AOA COCA has also long required that COMs include student performance on the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 1, Level 2-Cognitive Evaluation (CE), and Level 2-Performance Evaluation (PE). These examinations must be completed before graduation. In July 2013, AOA COCA added a requirement that COMs also track performance on COMLEX-USA Level 3, which is taken after graduation (Standard 6.8.2).
1 Furthermore, beginning with the 2013-2014 academic year, AOA COCA requires that each COM report the first-time Level 3 pass rate from the previous 4 academic years on its website, catalog, and promotional materials (Standard 8.5),
1 thus making this information available to potential applicants and the public.
In Standard 5.1.1, AOA COCA also requires that “To ensure the COM meets its mission and objectives, the COM must tie its admissions process and criteria to the outcome performance of its graduates.”
1 The guideline for this standard emphasizes recruitment of students “who may be expected to have the ability to complete the curriculum of study in accordance with the COM's mission and objectives,” but the wording of the standard itself might be thought to include outcomes on Level 3.
The new accreditation standards focus fresh attention on the relationship between performance on Level 3 and admissions data and performance within the COM curriculum (including COMLEX-USA Level 1 and 2 scores). Multiple studies on osteopathic medical education have shown the relationship between admissions data and performance during the COM curriculum
2-7 or the relationship between COM performance and performance on components of Levels 1 or 2.
8-14 Studies have also examined the relationships among performance on the various levels of COMLEX-USA, including Level 3, and an increasing number of studies have shown a relationship between Level 3 performance and other measures of performance in and after residency.
15-19 Multiple studies on the United States Medical Licensing Examination (USMLE) Step 3, the analogous examination used by MD degree–granting medical schools,
20-22 have found that performance on USMLE Step 3 depended in part on whether graduates sought a broad-based residency or one with a more narrow focus.
23-25 However, no studies could be identified that correlated COMLEX-USA Level 3 performance to admissions variables or performance in the COM curriculum other than to scores on Levels 1 and 2. If performance on Level 3 is to be used in accreditation decisions about COMs, and if admissions variables are at some point to be regarded as relating to Level 3 score, then these relationships need to be elucidated.
The current study was designed to examine the relationship of admissions variables and COM performance to performance on Level 3. Variables included undergraduate grade point averages (GPAs), Medical College Admission Test (MCAT) scores, COM GPAs (year 1, year 2, and clinical rotation [years 3 and 4]), and COMLEX-USA Level 1, Level 2-CE, and Level 2-PE scores. In addition, we sought to assess whether a relationship existed between Level 3 scores and sex, curriculum track, year of graduation, and residency specialty in the first postgraduate year.
The mission of the West Virginia School of Osteopathic Medicine (WVSOM) is focused on preparing graduates to provide primary care in rural areas of West Virginia.
26 Data from all WVSOM graduates (classes 2008-2011) were initially included, even if a student had initially failed a program requirement, such as a course or Level 1 or 2 of COMLEX-USA.
The majority of WVSOM's students in the classes studied had been enrolled in a systems-based curriculum. A smaller number of students had been enrolled in a modified problem-based learning curriculum, which was implemented with the graduating class of 2007 and discontinued with the class of 2013. For the purposes of this study, a student's curriculum track was designated at the time of matriculation. Students in both curriculum tracks completed the same curriculum in the clinical rotations.
The clinical curriculum at WVSOM consists primarily of rotations in ambulatory and hospital settings. Grades in WVSOM's clinical curriculum are determined primarily by preceptor ratings (ranging from 70% to 100% of the grade), with some rotations requiring postrotation written tests or other requirements, such as logs or graded case studies. The school conducts Objective Structured Clinical Evaluations, but for the graduates included in this analysis, performance on these evaluations was not a factor in any GPA. Furthermore, for the years studied, no score on any test by the National Board of Osteopathic Medical Examiners was part of any GPA. The class size increased with the class that graduated in 2010, so we also looked for differences in Level 3 scores based on year of graduation.
The present study was exempt from human subjects review by the institutional review board at WVSOM. Analyses were performed using the SAS 9.2 statistical software program (SAS Institute, Inc).
Admissions variables included undergraduate science, nonscience, and total GPAs; MCAT Verbal Reasoning, Physical Sciences, and Biological Sciences subtest scores; and total MCAT score. For applicants who had taken the MCAT more than once, the highest total MCAT score and the highest subtest scores were used.
The COM performance variables included GPA at the end of year 1, year 2, and the clinical rotation years. At WVSOM, grades are on a scale from 0 to 100, with a score of 70 required for passing. Grade point averages were calculated according to the procedures used for student transcripts. The other COM variable was COMLEX-USA Level 1 and 2 scores. The primary path to licensure for osteopathic physicians, the COMLEX-USA, has 3 levels.
27 Level 1 is an 8-hour computer-based examination completed at the end of COM year 2. Level 2 is taken during the clinical rotations and has 2 components: the Level 2-CE, an 8-hour computer-based examination, and the Level 2-PE, a 1-day performance examination in which each candidate takes part in 12 standardized patient encounters. If a WVSOM student had multiple scores on a level or component of the COMLEX-USA, the score obtained on the first attempt was used. Level 3 is currently an 8-hour computer-based examination of knowledge obtained during residency training.
Based on the residency program that graduates entered for their first postgraduate year, graduates were assigned to 1 of 3 categories: broad-based (family practice, internal medicine, pediatrics, emergency medicine, or any combination of these specialties); traditional internship or transitional year; and not broad-based (all other residencies).
We anticipated that too few graduates would fail Level 3 to allow meaningful analysis based on failures; therefore, we looked at performance on Level 3 as a continuous variable (3-digit scores). For analysis of differences based on sex and curriculum track, an independent 2-sample t test was used. An analysis of variance was used for differences between year of graduation and residency specialty. For all continuous variables—admissions variables, COM GPAs, and Level 1 and Level 2-CE scores—we calculated Pearson product moment correlations with Level 3 scores and statistical significance. We also squared the correlation to obtain the percentage of variation in Level 3 scores that could be accounted for by knowing the other variable. As Level 2-PE is reported to COMs only as pass/fail and, therefore, is a dichotomous variable, we used an independent 2-sample t test.
Because we ran multiple significance tests, a Bonferroni correction was applied. Using the standard type I error rate of α=.05, the Bonferroni adjustment created an alternate P value cutoff of αadjusted=.0029, where αadjusted=.05/17, because 17 statistical tests were performed.
During the 4-year study period (2008-2011), WVSOM had 556 graduates. As of April 1, 2014, Level 3 scores were available for 552 graduates (99.3%). A total of 4 students were excluded from the data analysis (3 from the class of 2010 and 1 from the class of 2011) because their Level 3 scores were not available at the time of analysis. Of the 552 graduates with a Level 3 score, 521 (94.4%) passed the examination on their first attempt, and all of those who failed on the first attempt (31 [100%]) eventually passed.
Table 1 presents the characteristics of the sample. No statistically significant differences were found in Level 3 performance based on sex, curriculum track, graduating class, or residency specialty.
Correlations of continuous variables with COMLEX-USA Level 3 are shown in
Table 2, and the
Figure shows the percentage of variation in Level 3 scores accounted for by these variables. Most correlations between these variables and Level 3 scores were statistically significant, although they varied in practical value. The strongest correlation between Level 3 and any admissions variable was found with the total MCAT score (
r2=0.205), accounting for 4.2% (the correlation squared) of the variation in Level 3. Of performance measures in COMs, the greatest percentage of variation in Level 3 was accounted for by year 2 GPA, which accounted for 35.4% of the variation in Level 3. Level 1 scores accounted for 38.5% of the variation in Level 3 scores, and Level 2-CE scores provided the greatest percentage of variation (45.7%).
The mean (SD) Level 3 score for the 14 students who had initially failed the Level 2-PE was 488.5 (124.6), and the mean score for the 538 students initially passing the Level 2-PE was 517.7 (112.2). An independent 2-sample t test showed that this difference was not statistically significant (P=.338).
To our knowledge, no studies have documented the degree of relationship between admissions variables or COM GPA and COMLEX-USA Level 3 scores. The current study found weak relationships between admissions variables and Level 3 performance, with the strongest relationship (total MCAT score) accounting for less than 5% of the variation in Level 3 performance. The admissions data identified in other studies as predicting success in year 1 and year 2 and success on Level 1 were of little help in predicting performance on Level 3. An optimistic view of these findings is that students who have relatively low admissions statistics but successfully graduate have overcome their early disadvantage, and on their first attempt attained Level 3 scores that approximate those of students who had better admissions statistics. Furthermore, the relationship of Level 3 scores with admissions variables was weak, indicating that the AOA COCA requirement of a college to “tie its admissions process and criteria to the outcome performance of its graduates” should perhaps be revisited in the future. The findings of the present study suggest that AOA COCA reviewers should continue to focus on whether a college admits students who “have the ability to complete the curriculum”
1 rather than extending this standard to apply to performance during residency.
Regarding the relationship between performance in the COM curriculum and performance on Level 3, the year 2 GPA had the strongest relationship, accounting for 35.4% of the variation in Level 3 scores. Therefore, year 2 GPA seems to be an appropriate consideration in the selection of residents, which is moderately supported in the literature.
15-19 Predicting clinical performance in residency or in professional practice is a completely different matter, however. Research by Mc-Gaghie and colleagues
28 found a weak relationship between USMLE Part 1 and Part 2 scores and objective measures of clinical performance in residency, so one might assume that low correlations would be found between COMLEX-USA Level 3 and objectively measured clinical performance in residency. However, researchers in Canada found a relationship between scores on Canada's licensing examination taken in residency and patient outcomes.
29-32 On the basis of these findings, more research is needed.
Regarding clinical rotation GPA, ratings by preceptors comprise 70% to 100% of WVSOM's GPAs on clinical rotations, so clinical rotation GPA is less a measure of knowledge (as it would be measured on a multiple-choice test) than a measure of clinical performance. Therefore, like the Level 2-PE score, clinical rotation GPA should not be expected to correlate strongly with the Level 3 knowledge examination score. Future research might specifically examine the clinical core competencies being taught and assessed in both predoctoral and residency training, including the ability to perform osteopathic manipulative treatment and comparing these subscores with Level 3 performance.
The present study found correlations greater than 0.60 between Level 3 scores and both the Level 1 and the Level 2-CE scores. Poor performance on Level 1 or Level 2-CE seems to be a “red flag” for poor performance on Level 3. The relationship between Level 2-PE and Level 3 scores was small and not statistically significant. This finding, too, matched findings in the literature,
8,13 suggesting that the Level 2-PE may tap different skills from those assessed by computer-based examinations.
Of the graduates with a score reported, 521 (94.4%) passed the Level 3 examination on their first attempt, reflecting the national percentage of graduates who passed Level 3 on the first attempt between 2007 and 2012 (93%).
33,34 (The national average increased to 95.1% in the 2012-2013 academic year.)
A key question is the extent to which performance on Level 3 should be attributed to a resident's COM or to the graduate medical education program. The AOA COCA standard requiring colleges to report first-time pass rates on Level 3 implies a belief that COMs should be held accountable for this performance; however, postgraduate education is a time of tremendous learning. Colleges of osteopathic medicine should be held accountable for their students' scores on Level 1, Level 2-CE, and Level 2-PE, but it is our opinion that residency programs should share the responsibility for scores on Level 3. In the current study, the strongest correlation with performance on Level 3 was Level 2-CE scores, which accounted for about half the variation in Level 3 scores, leaving half the variation in Level 3 scores not accounted for. A multisite study might analyze Level 3 scores while controlling for both Level 2-CE performance and residency program type to tease out the amount of variation in Level 3 scores that would be attributed to COMs.
The methodology in the present study was slightly different from that used in other studies, in that we included every graduate from these classes for whom a first attempt score on Level 3 was available. Studies such as that by Dixon
4 included only those students who had graduated in 4 years. Also, we used the highest total MCAT score and the subscores associated with that attempt rather than the highest subscore obtained, as some others have done (eg, Evans et al
5). Researchers using other methods might obtain different results.
The modified problem-based learning program that was briefly in effect at WVSOM was substantially different from the authentic problem-based learning program as described by Barrows and Tamblyn,
35 in that WVSOM's students in this curriculum track spent large portions of their time in traditional learning activities. Therefore, no conclusions should be drawn from the current study to the larger literature on problem-based learning.
The graduates in the current study were from a single, unique COM focused on educating students for service in West Virginia in primary care and rural medicine. Studies have identified WVSOM as having the highest percentage of graduates in rural practice of any MD- or DO-granting medical school in the nation
36 and as having the largest number of graduates in rural practice in Appalachia.
37 Furthermore, more than 65% of recent WVSOM graduates initially entered primary care residencies.
38 The admissions process at WVSOM and the curriculum are focused on these outcomes and may not be typical of other COMs.
We did not attempt to quantify the admissions committee's assessment of the attitudes, values, and career plans of applicants, information that the WVSOM faculty use as part of the “holistic review” of applicants,
37 data that may be related to Level 3 scores.
40,41 Furthermore, the only residency measure included in the current study was the Level 3 score. Future studies of residency performance and Level 3 should include direct measures of clinical competence or patient outcomes data.
28-32
Changes in the assessment of potential and current osteopathic medical students are anticipated at a national level. Revision of the MCAT is planned,
42 which may result in an examination that better predicts performance in osteopathic medical school and in residency. Regarding COMEX-USA Level 3, changes are under development and being pilot tested for adoption, with a target date of 2017 or 2018. We expect the current knowledge component to remain, and a second day of testing to be added to assess core competencies that are not well measured by multiple-choice tests (personal communication, B. Bates, DO, 2014). This new component of Level 3 might have a stronger relationship with performance-based examinations. Further research will be needed on both of these new examinations.
The current study at WVSOM found little relationship between admissions variables and performance on COMLEX-USA Level 3. A strong relationship was found between performance on Level 3 and the COM year 2 GPA and Level 1 and Level 2-CE scores. Performance on Level 2-PE was not related to performance on Level 3.
We thank the following individuals for their thoughtful suggestions on earlier drafts of this manuscript: Bruce Bates, DO; Hugh Clements-Jewery, PhD; and Brian Griffith, PhD. We are also grateful for the assistance in manuscript preparation provided by Bethany M. Hampton, BS.