Free
Medical Education  |   February 2015
Relationship of Admissions Variables and College of Osteopathic Medicine Variables to Performance on COMLEX-USA Level 3
Author Notes
  • From the West Virginia School of Osteopathic Medicine in Lewisburg 
  •    Dr Fisk holds a master's degree in educational technology.
     
  • Address correspondence to Helen H. Baker, PhD, MBA, West Virginia School of Osteopathic Medicine, 400 N Lee St, Lewisburg, WV 24901-1128. E-mail: hbaker@osteo.wvsom.edu  
  •    Author Contributions  All authors provided substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; all authors drafted the article or revised it critically for important intellectual content; and Drs Baker, Boisvert, Fisk, and Pence gave final approval of the version of the article to be published.
     
Article Information
Medical Education / Medical School Admissions / Graduate Medical Education / COMLEX-USA
Medical Education   |   February 2015
Relationship of Admissions Variables and College of Osteopathic Medicine Variables to Performance on COMLEX-USA Level 3
The Journal of the American Osteopathic Association, February 2015, Vol. 115, 100-109. doi:10.7556/jaoa.2015.021
The Journal of the American Osteopathic Association, February 2015, Vol. 115, 100-109. doi:10.7556/jaoa.2015.021
Abstract

Context: New accreditation standards require that all US colleges of osteopathic medicine (COMs) publically report the first-time pass rates of graduates on the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 3. Little is known about the extent to which admissions variables or COM performance measures relate to Level 3 performance.

Objective: To examine the relationship of admissions variables and COM performance to scores on Level 3 and to assess whether a relationship existed between Level 3 scores and sex, curriculum track, year of graduation, and residency specialty in the first postgraduate year.

Methods: Data were analyzed from 4 graduating classes (2008-2011) of the West Virginia School of Osteopathic Medicine in Lewisburg. Relationships were examined between first-attempt scores on COMLEX-USA Level 3 and Medical College Admission Test (MCAT) scores; undergraduate grade point averages (GPAs); GPAs in COM year 1, year 2, and clinical rotation years (years 3 and 4); and first-attempt scores on COMLEX-USA Level 1, Level 2-Cognitive Evaluation, and Level 2-Performance Evaluation.

Results: Of the 556 graduates during this 4-year period, COMLEX-USA Level 3 scores were available for 552 graduates (99.3%). No statistically significant differences were found in Level 3 scores based on sex, curriculum track, graduating class, or residency specialty. The strongest relationship between Level 3 scores and any admissions variable was with total MCAT score, which accounted for 4.2% of the variation in Level 3 scores. The strongest relationship between Level 3 scores and COM year performance measures was with year 2 GPA, which accounted for 35.4% of the variation in Level 3 scores. Level 1 scores accounted for 38.5% of the variation in Level 3 scores, and Level 2-Cognitive Evaluation scores accounted for the greatest percentage of variation (45.7%). The correlation of Level 3 scores with passing the Level 2-Performance Evaluation on the first attempt was not statistically significant.

Conclusion: A weak relationship was found between admissions variables and performance on COMLEX-USA Level 3, suggesting that graduates with lower MCAT scores and undergraduate GPAs may have overcome their early disadvantage. Strong relationships were found between Level 3 scores and year 2 GPAs, as well as scores on COMLEX-USA Level 1 and Level 2-Cognitive Evaluation.

Colleges of osteopathic medicine (COMs) accredited by the American Osteopathic Association Commission on Osteopathic College Accreditation (AOA COCA) have long been required to maintain “a longitudinal record marking the career tracks, choices, and achievements of their graduates” (Standard 6.6), with instructions that this database should be used “as part of a process to determine how well students accomplish the COM's educational goals” (Standard 6.8.3).1 As part of this assessment, AOA COCA has also long required that COMs include student performance on the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 1, Level 2-Cognitive Evaluation (CE), and Level 2-Performance Evaluation (PE). These examinations must be completed before graduation. In July 2013, AOA COCA added a requirement that COMs also track performance on COMLEX-USA Level 3, which is taken after graduation (Standard 6.8.2).1 Furthermore, beginning with the 2013-2014 academic year, AOA COCA requires that each COM report the first-time Level 3 pass rate from the previous 4 academic years on its website, catalog, and promotional materials (Standard 8.5),1 thus making this information available to potential applicants and the public. 
In Standard 5.1.1, AOA COCA also requires that “To ensure the COM meets its mission and objectives, the COM must tie its admissions process and criteria to the outcome performance of its graduates.”1 The guideline for this standard emphasizes recruitment of students “who may be expected to have the ability to complete the curriculum of study in accordance with the COM's mission and objectives,” but the wording of the standard itself might be thought to include outcomes on Level 3. 
The new accreditation standards focus fresh attention on the relationship between performance on Level 3 and admissions data and performance within the COM curriculum (including COMLEX-USA Level 1 and 2 scores). Multiple studies on osteopathic medical education have shown the relationship between admissions data and performance during the COM curriculum2-7 or the relationship between COM performance and performance on components of Levels 1 or 2.8-14 Studies have also examined the relationships among performance on the various levels of COMLEX-USA, including Level 3, and an increasing number of studies have shown a relationship between Level 3 performance and other measures of performance in and after residency.15-19 Multiple studies on the United States Medical Licensing Examination (USMLE) Step 3, the analogous examination used by MD degree–granting medical schools,20-22 have found that performance on USMLE Step 3 depended in part on whether graduates sought a broad-based residency or one with a more narrow focus.23-25 However, no studies could be identified that correlated COMLEX-USA Level 3 performance to admissions variables or performance in the COM curriculum other than to scores on Levels 1 and 2. If performance on Level 3 is to be used in accreditation decisions about COMs, and if admissions variables are at some point to be regarded as relating to Level 3 score, then these relationships need to be elucidated. 
The current study was designed to examine the relationship of admissions variables and COM performance to performance on Level 3. Variables included undergraduate grade point averages (GPAs), Medical College Admission Test (MCAT) scores, COM GPAs (year 1, year 2, and clinical rotation [years 3 and 4]), and COMLEX-USA Level 1, Level 2-CE, and Level 2-PE scores. In addition, we sought to assess whether a relationship existed between Level 3 scores and sex, curriculum track, year of graduation, and residency specialty in the first postgraduate year. 
Methods
Participants and Setting
The mission of the West Virginia School of Osteopathic Medicine (WVSOM) is focused on preparing graduates to provide primary care in rural areas of West Virginia.26 Data from all WVSOM graduates (classes 2008-2011) were initially included, even if a student had initially failed a program requirement, such as a course or Level 1 or 2 of COMLEX-USA. 
The majority of WVSOM's students in the classes studied had been enrolled in a systems-based curriculum. A smaller number of students had been enrolled in a modified problem-based learning curriculum, which was implemented with the graduating class of 2007 and discontinued with the class of 2013. For the purposes of this study, a student's curriculum track was designated at the time of matriculation. Students in both curriculum tracks completed the same curriculum in the clinical rotations. 
The clinical curriculum at WVSOM consists primarily of rotations in ambulatory and hospital settings. Grades in WVSOM's clinical curriculum are determined primarily by preceptor ratings (ranging from 70% to 100% of the grade), with some rotations requiring postrotation written tests or other requirements, such as logs or graded case studies. The school conducts Objective Structured Clinical Evaluations, but for the graduates included in this analysis, performance on these evaluations was not a factor in any GPA. Furthermore, for the years studied, no score on any test by the National Board of Osteopathic Medical Examiners was part of any GPA. The class size increased with the class that graduated in 2010, so we also looked for differences in Level 3 scores based on year of graduation. 
The present study was exempt from human subjects review by the institutional review board at WVSOM. Analyses were performed using the SAS 9.2 statistical software program (SAS Institute, Inc). 
Variables and Analysis
Admissions variables included undergraduate science, nonscience, and total GPAs; MCAT Verbal Reasoning, Physical Sciences, and Biological Sciences subtest scores; and total MCAT score. For applicants who had taken the MCAT more than once, the highest total MCAT score and the highest subtest scores were used. 
The COM performance variables included GPA at the end of year 1, year 2, and the clinical rotation years. At WVSOM, grades are on a scale from 0 to 100, with a score of 70 required for passing. Grade point averages were calculated according to the procedures used for student transcripts. The other COM variable was COMLEX-USA Level 1 and 2 scores. The primary path to licensure for osteopathic physicians, the COMLEX-USA, has 3 levels.27 Level 1 is an 8-hour computer-based examination completed at the end of COM year 2. Level 2 is taken during the clinical rotations and has 2 components: the Level 2-CE, an 8-hour computer-based examination, and the Level 2-PE, a 1-day performance examination in which each candidate takes part in 12 standardized patient encounters. If a WVSOM student had multiple scores on a level or component of the COMLEX-USA, the score obtained on the first attempt was used. Level 3 is currently an 8-hour computer-based examination of knowledge obtained during residency training. 
Based on the residency program that graduates entered for their first postgraduate year, graduates were assigned to 1 of 3 categories: broad-based (family practice, internal medicine, pediatrics, emergency medicine, or any combination of these specialties); traditional internship or transitional year; and not broad-based (all other residencies). 
We anticipated that too few graduates would fail Level 3 to allow meaningful analysis based on failures; therefore, we looked at performance on Level 3 as a continuous variable (3-digit scores). For analysis of differences based on sex and curriculum track, an independent 2-sample t test was used. An analysis of variance was used for differences between year of graduation and residency specialty. For all continuous variables—admissions variables, COM GPAs, and Level 1 and Level 2-CE scores—we calculated Pearson product moment correlations with Level 3 scores and statistical significance. We also squared the correlation to obtain the percentage of variation in Level 3 scores that could be accounted for by knowing the other variable. As Level 2-PE is reported to COMs only as pass/fail and, therefore, is a dichotomous variable, we used an independent 2-sample t test. 
Because we ran multiple significance tests, a Bonferroni correction was applied. Using the standard type I error rate of α=.05, the Bonferroni adjustment created an alternate P value cutoff of αadjusted=.0029, where αadjusted=.05/17, because 17 statistical tests were performed. 
Results
During the 4-year study period (2008-2011), WVSOM had 556 graduates. As of April 1, 2014, Level 3 scores were available for 552 graduates (99.3%). A total of 4 students were excluded from the data analysis (3 from the class of 2010 and 1 from the class of 2011) because their Level 3 scores were not available at the time of analysis. Of the 552 graduates with a Level 3 score, 521 (94.4%) passed the examination on their first attempt, and all of those who failed on the first attempt (31 [100%]) eventually passed. 
Table 1 presents the characteristics of the sample. No statistically significant differences were found in Level 3 performance based on sex, curriculum track, graduating class, or residency specialty. 
Table 1.
COMLEX-USA Level 3 Scores by Variables of the Graduating Classes of 2008 to 2011 at the West Virginia School of Osteopathic Medicine (N=552)
Level 3 Score
Variable No. (%) Mean (SD) P Valuea
Sex
Female 271 (49) 513 (105) .42b
Male 281 (51) 521 (119)
Curriculum Track
Systems-based learning 449 (81) 521 (113) .05b
Modified problem-based learning 103 (19) 498 (109)
Graduating Class
2008 96 (17) 504 (126) .15c
2009 100 (18) 500 (106)
2010 165 (30) 526 (103)
2011 191 (35) 525 (116)
Residency Specialty in First Postgraduate Year
Broad-based categoricald 351 (64) 521 (111) .52c
Traditional internship or transitional year 89 (16) 505 (109)
Not broad-based, categoricale 112 (20) 515 (120)
  a Using the standard type I error rate of α=.05, the Bonferroni adjustment created an alternate P value cutoff of an adjusted α of .0029, where the adjusted α was .05/17, because 17 statistical tests were performed in the analysis reported in this study.
  b Independent 2-sample t test.
  c Analysis of variance.
  d Family practice, internal medicine, pediatrics, emergency medicine, or any combination of these 3 alternatives.
  e All residencies other than those categorized as broad-based categorical or as traditional internship or transitional year.
  Abbreviation: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA.
Table 1.
COMLEX-USA Level 3 Scores by Variables of the Graduating Classes of 2008 to 2011 at the West Virginia School of Osteopathic Medicine (N=552)
Level 3 Score
Variable No. (%) Mean (SD) P Valuea
Sex
Female 271 (49) 513 (105) .42b
Male 281 (51) 521 (119)
Curriculum Track
Systems-based learning 449 (81) 521 (113) .05b
Modified problem-based learning 103 (19) 498 (109)
Graduating Class
2008 96 (17) 504 (126) .15c
2009 100 (18) 500 (106)
2010 165 (30) 526 (103)
2011 191 (35) 525 (116)
Residency Specialty in First Postgraduate Year
Broad-based categoricald 351 (64) 521 (111) .52c
Traditional internship or transitional year 89 (16) 505 (109)
Not broad-based, categoricale 112 (20) 515 (120)
  a Using the standard type I error rate of α=.05, the Bonferroni adjustment created an alternate P value cutoff of an adjusted α of .0029, where the adjusted α was .05/17, because 17 statistical tests were performed in the analysis reported in this study.
  b Independent 2-sample t test.
  c Analysis of variance.
  d Family practice, internal medicine, pediatrics, emergency medicine, or any combination of these 3 alternatives.
  e All residencies other than those categorized as broad-based categorical or as traditional internship or transitional year.
  Abbreviation: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA.
×
Correlations of continuous variables with COMLEX-USA Level 3 are shown in Table 2, and the Figure shows the percentage of variation in Level 3 scores accounted for by these variables. Most correlations between these variables and Level 3 scores were statistically significant, although they varied in practical value. The strongest correlation between Level 3 and any admissions variable was found with the total MCAT score (r2=0.205), accounting for 4.2% (the correlation squared) of the variation in Level 3. Of performance measures in COMs, the greatest percentage of variation in Level 3 was accounted for by year 2 GPA, which accounted for 35.4% of the variation in Level 3. Level 1 scores accounted for 38.5% of the variation in Level 3 scores, and Level 2-CE scores provided the greatest percentage of variation (45.7%). 
Table 2.
Correlations of Continuous Variables With COMLEX-USA Level 3 Scores for the Graduating Classes of 2008 to 2011 at the West Virginia School of Osteopathic Medicine (N=552)
Variable Correlation With Level 3 P Value % of Variation Accounted fora
Admissions Variables
GPA
Nonscience 0.097 .023a 0.9 .
Science 0.160 <.001 2.6
Total 0.152 <.001 2.3
MCAT score
Verbal 0.138 .001 1.9
Physical sciences 0.111 .009a 1.2
Biological sciences 0.182 <.001 3.3
Total 0.205 <.001 4.2
Performance in the COM Curriculum, GPA
Year 1 0.462 <.001 21.4
Year 2 0.595 <.001 35.4
Clinical years 0.312 <.001 9.8
COMLEX-USA Scores
Level 1 0.620 <.001 38.5
Level 2-CE 0.676 <.001 45.7
  a Correlation squared.
  b Not statistically significant. Using the standard type I error rate of α=.05, the Bonferroni adjustment created an alternate P value cutoff of an adjusted a of .0029, where the adjusted a was .05/17, because 17 statistical tests were performed in the analysis reported in this study.
  Abbreviations: COM, college of osteopathic medicine; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; GPA, grade point average; MCAT, Medical College Admission Test.
Table 2.
Correlations of Continuous Variables With COMLEX-USA Level 3 Scores for the Graduating Classes of 2008 to 2011 at the West Virginia School of Osteopathic Medicine (N=552)
Variable Correlation With Level 3 P Value % of Variation Accounted fora
Admissions Variables
GPA
Nonscience 0.097 .023a 0.9 .
Science 0.160 <.001 2.6
Total 0.152 <.001 2.3
MCAT score
Verbal 0.138 .001 1.9
Physical sciences 0.111 .009a 1.2
Biological sciences 0.182 <.001 3.3
Total 0.205 <.001 4.2
Performance in the COM Curriculum, GPA
Year 1 0.462 <.001 21.4
Year 2 0.595 <.001 35.4
Clinical years 0.312 <.001 9.8
COMLEX-USA Scores
Level 1 0.620 <.001 38.5
Level 2-CE 0.676 <.001 45.7
  a Correlation squared.
  b Not statistically significant. Using the standard type I error rate of α=.05, the Bonferroni adjustment created an alternate P value cutoff of an adjusted a of .0029, where the adjusted a was .05/17, because 17 statistical tests were performed in the analysis reported in this study.
  Abbreviations: COM, college of osteopathic medicine; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; GPA, grade point average; MCAT, Medical College Admission Test.
×
Figure.
Percentage of variation accounted for by correlations of continuous variables with the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 3 scores for the graduating classes of 2008-2011 at the West Virginia School of Osteopathic Medicine (N=552). Abbreviations: CE, Cognitive Evaluation; GPA, grade point average; MCAT, Medical College Admission Test.
Figure.
Percentage of variation accounted for by correlations of continuous variables with the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 3 scores for the graduating classes of 2008-2011 at the West Virginia School of Osteopathic Medicine (N=552). Abbreviations: CE, Cognitive Evaluation; GPA, grade point average; MCAT, Medical College Admission Test.
The mean (SD) Level 3 score for the 14 students who had initially failed the Level 2-PE was 488.5 (124.6), and the mean score for the 538 students initially passing the Level 2-PE was 517.7 (112.2). An independent 2-sample t test showed that this difference was not statistically significant (P=.338). 
Discussion
To our knowledge, no studies have documented the degree of relationship between admissions variables or COM GPA and COMLEX-USA Level 3 scores. The current study found weak relationships between admissions variables and Level 3 performance, with the strongest relationship (total MCAT score) accounting for less than 5% of the variation in Level 3 performance. The admissions data identified in other studies as predicting success in year 1 and year 2 and success on Level 1 were of little help in predicting performance on Level 3. An optimistic view of these findings is that students who have relatively low admissions statistics but successfully graduate have overcome their early disadvantage, and on their first attempt attained Level 3 scores that approximate those of students who had better admissions statistics. Furthermore, the relationship of Level 3 scores with admissions variables was weak, indicating that the AOA COCA requirement of a college to “tie its admissions process and criteria to the outcome performance of its graduates” should perhaps be revisited in the future. The findings of the present study suggest that AOA COCA reviewers should continue to focus on whether a college admits students who “have the ability to complete the curriculum”1 rather than extending this standard to apply to performance during residency. 
Regarding the relationship between performance in the COM curriculum and performance on Level 3, the year 2 GPA had the strongest relationship, accounting for 35.4% of the variation in Level 3 scores. Therefore, year 2 GPA seems to be an appropriate consideration in the selection of residents, which is moderately supported in the literature.15-19 Predicting clinical performance in residency or in professional practice is a completely different matter, however. Research by Mc-Gaghie and colleagues28 found a weak relationship between USMLE Part 1 and Part 2 scores and objective measures of clinical performance in residency, so one might assume that low correlations would be found between COMLEX-USA Level 3 and objectively measured clinical performance in residency. However, researchers in Canada found a relationship between scores on Canada's licensing examination taken in residency and patient outcomes.29-32 On the basis of these findings, more research is needed. 
Regarding clinical rotation GPA, ratings by preceptors comprise 70% to 100% of WVSOM's GPAs on clinical rotations, so clinical rotation GPA is less a measure of knowledge (as it would be measured on a multiple-choice test) than a measure of clinical performance. Therefore, like the Level 2-PE score, clinical rotation GPA should not be expected to correlate strongly with the Level 3 knowledge examination score. Future research might specifically examine the clinical core competencies being taught and assessed in both predoctoral and residency training, including the ability to perform osteopathic manipulative treatment and comparing these subscores with Level 3 performance. 
The present study found correlations greater than 0.60 between Level 3 scores and both the Level 1 and the Level 2-CE scores. Poor performance on Level 1 or Level 2-CE seems to be a “red flag” for poor performance on Level 3. The relationship between Level 2-PE and Level 3 scores was small and not statistically significant. This finding, too, matched findings in the literature,8,13 suggesting that the Level 2-PE may tap different skills from those assessed by computer-based examinations. 
Of the graduates with a score reported, 521 (94.4%) passed the Level 3 examination on their first attempt, reflecting the national percentage of graduates who passed Level 3 on the first attempt between 2007 and 2012 (93%).33,34 (The national average increased to 95.1% in the 2012-2013 academic year.) 
A key question is the extent to which performance on Level 3 should be attributed to a resident's COM or to the graduate medical education program. The AOA COCA standard requiring colleges to report first-time pass rates on Level 3 implies a belief that COMs should be held accountable for this performance; however, postgraduate education is a time of tremendous learning. Colleges of osteopathic medicine should be held accountable for their students' scores on Level 1, Level 2-CE, and Level 2-PE, but it is our opinion that residency programs should share the responsibility for scores on Level 3. In the current study, the strongest correlation with performance on Level 3 was Level 2-CE scores, which accounted for about half the variation in Level 3 scores, leaving half the variation in Level 3 scores not accounted for. A multisite study might analyze Level 3 scores while controlling for both Level 2-CE performance and residency program type to tease out the amount of variation in Level 3 scores that would be attributed to COMs. 
The methodology in the present study was slightly different from that used in other studies, in that we included every graduate from these classes for whom a first attempt score on Level 3 was available. Studies such as that by Dixon4 included only those students who had graduated in 4 years. Also, we used the highest total MCAT score and the subscores associated with that attempt rather than the highest subscore obtained, as some others have done (eg, Evans et al5). Researchers using other methods might obtain different results. 
The modified problem-based learning program that was briefly in effect at WVSOM was substantially different from the authentic problem-based learning program as described by Barrows and Tamblyn,35 in that WVSOM's students in this curriculum track spent large portions of their time in traditional learning activities. Therefore, no conclusions should be drawn from the current study to the larger literature on problem-based learning. 
The graduates in the current study were from a single, unique COM focused on educating students for service in West Virginia in primary care and rural medicine. Studies have identified WVSOM as having the highest percentage of graduates in rural practice of any MD- or DO-granting medical school in the nation36 and as having the largest number of graduates in rural practice in Appalachia.37 Furthermore, more than 65% of recent WVSOM graduates initially entered primary care residencies.38 The admissions process at WVSOM and the curriculum are focused on these outcomes and may not be typical of other COMs. 
We did not attempt to quantify the admissions committee's assessment of the attitudes, values, and career plans of applicants, information that the WVSOM faculty use as part of the “holistic review” of applicants,37 data that may be related to Level 3 scores.40,41 Furthermore, the only residency measure included in the current study was the Level 3 score. Future studies of residency performance and Level 3 should include direct measures of clinical competence or patient outcomes data.28-32 
Changes in the assessment of potential and current osteopathic medical students are anticipated at a national level. Revision of the MCAT is planned,42 which may result in an examination that better predicts performance in osteopathic medical school and in residency. Regarding COMEX-USA Level 3, changes are under development and being pilot tested for adoption, with a target date of 2017 or 2018. We expect the current knowledge component to remain, and a second day of testing to be added to assess core competencies that are not well measured by multiple-choice tests (personal communication, B. Bates, DO, 2014). This new component of Level 3 might have a stronger relationship with performance-based examinations. Further research will be needed on both of these new examinations. 
Conclusion
The current study at WVSOM found little relationship between admissions variables and performance on COMLEX-USA Level 3. A strong relationship was found between performance on Level 3 and the COM year 2 GPA and Level 1 and Level 2-CE scores. Performance on Level 2-PE was not related to performance on Level 3. 
Acknowledgments
We thank the following individuals for their thoughtful suggestions on earlier drafts of this manuscript: Bruce Bates, DO; Hugh Clements-Jewery, PhD; and Brian Griffith, PhD. We are also grateful for the assistance in manuscript preparation provided by Bethany M. Hampton, BS. 
   Financial Disclosures: None reported.
 
   Support: None reported.
 
References
American Osteopathic Association Commission on Osteopathic College Accreditation. Accreditation of Colleges of Osteopathic Medicine: COM Accreditation Standards and Procedures. Chicago, IL: American Osteopathic Association; 2014.
Baker H, Cope M, Fisk R, Gorby J, Foster R. Relationship of preadmission variables and first- and second-year course performance to performance on the National Board of Osteopathic Medical Examiners' COMLEX-USA Level 1 examination. J Am Osteopath Assoc. 2000;100(3):153-161. [PubMed]
Dixon D. Relation between variables of preadmission, medical school performance, and COMLEX-USA Levels 1 and 2 performance. J Am Osteopath Assoc. 2004;104(8):332-336. [PubMed]
Dixon D. Prediction of osteopathic medical school performance on the basis of MCAT score, GPA, sex, undergraduate major, and undergraduate institution. J Am Osteopath Assoc. 2012;112(4):175-181. [PubMed]
Evans P, Goodson L, Schoffman S. Relationship between academic achievement and student performance on the Comprehensive Osteopathic Medical Licensing Examination-USA Level 2. J Am Osteopath Assoc. 2003;103(7):331-336. [PubMed]
Evans P, Wen FK. Does the medical college admission test predict global academic performance in osteopathic medical school? J Am Osteopath Assoc. 2007;107(4):157-162. [PubMed]
Wong SK, Ramirez JR, Helf SC. Student performance on Levels 1 and 2-CE of COMLEX-USA: do elective upper-level undergraduate science courses matter? J Am Osteopath Assoc. 2009;109(11):592-598. [PubMed]
Baker HH, Cope MK, Adelman MD, Schuler S, Foster RW, Gimpel JR. Relationships between scores on the COMLEX-USA Level 2-Performance Evaluation and selected school-based performance measures. J Am Osteopath Assoc. 2006;106(5):290-295. [PubMed]
Baker H, Foster R, Bates Bet al. Relationship between academic achievement and COMLEX-USA Level 1 performance: a multisite study. J Am Osteopath Assoc. 2000;100(4):238-242. [PubMed]
Cope MK, Baker HH, Foster RW, Boisvert CS. Relationships between clinical rotation subscores, COMLEX-USA examination results, and school-based performance measures. J Am Osteopath Assoc. 2007;107(11):502-510. [PubMed]
Evans P, Goodson L, Schoffman S, Baker H. Relations between academic performance by medical students and COMLEX-USA Level 2: a multisite analysis. J Am Osteopath Assoc. 2003;103(11):551-556. [PubMed]
Hartman S, Bates B, Sprafka S. Correlation of scores for the Comprehensive Osteopathic Medical Licensing Examination with osteopathic medical school grades. J Am Osteopath Assoc. 2001;101(6):347-349. [PubMed]
Roberts WL, Pugliano G, Langenau E, Boulet JR. Modeling relationships between traditional preadmission measures and clinical skills performance on a medical licensure examination. Adv Health Sci Educ Theory Pract. 2012;17(3):403-17. doi:10.1007/s10459-011-9321-4. [CrossRef] [PubMed]
Meoli F, Wallace W, Kaiser-Smith J, Shen L. Relationship of osteopathic medical licensure examinations with undergraduate admission measures and predictive value of identifying future performance in osteopathic principles and practice/osteopathic manipulative medicine courses and rotations. J Am Osteopath Assoc. 2002;102(11):615-620. [PubMed]
Cavalieri T, Shen L, Slick G. Predictive validity of osteopathic medical licensing examinations for osteopathic medical knowledge measured by graduate written examinations. J Am Osteopath Assoc. 2003;103(7):337-342. [PubMed]
Langenau E, Pugliano G, Roberts W, Hostoffer R. Summary of ACOP (American College of Osteopathic Pediatricians) program directors' annual reports for first-year residents and relationships between resident competency performance ratings and COMLEX-USA test scores. Electronic J Am Coll Osteopath Pediatricians. 2010;2(7). http://files.eric.ed.gov/fulltext/ED514030.pdf. Accessed December 19, 2014.
Langenau EE, Pugliano G, Roberts WL. Relationships between high-stakes clinical skills exam scores and program director global competency ratings of first-year pediatric residents. Med Educ Online. 2011;16:7362-7362. doi:10.3402/meo.v16i0.7362. [CrossRef] [PubMed]
Li F, Gimpel JR, Arenson E, Song H, Bates BP, Ludwin F. Relationship between COMLEX-USA scores and performance on the American Osteopathic Board of Emergency Medicine Part I certifying examination. J Am Osteopath Assoc. 2014;114(4):260-266. doi:10.7556/jaoa.2014.051. [CrossRef] [PubMed]
Sevensma SC, Navarre G, Richards RK. COMLEX-USA and in-service examination scores: tools for evaluating medical knowledge among residents. J Am Osteopath Assoc. 2008;108(12):713-716. [PubMed]
Donnon T, Paolucci EO, Violato C. The predictive validity of the MCAT for medical school performance and medical board licensing examinations: a meta-analysis of the published research. Acad Med. 2007;82(1):100-106. doi:10.1097/01. ACM.0000249878.25186.b7. [CrossRef] [PubMed]
Callahan CA, Hojat M, Veloski J, Erdmann JB, Gonnella JS. The predictive validity of three versions of the MCAT in relation to performance in medical school, residency, and licensing examinations: a longitudinal study of 36 classes of Jefferson Medical College. Acad Med. 2010;85(6):980-987. doi:10.1097/ACM.0b013e3181cece3d. [CrossRef] [PubMed]
Julian ER. Validity of the Medical College Admission Test for predicting medical school performance. Acad Med. 2005;80(10):910-917. [CrossRef] [PubMed]
Andriole DA, Jeffe DB, Hageman HL, Whelan AJ. What predicts USMLE Step 3 performance? Acad Med. 2005;80(10):S21-S24. [CrossRef] [PubMed]
Sawhill AJ, Dillon GF, Ripkey DR, Hawkins RE, Swanson DB. The impact of postgraduate training and timing on USMLE Step 3 performance. Acad Med. 2003;78(10):S10-S12. [CrossRef] [PubMed]
Feinberg RA, Swygert KA, Haist SA, Dillon GF, Murray CT. The impact of postgraduate training on USMLE Step 3 and its computer-based case simulation component. J Gen Intern Med. 2012;27(1):65-70. doi:10.1007/s11606-011-1835-1. [CrossRef] [PubMed]
About us. West Virginia School of Osteopathic Medicine website. http://www.wvsom.edu/AboutWVSOM/about-home. Accessed January 5, 2015.
National Board of Osteopathic Medical Examiners. COMLEX-USA Bulletin of Information 2013-2014. Conshohocken, PA: National Board of Osteopathic Medical Examiners; 2013.
McGaghie WC, Cohen ER, Wayne DB. Are United States Medical Licensing Exam Step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? Acad Med. 2011;86(1):48-52. doi:10.1097/ACM.0b013e3181ffacdb. [CrossRef] [PubMed]
Abrahamowicz M, Dauphinee WD, Hanley JAet al. Association between licensure examination scores and practice in primary care. JAMA. 2002;288(23):3019. doi:10.1001/jama.288.23.3019. [CrossRef] [PubMed]
Kawasumi Y, Ernst P, Abrahamowicz M, Tamblyn R. Association between physician competence at licensure and the quality of asthma management among patients with out-of-control asthma. Arch Intern Med. 2011;171(14):1292-1294. doi:10.1001/archinternmed.2011.319. [CrossRef] [PubMed]
Klass D, Abrahamowicz M, Dauphinee Det al. Doctor scores on national qualifying examinations predict quality of care in future practice. Med Educ. 2009;43(12):1166-1173. doi:10.1111/j.1365-2923.2009.03534.x. [CrossRef] [PubMed]
Tamblyn R, Abrahamowicz M, Brailovsky Cet al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA. 1998;280(11):969. [CrossRef] [PubMed]
National Board of Osteopathic Medical Examiners. A Clear Vision Being Realized - NBOME 2011 Annual Report. Conshohocken, PA: National Board of Osteopathic Medical Examiners; 2011:19.
National Board of Osteopathic Medical Examiners. Integrity: Standing for Something - NBOME 2013 Annual Report. Conshohocken, PA: National Board of Osteopathic Medical Examiners; 2013:15.
Barrows H, Tamblyn R. Problem-Based Learning: An Approach to Medical Education. Vol 1. New York, NY: Springer; 1980.
Chen F, Fordyce M, Andes S, Hart LG. Which medical s chools produce rural physicians? a 15-year update [published correction appears in Acad Med. 2010:85(6):998]. Acad Med. 2010;85(4):594-598. doi:10.1097/ACM.0b013e3181d280e9. [CrossRef] [PubMed]
Baker HH, Pathman DE, Nemitz JW, Boisvert CS, Schwartz RJ, Ridpath LC. Which U.S. medical schools are providing the most physicians for the Appalachian region of the United States? Acad Med. 2012;87(4):498-505. doi:10.1097/ACM.0b013e318248f3be. [CrossRef] [PubMed]
West Virginia Higher Education Policy Commission. West Virginia Health Sciences and Rural Health Report Card 2013. Charleston: West Virginia Higher Education Policy Commission; 2014:28.
Witzburg RA, Sondheimer HM. Holistic review—shaping the medical profession one applicant at a time. N Engl J Med. 2013;368(17):1565-1567. doi:10.1056/NEJMp1300411. [CrossRef] [PubMed]
Kreiter CD, Axelson RD. A perspective on medical school admission research and practice over the last 25 years. Teach Learn Med. 2013;25(suppl 1):S50-S56. doi:10.1080/10401334.2013.842910. [CrossRef] [PubMed]
Mahon KE, Henderson MK, Kirch DG. Selecting tomorrow's physicians: the key to the future health care workforce. Acad Med. 2013;88(12):1806-1811. [CrossRef] [PubMed]
Cohen JJ. Will changes in the MCAT and USMLE ensure that future physicians have what it takes? JAMA. 2013;310(21):2253-2254. doi:10.1001/jama.2013.283389. [CrossRef] [PubMed]
Figure.
Percentage of variation accounted for by correlations of continuous variables with the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 3 scores for the graduating classes of 2008-2011 at the West Virginia School of Osteopathic Medicine (N=552). Abbreviations: CE, Cognitive Evaluation; GPA, grade point average; MCAT, Medical College Admission Test.
Figure.
Percentage of variation accounted for by correlations of continuous variables with the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 3 scores for the graduating classes of 2008-2011 at the West Virginia School of Osteopathic Medicine (N=552). Abbreviations: CE, Cognitive Evaluation; GPA, grade point average; MCAT, Medical College Admission Test.
Table 1.
COMLEX-USA Level 3 Scores by Variables of the Graduating Classes of 2008 to 2011 at the West Virginia School of Osteopathic Medicine (N=552)
Level 3 Score
Variable No. (%) Mean (SD) P Valuea
Sex
Female 271 (49) 513 (105) .42b
Male 281 (51) 521 (119)
Curriculum Track
Systems-based learning 449 (81) 521 (113) .05b
Modified problem-based learning 103 (19) 498 (109)
Graduating Class
2008 96 (17) 504 (126) .15c
2009 100 (18) 500 (106)
2010 165 (30) 526 (103)
2011 191 (35) 525 (116)
Residency Specialty in First Postgraduate Year
Broad-based categoricald 351 (64) 521 (111) .52c
Traditional internship or transitional year 89 (16) 505 (109)
Not broad-based, categoricale 112 (20) 515 (120)
  a Using the standard type I error rate of α=.05, the Bonferroni adjustment created an alternate P value cutoff of an adjusted α of .0029, where the adjusted α was .05/17, because 17 statistical tests were performed in the analysis reported in this study.
  b Independent 2-sample t test.
  c Analysis of variance.
  d Family practice, internal medicine, pediatrics, emergency medicine, or any combination of these 3 alternatives.
  e All residencies other than those categorized as broad-based categorical or as traditional internship or transitional year.
  Abbreviation: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA.
Table 1.
COMLEX-USA Level 3 Scores by Variables of the Graduating Classes of 2008 to 2011 at the West Virginia School of Osteopathic Medicine (N=552)
Level 3 Score
Variable No. (%) Mean (SD) P Valuea
Sex
Female 271 (49) 513 (105) .42b
Male 281 (51) 521 (119)
Curriculum Track
Systems-based learning 449 (81) 521 (113) .05b
Modified problem-based learning 103 (19) 498 (109)
Graduating Class
2008 96 (17) 504 (126) .15c
2009 100 (18) 500 (106)
2010 165 (30) 526 (103)
2011 191 (35) 525 (116)
Residency Specialty in First Postgraduate Year
Broad-based categoricald 351 (64) 521 (111) .52c
Traditional internship or transitional year 89 (16) 505 (109)
Not broad-based, categoricale 112 (20) 515 (120)
  a Using the standard type I error rate of α=.05, the Bonferroni adjustment created an alternate P value cutoff of an adjusted α of .0029, where the adjusted α was .05/17, because 17 statistical tests were performed in the analysis reported in this study.
  b Independent 2-sample t test.
  c Analysis of variance.
  d Family practice, internal medicine, pediatrics, emergency medicine, or any combination of these 3 alternatives.
  e All residencies other than those categorized as broad-based categorical or as traditional internship or transitional year.
  Abbreviation: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA.
×
Table 2.
Correlations of Continuous Variables With COMLEX-USA Level 3 Scores for the Graduating Classes of 2008 to 2011 at the West Virginia School of Osteopathic Medicine (N=552)
Variable Correlation With Level 3 P Value % of Variation Accounted fora
Admissions Variables
GPA
Nonscience 0.097 .023a 0.9 .
Science 0.160 <.001 2.6
Total 0.152 <.001 2.3
MCAT score
Verbal 0.138 .001 1.9
Physical sciences 0.111 .009a 1.2
Biological sciences 0.182 <.001 3.3
Total 0.205 <.001 4.2
Performance in the COM Curriculum, GPA
Year 1 0.462 <.001 21.4
Year 2 0.595 <.001 35.4
Clinical years 0.312 <.001 9.8
COMLEX-USA Scores
Level 1 0.620 <.001 38.5
Level 2-CE 0.676 <.001 45.7
  a Correlation squared.
  b Not statistically significant. Using the standard type I error rate of α=.05, the Bonferroni adjustment created an alternate P value cutoff of an adjusted a of .0029, where the adjusted a was .05/17, because 17 statistical tests were performed in the analysis reported in this study.
  Abbreviations: COM, college of osteopathic medicine; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; GPA, grade point average; MCAT, Medical College Admission Test.
Table 2.
Correlations of Continuous Variables With COMLEX-USA Level 3 Scores for the Graduating Classes of 2008 to 2011 at the West Virginia School of Osteopathic Medicine (N=552)
Variable Correlation With Level 3 P Value % of Variation Accounted fora
Admissions Variables
GPA
Nonscience 0.097 .023a 0.9 .
Science 0.160 <.001 2.6
Total 0.152 <.001 2.3
MCAT score
Verbal 0.138 .001 1.9
Physical sciences 0.111 .009a 1.2
Biological sciences 0.182 <.001 3.3
Total 0.205 <.001 4.2
Performance in the COM Curriculum, GPA
Year 1 0.462 <.001 21.4
Year 2 0.595 <.001 35.4
Clinical years 0.312 <.001 9.8
COMLEX-USA Scores
Level 1 0.620 <.001 38.5
Level 2-CE 0.676 <.001 45.7
  a Correlation squared.
  b Not statistically significant. Using the standard type I error rate of α=.05, the Bonferroni adjustment created an alternate P value cutoff of an adjusted a of .0029, where the adjusted a was .05/17, because 17 statistical tests were performed in the analysis reported in this study.
  Abbreviations: COM, college of osteopathic medicine; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; GPA, grade point average; MCAT, Medical College Admission Test.
×