Free
Medical Education  |   June 2014
Predictive Relationship of Osteopathic Manual Medicine Grades and COMLEX-USA Level 1 Total Scores and Osteopathic Principles and Practice Subscores
Author Notes
  • From the Department of Osteopathic Manual Medicine (Dr Lewis), the Department of Academic Affairs (Dr Johnson), and the Department of Physiology and Pharmacology (Dr Finnerty) at Des Moines University in Iowa 
  • Address correspondence to Drew D. Lewis, DO, Department of Osteopathic Manual Medicine, Des Moines University, 3200 Grand Ave, Des Moines, IA 50312-4198. E-mail: drew.d.lewis@dmu.edu  
Article Information
Medical Education / Being a DO / COMLEX-USA
Medical Education   |   June 2014
Predictive Relationship of Osteopathic Manual Medicine Grades and COMLEX-USA Level 1 Total Scores and Osteopathic Principles and Practice Subscores
The Journal of the American Osteopathic Association, June 2014, Vol. 114, 480-485. doi:10.7556/jaoa.2014.097
The Journal of the American Osteopathic Association, June 2014, Vol. 114, 480-485. doi:10.7556/jaoa.2014.097
Abstract

Context: Osteopathic manual medicine (OMM) encompasses hands-on diagnosis and treatment as part of patient care. The area of osteopathic principles and practice (OPP) is considered a core competency for students and practitioners of this medical tradition. The Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) is a useful tool for assessing candidates' competency.

Objectives: To examine the relationship of COMLEX-USA Level 1 total scores and OPP subscores with OMM course grades, and to determine if these grades are predictive of COMLEX-USA Level 1 OPP performance.

Methods: The authors collected data—COMLEX-USA Level 1 total and OPP subscores, OMM grades (written, practical, and total for first and second academic years), sex, and age—for a cohort of osteopathic medical students at a single institution, and these data were then analyzed by means of correlation analysis.

Results: Records were obtained from a second-year class of osteopathic medical students (N=217). The authors' analysis of total scores and OPP subscores on COMLEX-USA Level 1 yielded a statistically significant correlation with all variables. Although the correlations were moderate, second-year written examination grades showed the strongest association with the COMLEX-USA Level 1 OPP subscores (r=0.530) and total scores (r=0.566).

Conclusion: Performance in the second-year OMM written examination could identify students potentially at risk for poor performance on COMLEX-USA Level 1.

Osteopathic manual medicine (OMM), also known as osteopathic manipulative medicine and osteopathic principles and practice (OPP), is the aspect of training and practice that most profoundly distinguishes osteopathic physicians from allopathic physicians.1 Osteopathic manual medicine and its clinical application, osteopathic manipulative treatment (OMT), encompass a philosophical approach that values hands-on diagnosis and treatment in addition to standard medical practices as important to providing maximum health care. 
The domain of OMM is 1 of 7 core competencies that the National Board of Osteopathic Medical Examiners' guidelines2 outline as an important core area from which to measure physician competency. 
As the primary pathway to licensure for osteopathic medical students3—and as a national external standard of reference—the Comprehensive Osteopathic Medical Licensure Examination-USA (COMLEX-USA) is a useful tool for measuring osteopathic candidates' competency. For osteopathic medical students, COMLEX-USA Level 1 in particular remains an important milestone in their training process: at our institution and many others, a student must pass this examination before he or she can advance to clinical training. It is also an absolute requirement for graduation. In addition to assessing a student's application of foundational and basic biomedical science knowledge relevant to clinical presentations,3 the COMLEX-USA Level 1 assesses his or her cognitive knowledge in OMM. 
Previous studies, such as that of Hartman et al,4 evaluated the relationship of COMLEX-USA scores to many variables and, above all, helped to demonstrate the validity of the standardized test. In addition, other studies4-8 have found a statistically significant correlation between COMLEX-USA scores and medical school grades. Hartman et al4 found a strong correlation between COMLEX-USA Level 1 and Level 2-Cognitive Evaluation (CE) scores and grades in the first 2 years of osteopathic medical school. Baker et al5 found that academic performance in a student's first 2 years of medical school was strongly associated with performance on COMLEX-USA Level 1. Further, Baker et al6 demonstrated a moderate correlation (0.37) between COMLEX-USA Level 2-Performance Evaluation (PE) examination performance and weighted grades in first- and second-year classes for OPP. Meoli et al7 reported that COMLEX-USA Level 1 scores are good predictors of students' OPP/OMM course performance in the third and fourth academic years; they did not, however, analyze first- and second-year OMM grades. Evans et al8 found a strong correlation with COMLEX-USA Level 2-CE performance and student academic performance, including basic science grade point average and total grade point average in medical school. Other studies have looked at variables that predict performance on COMLEX-USA. In a study9 from 2004 and another10 from 2012, Dixon documented multiple preadmission variables that predict performance on COMLEX-USA Level 1, Level 2-CE, and Level 2-PE. 
To our knowledge, no study has been performed that specifically examines students' grades in OMM or OPP courses during the first and second years of osteopathic medical school compared with their total scores and OPP subscores of COMLEX-USA Level 1. Courses in OMM necessarily involve both cognitive (written examinations) and psychomotor (practical skills testing) approaches to evaluate students' knowledge and skills. 
The primary objective of the present study was to examine the relationship between COMLEX-USA Level 1 performance (total and OPP subscores) with prior performance in first- and second-year OMM courses. The secondary objective of the study was to determine if OMM course grades could identify students at higher risk of poor performance (total score or OPP subscore) on COMLEX-USA Level 1. 
Methods
The present study was reviewed and approved by the university's institutional review board, which deemed the present study's use of preexisting data “noninterventional,” thus exempting the study from the requirement of obtaining informed consent. Study participants were second-year osteopathic medical students at a single college of osteopathic medicine. The total scores and OPP subscores on COMLEX-USA Level 1, course grades from academic years 1 and 2 (written, practical, and total), sex, and age were collected. If a student completed COMLEX-USA Level 1 multiple times, we used his or her first score only. 
The first-year course total grades included a written score compiled from 2 multiple-choice written examinations and a practical score compiled from 4 practical examinations. The second-year total grade scores were compiled in an identical manner. Total score, or grade, was calculated as a percentage of total points earned divided by total possible points. From the first year through the second, the OMM written examinations have both new and review material, making the second-year version effectively cumulative in scope. The practical examinations encompass psychomotor skills tests administered by the OMM department, during which students are required to demonstrate skills developed in the OMM laboratory sessions and honed with hands-on practice. The practical examinations also differ from written examinations because they are largely focused on the most current laboratory material and include only a small review section. For the fourth practical examination of each year, however, students perform a full-body osteopathic structural examination and comprehensive OMT. 
A correlation analysis was used to determine the association of the variables regarding predicting performance on the OPP subtest of COMLEX-USA Level 1, as well as the total test score. Statistical significance was set at α=.05. 
Results
Analysis was conducted for all students with a complete set of data (N=217). The mean (standard deviation [SD]) age for our study group was 27 (3.4) years and the median age was 26.0 years; 110 of 217 participants (51%) were men. 
Table 1 provides an overview of the mean (SD) values of the group for the variables examined. The apparent distribution of the scores for the various assessments was fairly close to normal with the median approximating the mean. 
Table 1.
Mean (SD) and Median Scores of Osteopathic Medical Students (N=217) on First- and Second-Year Examinations and on COMLEX-USA Level 1
Score
Examination Mean (SD) Median
Year 1 Score, %
 Practical 91.7 (4.8) 92.8
 Written 89.4 (5.2) 90.2
 Total 90.6 (4.3) 91.5
Year 2 Score, %
 Practical 94.7 (3.3) 95.5
 Written 85.9 (6.4) 86.7
 Total 91.0 (4.0) 91.7
COMLEX-USA Level 1, OPP Subscore 546.0 (119.5) 531.0
COMLEX-USA Level 1, Total Score 528.9 (87.3) 523.0
  Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Medical Examination-USA; OPP, osteopathic principles and practice; SD, standard deviation.
Table 1.
Mean (SD) and Median Scores of Osteopathic Medical Students (N=217) on First- and Second-Year Examinations and on COMLEX-USA Level 1
Score
Examination Mean (SD) Median
Year 1 Score, %
 Practical 91.7 (4.8) 92.8
 Written 89.4 (5.2) 90.2
 Total 90.6 (4.3) 91.5
Year 2 Score, %
 Practical 94.7 (3.3) 95.5
 Written 85.9 (6.4) 86.7
 Total 91.0 (4.0) 91.7
COMLEX-USA Level 1, OPP Subscore 546.0 (119.5) 531.0
COMLEX-USA Level 1, Total Score 528.9 (87.3) 523.0
  Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Medical Examination-USA; OPP, osteopathic principles and practice; SD, standard deviation.
×
To assess the relationships among the variables, a simple correlation analysis was first performed (Table 2). Although all the correlation coefficients were statistically significant (P<.001), the strength of the associations varied from weak to moderate. Most notably, the OMM course grade for the written examination at year 2 showed the strongest correlation with both the COMLEX-USA Level 1 total score and the OPP subscores (r=0.566 and r=0.530, respectively). 
Table 2.
Correlations (r) of First- and Second-Year Examination Scores and COMLEX-USA Level 1 Total and OPP Subscores of Osteopathic Medical Students (N=217) (P<.001)
Examination OPP Subscore Total Score
Year 1
 Practical 0.305 0.270
 Written 0.449 0.533
 Total 0.456 0.491
Year 2
 Practical 0.276 0.264
 Written 0.530 0.566
 Total 0.506 0.524
  Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Medical Examination-USA; OPP, osteopathic principles and practice.
Table 2.
Correlations (r) of First- and Second-Year Examination Scores and COMLEX-USA Level 1 Total and OPP Subscores of Osteopathic Medical Students (N=217) (P<.001)
Examination OPP Subscore Total Score
Year 1
 Practical 0.305 0.270
 Written 0.449 0.533
 Total 0.456 0.491
Year 2
 Practical 0.276 0.264
 Written 0.530 0.566
 Total 0.506 0.524
  Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Medical Examination-USA; OPP, osteopathic principles and practice.
×
Figure 1 and Figure 2 present scatterplots of the COMLEX-USA Level 1 and second-year written examination scores. A regression analysis of the second-year written examination score with both the COMLEX-USA Level 1 scores was performed and suggested that a second-year written examination score of 70% would result in a predicted COMLEX-USA Level 1 total score or OPP subscore of 400 or less. 
Figure 1.
Scatterplot depicting the relationship between second-year written examination scores and COMLEX-USA Level 1 total scores for osteopathic medical students (N=217). Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; OMM, osteopathic manual medicine; SEE, standard error estimate.
Figure 1.
Scatterplot depicting the relationship between second-year written examination scores and COMLEX-USA Level 1 total scores for osteopathic medical students (N=217). Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; OMM, osteopathic manual medicine; SEE, standard error estimate.
Figure 2.
Scatterplot depicting the relationship between second-year written examination scores and COMLEX-USA Level 1 OPP subscores for osteopathic medical students. Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; OMM, osteopathic manual medicine; OPP, osteopathic principles and practice; SEE, standard error estimate.
Figure 2.
Scatterplot depicting the relationship between second-year written examination scores and COMLEX-USA Level 1 OPP subscores for osteopathic medical students. Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; OMM, osteopathic manual medicine; OPP, osteopathic principles and practice; SEE, standard error estimate.
Discussion
Although it is primarily a licensure examination, COMLEX-USA Level 1 can also be used as an external assessment to evaluate the level of preparation that a medical school has provided for its students. In this way, examination performance can inspire program improvement efforts. 
Whereas the correlations in the study varied from weak to moderate in both class years, the strongest correlation occurred—for both the COMLEX-USA Level 1 total scores—with the second-year written examination scores grades. 
It is not entirely surprising that the written examination scores correlated more strongly with COMLEX-USA Level 1 scores than practical examination scores: the written examination assesses understanding and knowledge of basic science concepts,3 as opposed to the practical examinations, which were largely focused on the most current laboratory material. Each practical examination contained a small comprehensive section, but only the final practical examination for each OMM course evaluates the student's ability to provide a full-body osteopathic structural examination and OMM treatment. The second-year OMM total score contains the OMM practical score as 1 component, which would help to explain the lower correlation found in the analysis of year 2 testing (Table 2). A limited spread of scores reduces the discrimination between top-level achievement and average achievement and is a known challenge of performing analysis on skills-based testing (ie, practical examinations).11 The development of more consistent scoring rubrics for OMM practical skills assessment may reduce subjectivity and benefit from further study. In addition, future researchers should examine the predictive value of OMM practical scores and scores on COMLEX-USA Level 2-PE. 
The written examinations were constructed to include comprehensive review questions from previous examination materials. Thus, with each subsequent test students are challenged with a more comprehensive question set. Therefore, the written examinations in general—and the second (ie, final) written examination for second-year students in particular—would be the most inclusive of content tested on the board examinations. Finally, because students studied for both examinations simultaneously, there is a temporal relationship between the second written examination for second-year students and COMLEX-USA Level 1. This effect may contribute to the moderate and strong correlation between these 2 examinations. Whereas the second-year written examination is not an overly strong predictor of COMLEX-USA Level 1 performance, whether total score (r2=0.320) or OPP subscore (r2=0.281), poor performance on a course-based examination suggests a marginal, at best, performance on COMLEX-USA Level 1. 
Further research on relationships between OMM course grades and COMLEX-USA performance may prove useful for internal purposes. An osteopathic medical education program may use data from such research to evaluate if the OPP curriculum and assessment is a reflection of the broad concepts tested by COMLEX-USA and if courses are consistent with the profession-wide consensus on competency in this important domain. 
According to Hartman et al,4 a psychometrically sound licensing examination for physicians is expected to correlate strongly with medical school curriculum.4 The statistically significant correlations between first- and second-year course grades and COMLEX-USA Level 1 OPP subscores support the construct validity of the latter for assessing mastery of a broad preclinical OMM curriculum. In addition, it reciprocally supports the validity of an OMM curriculum for preparing students for COMLEX-USA Level 1. 
One confounding issue for researchers attempting to correlate performance data is the difficulty in accounting for learning that occurs outside the OMM course, whether by means of overlapping medical school curricula (eg, anatomy, physiology, orthopedics, physical medicine and rehabilitation, sports medicine) or of outside or extracurricular knowledge (eg, board preparation courses, review texts, question banks). To attribute performance on COMLEX-USA to preparation within a particular course is problematic, even with coefficients of determination found in the present study (32% for total score and 28% for OPP subscore). Further, students who tend to perform well on COMLEX-USA tend to do so in all areas. A retrospective analysis of student outcomes across multiple years at our institution has revealed that students in the upper quartile of class rank perform well in all areas tested on COMLEX-USA Level 1 (internal data, not shown). 
The generalizability of the present study is limited because the data were derived from 1 class of students at 1 osteopathic medical school. Another limitation is that OMM course content, curricular structure, and assessments, while generally similar, will vary among the colleges. 
Areas of Future Research
In addition to osteopathic medical schools, future research might also focus on any of the following items that comprise a school's curriculum: numbers of didactic and laboratory hours, numbers of faculty, faculty-to-student ratio in hands-on teaching, and curricular content. Evaluating the relationship between OMM curricula and COMLEX-USA Level 1 performance at other COMs is warranted. Researchers could focus on the relationships of preclinical OMM course scores and OMM course scores during the clinical training with OPP subscore performance on COMLEX-USA Level 2-CE and passing rate on COMLEX-USA Level 2-PE. Evaluation of best practices for delivery of an OMM curriculum may involve not only an analysis of the relationship between OMM grades and COMLEX-USA scores, but also ranking osteopathic medical school's OPP subscores to identify curricula content and delivery methods that best prepare future osteopathic physicians for licensing examinations. 
Conclusion
There is a correlation between second-year OMM written examination scores and COMLEX-USA Level 1 performance. Such markers could be useful in an institutional strategy to support osteopathic medical student success through identification of those who are at risk for poor performance on COMLEX-USA. 
   Financial Disclosures: None reported.
 
   Support: None reported.
 
   Editor's Note: In this article, the authors use the term osteopathic manual medicine to describe the application of osteopathic philosophy, structural diagnosis, and use of osteopathic manipulative treatment in the diagnosis and management of the patient. The style guidelines of The Journal of the American Osteopathic Association and AOA policy prefer the term osteopathic manipulative medicine. The authors believe that the term osteopathic manual medicine is more appropriate because it is more encompassing than osteopathic manipulative medicine.
 
References
What is a DO? American Osteopathic Association website. https://www.osteopathic.org/osteopathic-health/about-dos/what-is-a-do/Pages/default.aspx. Accessed January 31, 2013.
National Board of Osteopathic Medical Examiners. Fundamental Osteopathic Medical Competency Domains: Guidelines for Osteopathic Medical Licensure and the Practice of Osteopathic Medicine. Conshohocken, PA: National Board of Osteopathic Medical Examiners; June 2011. http://www.nbome.org/docs/NBOME%20Fundamental%20Osteopathic%20Medical%20Competencies.pdf. Accessed July 22, 2012.
National Board of Osteopathic Medical Examiners. COMLEX-USA Bulletin of Information: 2013-2014. Conshohocken, PA: National Board of Osteopathic Medical Examiners; July 1 , 2013. http://www.nbome.org/docs/comlexBOI.pdf. Accessed August 31, 2013.
Hartman SE, Bates BP, Sprafka SA. Correlation of scores for the Comprehensive Osteopathic Medical Licensing Examination with osteopathic medical school grades. J Am Osteopath Assoc. 2001;101(6):347-349. http://www.jaoa.org/content/101/6/347.long. Accessed April 30, 2012.
Baker HH, Foster RW, Bates BPet al. Relationship between academic achievement and COMLEX-USA Level 1 performance: a multisite study. J Am Osteopath Assoc. 2000;100(4):238-242. http://www.jaoa.org/content/100/3/153.full.pdf+html. Accessed April 30, 2012.
Baker HH, Cope MK, Adelman MD, Schuler S, Foster RW, Gimpel JR. Relationship between scores on the COMLEX-USA Level 2-Performance Evaluation and selected school-based performance measures. J Am Osteopath Assoc. 2006;106(5):290-295. http://www.jaoa.org/content/106/5/290.full. Accessed April 30, 2012.
Meoli FG, Wallace WS, Kaiser-Smith J, Shen L. Relationship of osteopathic medical licensure examinations with undergraduate admission measures and predictive value of identifying future performance in osteopathic principles and practice/osteopathic manipulative medicine courses and rotations. J Am Osteopath Assoc. 2002;102(11):615-620. http://www.jaoa.org/content/102/11/615.long. Accessed April 30, 2012.
Evans P, Goodson L, Schoffman S. Relationship between academic achievement and student performance on the Comprehensive Osteopathic Medical Licensing Examination-USA Level 2. J Am Osteopath Assoc. 2003;103(7):331-336. http://www.jaoa.org/content/103/7/331.full.pdf+html. Accessed April 30, 2012.
Dixon D. Relation between variables of preadmission, medical school performance, and COMLEX-USA Levels 1 and 2 performance. J Am Osteopath Assoc. 2004;104(8):332-336. http://www.jaoa.org/content/104/8/332.full. Accessed April, 30, 2012.
Dixon D. Prediction of osteopathic medical school performance on the basis of MCAT Score, GPA, sex, undergraduate major, and undergraduate institution. J Am Osteopath Assoc. 2012;112(4):175-181. http://www.jaoa.org/content/112/4/175.full. Accessed September 13, 2012.
Smee S. Skill based assessment [review]. BMJ. 2003;326(7391):703-706. doi:10.1136/bmj.326.7391.703. [CrossRef] [PubMed]
Figure 1.
Scatterplot depicting the relationship between second-year written examination scores and COMLEX-USA Level 1 total scores for osteopathic medical students (N=217). Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; OMM, osteopathic manual medicine; SEE, standard error estimate.
Figure 1.
Scatterplot depicting the relationship between second-year written examination scores and COMLEX-USA Level 1 total scores for osteopathic medical students (N=217). Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; OMM, osteopathic manual medicine; SEE, standard error estimate.
Figure 2.
Scatterplot depicting the relationship between second-year written examination scores and COMLEX-USA Level 1 OPP subscores for osteopathic medical students. Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; OMM, osteopathic manual medicine; OPP, osteopathic principles and practice; SEE, standard error estimate.
Figure 2.
Scatterplot depicting the relationship between second-year written examination scores and COMLEX-USA Level 1 OPP subscores for osteopathic medical students. Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; OMM, osteopathic manual medicine; OPP, osteopathic principles and practice; SEE, standard error estimate.
Table 1.
Mean (SD) and Median Scores of Osteopathic Medical Students (N=217) on First- and Second-Year Examinations and on COMLEX-USA Level 1
Score
Examination Mean (SD) Median
Year 1 Score, %
 Practical 91.7 (4.8) 92.8
 Written 89.4 (5.2) 90.2
 Total 90.6 (4.3) 91.5
Year 2 Score, %
 Practical 94.7 (3.3) 95.5
 Written 85.9 (6.4) 86.7
 Total 91.0 (4.0) 91.7
COMLEX-USA Level 1, OPP Subscore 546.0 (119.5) 531.0
COMLEX-USA Level 1, Total Score 528.9 (87.3) 523.0
  Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Medical Examination-USA; OPP, osteopathic principles and practice; SD, standard deviation.
Table 1.
Mean (SD) and Median Scores of Osteopathic Medical Students (N=217) on First- and Second-Year Examinations and on COMLEX-USA Level 1
Score
Examination Mean (SD) Median
Year 1 Score, %
 Practical 91.7 (4.8) 92.8
 Written 89.4 (5.2) 90.2
 Total 90.6 (4.3) 91.5
Year 2 Score, %
 Practical 94.7 (3.3) 95.5
 Written 85.9 (6.4) 86.7
 Total 91.0 (4.0) 91.7
COMLEX-USA Level 1, OPP Subscore 546.0 (119.5) 531.0
COMLEX-USA Level 1, Total Score 528.9 (87.3) 523.0
  Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Medical Examination-USA; OPP, osteopathic principles and practice; SD, standard deviation.
×
Table 2.
Correlations (r) of First- and Second-Year Examination Scores and COMLEX-USA Level 1 Total and OPP Subscores of Osteopathic Medical Students (N=217) (P<.001)
Examination OPP Subscore Total Score
Year 1
 Practical 0.305 0.270
 Written 0.449 0.533
 Total 0.456 0.491
Year 2
 Practical 0.276 0.264
 Written 0.530 0.566
 Total 0.506 0.524
  Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Medical Examination-USA; OPP, osteopathic principles and practice.
Table 2.
Correlations (r) of First- and Second-Year Examination Scores and COMLEX-USA Level 1 Total and OPP Subscores of Osteopathic Medical Students (N=217) (P<.001)
Examination OPP Subscore Total Score
Year 1
 Practical 0.305 0.270
 Written 0.449 0.533
 Total 0.456 0.491
Year 2
 Practical 0.276 0.264
 Written 0.530 0.566
 Total 0.506 0.524
  Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Medical Examination-USA; OPP, osteopathic principles and practice.
×