Abstract
The Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) assesses the competence of osteopathic physicians in training. It is designed to protect the public by setting minimum competence standards. All osteopathic medical students must pass COMLEX-USA Level 1, Level 2-Cognitive Evaluation, and Level 2-Performance Evaluation before being allowed to graduate from an osteopathic medical school. Residency training programs use COMLEX-USA scores as a major factor in deciding whom they will interview and admit into their programs. In addition, colleges of osteopathic medicine use student COMLEX-USA scores as an external assessment of their success in educating students. Because COMLEX-USA is a high-stakes examination series, it is important to understand predictive factors for performance. The authors review the literature on the relationship between COMLEX-USA scores and correlated student variables. Results from the Council on Osteopathic Student Government Presidents' survey on students' preparation methods and performance are also provided.
Licensure to practice medicine occurs at the state level. Initially, each state or licensing district created and administered their own examination. Because of the need for higher quality and more standardized examinations, organizations such as the National Board of Osteopathic Medical Examiners (NBOME) were established.
1 The NBOME and its allopathic counterpart, the National Board of Medical Examiners, are charged with creating the standardized examinations that ultimately determine licensure for today's physicians. The Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) is the standardized test series designed to assess an osteopathic physician (ie, DO) trainee for a minimum level of competence. The examination evaluates osteopathic medical students in 2 ways based on the Angoff and Hofstee statistical criteria.
2 The Angoff method sets a standard score for items one would need to know to practice osteopathic medicine safely. It uses a cut score that relies on experts who examine test content to predict how a minimally qualified candidate would be expected to perform on an item. The Hofstee method measures students' performance relative to that of other students taking the examination. It is used to set a minimum and maximum failure rate expectation. These 2 criteria are used to validate one another when they produce similar passing scores.
Passage of all 3 levels of COMLEX-USA or the United States Medical Licensing Examination (USMLE) is a major step that osteopathic medical students must complete before receiving a license to practice osteopathic medicine. Osteopathic medical students usually take the COMLEX-USA Levels 1 and 2 during their second and third years in school, respectively. The basic scientific concepts and principles taught in the first 2 years of undergraduate osteopathic medical education are emphasized on COMLEX-USA Level 1. Level 2 comprises 2 separate examinations: a written examination called the Cognitive Evaluation (CE) and a patient encounter examination called the Performance Evaluation. Level 2-CE focuses on biomedical concepts, clinical diagnostic reasoning, and the ability to treat patients. Level 3 of COMLEX-USA is typically taken while in graduate medical education (GME) training.
Although designed to test the competence of physician trainees, licensure examination scores are increasingly being used by GME training programs as a way to screen applicants, and scores are an important factor used to decide whom to accept into those programs. Of 1793 surveyed Accreditation Council for Graduate Medical Education program directors, 94% indicated using USMLE Step 1 or COMLEX-USA Level 1 scores in selecting applicants to interview, with an importance rating of 4.1 out of 5 points.
3 In this same survey, 36% of respondents indicated that they look for COMLEX-USA Level 1 pass scores only, whereas 41% indicated that they look for target scores.
3
In our experience, students today wonder whether their COMLEX-USA scores are more important than the osteopathic medical school they attend or any other factors. The use of COMLEX-USA scores in this way has caused much stress and anxiety for osteopathic medical students. To our knowledge, students use their COMLEX-USA results as a major determinant in potential placement into a GME program, both for specialties to pursue and programs to which they should apply. Adding to this stress is the lack of COMLEX-USA–specific preparation materials. Frequently, students from a specific COM use similar resources, which we believe occurs because either the COM provides them with preparation materials or the students base their study methods on peer-guided recommendations.
In the present article, we review the data currently available for students to use in their preparation methods. These data include individual variables that place students at a greater likelihood of passing or failing, as well as successful preparation methods used to attain certain COMLEX-USA scores on both Level 1 and Level 2-CE. We also summarize findings of a survey on COMLEX-USA preparation and performance.
To provide both COMs and osteopathic medical students with methods to predict how students will perform on licensing examinations, many studies have analyzed predictive values for student outcomes on their examinations taken before entering a COM, such as the Medical College Admission Test (MCAT). Correlations between MCAT scores and undergraduate grade point averages (GPAs) can predict a student's likely success in a COM overall, as well as his or her COMLEX-USA outcomes. Dixon
4 found that physical sciences and biological sciences MCAT subscores were correlated with COMLEX-USA Level 1 performance, whereas verbal reasoning, physical, and biological MCAT subscores were correlated with Level 2 performance. Cope et al
5 used preadmission variables in predictive models and reported that biological MCAT scores and cumulative undergraduate GPA were predictive of COMLEX-USA Level 1 scores. In contrast, Baker et al
6 found no correlations between COMLEX-USA Level 1 scores, preadmission variables, and medical school performance.
Some studies have looked at the relationship between Comprehensive Osteopathic Medical Achievement Test (COMAT) outcomes and COMLEX-USA scores. Li et al
7 found that internal medicine COMAT scores were the strongest predictor of Level 2-CE performance. Lewis et al
8 examined specific course grades in relation to COMLEX-USA outcomes and found a correlation between performance on the second-year osteopathic manipulative medicine examination and COMLEX-USA Level 1 outcomes. In yet another study, Li et al
9 looked at the predictive power of COMLEX-USA performance on resident outcomes and performance on the American Osteopathic Board of Emergency Medicine Part 1 certification examination. They found the greatest correlation with COMLEX-USA Level 3, followed by Level 2-CE, with the least correlation with COMLEX-USA Level 1 scores.
9
Aditya et al
10 looked at predictors for students scoring at least 600 on the COMLEX-USA Level 1 in relation to both correlated student variables and their preparation strategies. A score of 600 or higher was associated with a higher medical school GPA, a higher MCAT score, earlier preparation initiation, and students not ranking the Comprehensive Osteopathic Medical Self-Assessment Examination as the most helpful practice examination.
10 A second regression removing GPA and MCAT scores showed that a score of 600 was associated with earlier initiation of examination preparation and not ranking COMBANK (a question bank for osteopathic examinations) as the most helpful question bank.
10 These findings emphasize the importance of establishing a consistent program of study and maintaining a high GPA throughout medical school, as these factors will most likely lead to successful performance on the COMLEX-USA.
Using the study by Aditya et al
10 as a guide, in 2013, the Council of Osteopathic Student Government Presidents medical education representative (M.J.E.) developed surveys to analyze the most important factors in the first 2 years of medical school for student success, specific licensing examination preparation methods, and outcomes on COMLEX-USA Levels 1 and 2-CE. The survey was reviewed by leadership at the American Association of Colleges of Osteopathic Medicine. The study was reviewed by the institutional review board at the Touro College of Osteopathic Medicine in New York, New York, which determined it to be exempt from approval and informed consent.
The surveys (one for Level 1 and one for Level 2-CE and created using Google Forms) invited all osteopathic medical students who had taken COMLEX-USA in the past year to complete the questionnaires. The surveys were e-mailed to osteopathic medical students by their student government leaders via student e-mail listservs. Participation was not required, and all responses were anonymous. The surveys were conducted in 2013 and in 2014.
Students from 13 schools responded to the surveys, for a total of 399 responses on the COMLEX-USA Level 1 survey and 306 responses on the COMLEX-USA Level 2-CE survey. The mean score of Level 1 respondents was 547 (range, 385-840). The mean score of Level 2-CE respondents was 578 (range, 400-850).
When surveyed regarding COMLEX-USA Level 1, the majority of students reported spending either 3 to 4 weeks or 4 to 8 weeks preparing solely for their licensing examinations. Students who were in the top 20% of their class were found to have a mean COMLEX-USA Level 1 score of 610, whereas all other students were found to have a mean score of 522, indicating that the top 20% of the class was more likely to achieve higher scores on the examination. The top 4 student preparation resources were First Aid for the USMLE, USMLEWorld, COMBANK, and Doctors in Training. Of our respondents, 48% also took the USMLE Step 1, and they had a mean score of 224 (out of a possible 300).
Regarding COMLEX-USA Level 2-CE, the majority of students stated that their COM helped prepare them by using COMATs on clinical rotations. In contrast to Level 1 respondents, Level 2-CE respondents were much more likely to consider their education, or clinical rotations, as their licensing examination preparation and stated that they spent more than 6 weeks preparing for Level 2-CE.
For both COMLEX-USA Levels 1 and 2-CE, many students stressed that the harder one worked in the first 2 years of medical school, the easier it was to prepare for licensing examinations. Others stressed the continued importance of practice tests and practice questions.
Outcomes on Levels 1 and 2 of the COMLEX-USA series are important assessment tools used by COMs and GME programs. Students are therefore placing increasing importance on preparation for these examinations. Previous research has demonstrated the correlation of undergraduate GPA, MCAT scores, medical school GPA, performance on individual medical school examinations, and performance on COMAT with student COMLEX-USA outcomes. More recent research has demonstrated the possible relevance of students' perspectives on the COMLEX-USA and their licensing examination–focused preparation methods to their performance on COMLEX-USA. Our survey data showed that students value practice questions and time for test preparation above other variables.
The outcomes of the Council of Osteopathic Student Government Presidents survey are limited. Our findings provide a snapshot of COMLEX-USA and student perspectives at this time and not a comprehensive review. The study described herein also used students' survey responses only and did not include those of other stakeholders.
Standardized licensing examinations such as the COMLEX-USA series are an integral part of developing competent physicians. Research will likely continue to investigate the most effective preparation tools and predictors for performance outcomes.