Abstract
Context: The relationship between the Comprehensive Osteopathic Medical Achievement Test (COMAT) series of subject examinations and the Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Cognitive Evaluation (COMLEX-USA Level 2-CE) has not been thoroughly examined.
Objective: To investigate the factors associated with performance on COMAT subject examinations and how COMAT scores correlate with COMLEX-USA Level 2-CE scores.
Methods: We examined scores of participants from 2 COMAT examination cycles in 2011 and 2012. According to surveys, most schools used COMAT scores in clerkship and clinical rotation evaluation, which were classified as being used for “high-stakes” purposes. We matched first-attempt COMAT scores with first-attempt COMLEX-USA Level 2-CE scores, and we conducted correlation analyses between the scores from the 7 COMAT subject examinations, as well as between COMAT and COMLEX-USA Level 2-CE scores. Multiple linear regression analyses were performed to investigate how much variance in COMLEX-USA Level 2-CE scores was explained by COMAT scores.
Results: In 2011 and 2012, respectively, 3751 and 3786 COMAT candidates had COMLEX-USA Level 2-CE scores (53.0% and 93.9%, respectively, had ⩾1 high-stakes COMAT score). Intercorrelations between COMAT scores were low to moderate (r=0.27-0.53), as hypothesized. Correlations between COMAT and Level 2-CE scores were moderate to high, with the highest correlations for internal medicine COMAT scores (r=0.63-0.65). All regressions showed internal medicine scores as the strongest predictor of Level 2-CE performance. Groups with high-stakes scores had larger adjusted coefficients of determination than those with low-stakes scores (eg, R2=0.63 vs 0.52, respectively, in 2011). For 2012 candidates with high-stakes scores, all predictors were statistically significant.
Conclusion: The COMAT subject examination scores were moderately intercorrelated, as hypothesized, with higher correlations between COMAT and COMLEX-USA Level 2-CE scores. The COMAT performance was predictive of COMLEX-USA Level 2-CE performance.
During years 3 and 4 of osteopathic medical school, in addition to coursework and licensure examinations, students typically move through various clinical clerkships and rotations to gain patient-centered experience as practitioners of osteopathic medicine. In the past, most osteopathic medical schools used postrotation objective examinations and global rating tools completed by clinical preceptors to determine clerkship grades and criteria for academic promotion, but residency program directors expressed a need for standardized end-of-clerkship rotation examinations that were secure, integrated osteopathic principles, and provided direct evidence of student learning, along with some measure of educational equivalency at often-varied clinical education sites. In response, the National Board of Osteopathic Medical Examiners (NBOME) developed and began administration of the Comprehensive Osteopathic Medical Achievement Test (COMAT) series of subject examinations in 2011. These COMAT examinations were the first osteopathic achievement tests to feature not only examination blueprints, but also learning objectives and recommended teaching and learning resources. Currently, the NBOME offers COMAT examinations in the following 7 disciplines: family medicine, internal medicine, obstetrics-gynecology, osteopathic principles and practice (OPP), pediatrics, psychiatry, and surgery. A COMAT emergency medicine examination is expected to be released in 2015.
The Comprehensive Medical Licensing Examination (COMLEX-USA) is a series of assessments designed by the NBOME for the licensing of osteopathic physicians in the United States. The American Osteopathic Association Commission on Osteopathic College Accreditation requires that all osteopathic medical students pass the COMLEX-USA Level 1, Level 2-Cognitive Evaluation (CE), and Level 2-Performance Evaluation (PE) before graduation.
1 The Level 1 examination emphasizes the scientific concepts and principles necessary for understanding the mechanisms of health, clinical problems, and disease processes. Level 2-CE emphasizes the biomedical concepts and principles necessary for making appropriate clinical diagnoses based on patient history and physical examination findings. Level 2-PE is the associated standardized patient-based clinical skills component of the COMLEX-USA series. A passing score on these examinations indicates that the candidate's application of osteopathic medical knowledge and clinical skills have met an acceptable minimal competency standard for entry into graduate medical education (ie, graduation from osteopathic medical school). The ensuing final examination in the COMLEX-USA series (Level 3) is typically taken during the candidate's residency training.
The COMAT subject examinations primarily consist of single-best-answer questions in a clinical scenario format comparable to that of the COMLEX-USA series. At the time of the current study, COMAT examinations included 100 questions presented in a single 2-hour session for each content area. (Current COMAT examinations have 125 questions presented in 2½ hours.) The COMLEX-USA Level 2-CE includes 400 items delivered in 8 sections over 8 hours. Both series of examinations produce nationally standardized scores for reporting. The COMAT scores have a mean (SD) of 100 (10), and the COMLEX-USA Level 2-CE scores have a mean (SD) of 500 (100), with a passing score set at 400. As achievement tests, COMAT examinations do not have passing scores.
The NBOME initiated the current 3-level COMLEX-USA series beginning in 1995, and several studies have substantiated its validity and predictive capabilities.
2-7 Because COMAT is a relatively new series of examinations, this is to our knowledge the first published study of it, making our findings especially important.
The purpose of the current study was to determine the interrelationship between performance on individual COMAT examinations and between COMAT and COMLEX-USA Level 2-CE performance. On the basis of seminal research in medical education,
8 researchers have posited that achievement in a given domain, or subject area, is not entirely explainable by the transfer of some general knowledge between domains.
9 Rather, performance is content specific, and a physician's knowledge and skills are clustered within a specific domain. Therefore, performance in one subject area would not necessarily strongly predict performance in another distinct subject area. Research findings supports this theory
10 but also suggest that transferable skills may be important in explaining related performance.
11 Consequently, a measure of a distinct subject area should have a stronger relationship to a comprehensive measure than to a measure of an associated subject.
Given the theoretical framework guiding our study, we proposed the following 4 hypotheses:
-
COMAT subject examination scores will be intercorrelated, as well as positively correlated with COMLEX-USA Level 2-CE scores.
-
Intercorrelations between COMAT subject examination scores will be lower than the correlations between COMAT subject examination scores and the COMLEX-USA Level 2-CE scores.
-
Correlations between COMAT and COMLEX-USA Level 2-CE scores will be stable from one testing cycle to another.
-
COMAT scores used in the evaluation of clerkship or clinical rotation performance will be stronger predictors of COMLEX-USA Level 2-CE scores than COMAT scores not used for these purposes.
For the current study we selected participants from 2 COMAT cycles in 2011 and 2012. The 2011 cycle, the initial cycle for COMAT, extended from July 1, 2011, to June 30, 2012, and the 2012 cycle from July 1, 2012, to June 30, 2013. According to surveys conducted by the NBOME in 2011 and 2012, most osteopathic medical schools used COMAT scores either as part of students' final grade or for a pass-fail decision in their clerkships and clinical rotations. However, a few schools used COMAT only for formative purposes and did not ascribe COMAT scores to actual performance; this applied to 39.6% of the COMAT administrations in the 2011 cycle. For the 2012 cycle, a larger proportion of schools used COMAT scores to assess performance, with only 8.3% of the examinations used formatively. For the current study, we classified the scores from schools who used COMAT for clerkship or clinical rotation grades or pass-fail determination as being used for “high-stakes” purposes, and we classified the rest as being used for “low-stakes” purposes. We used only existing data that were properly obtained; participants could not be identified, directly or statistically. Therefore, the research was exempt from institutional review board approval.
After first-attempt COMAT score records were matched with first-attempt COMLEX-USA Level 2-CE scores, we conducted correlation analyses between the scores from the 7 COMAT subject examinations and between COMAT and COMLEX-USA Level 2-CE scores for the following 4 groups: (1) 2011 low-stakes administrations; (2) 2011 high-stakes administrations; (3) 2012 low-stakes administrations; and (4) 2012 high-stakes administrations. We also performed a simultaneous multiple linear regression analysis of the groups to determine how much of the variance in COMLEX-USA Level 2-CE scores was explained by COMAT scores. All analyses were conducted using SAS/STAT software (SAS for Windows, version 9.3; SAS Institute). The P value cutoff (α) was set a priori to .05 for all tests of statistical significance.
Table 1 lists the numbers of COMAT subject examination administrations, COMAT examinees, and COMAT examinees with matched COMLEX-USA Level 2-CE scores for 2011 and 2012. A total of 3897 examinees from 22 colleges of osteopathic medicine and 5625 from 24 colleges of osteopathic medicine took at least 1 COMAT subject examination in 2011 and 2012, respectively, and 2023 and 4640 examinees took at least 1 COMAT subject examination for high-stakes purposes in 2011 and 2012, respectively. Using NBOME identifiers, we matched first-attempt COMAT scores with a candidate's first-attempt COMLEX-USA Level 2-CE scores.
Typically, candidates take COMLEX-USA Level 2-CE in the year after their COMAT subject examinations. For our purposes, COMAT scores recorded after Level 2-CE scores or without matched Level 2-CE scores were dropped from the analysis. As a consequence, 3751 COMAT candidates in 2011 and 4069 in 2012 had Level 2-CE scores, with 1989 (53.0%) and 3803 (93.5%), respectively, having taken at least 1 COMAT subject examination for high-stakes purposes. Although both low- and high-stakes groups were included in subsequent analysis for the 2011 cycle, only the high-stakes group was included in the 2012 analysis, owing to the limited number of observations (n=266) in the 2012 low-stakes group.
Table 2 provides Pearson product-moment correlation results for the 7 high-stakes COMAT subject examinations in 2012. Intercorrelations between COMAT scores were moderate (
r=0.27-0.54).
Table 3 lists the correlations between COMAT and COMLEX-USA Level 2-CE scores for 2011 and 2012. Results indicated generally moderate to high correlations, plus a consistent pattern in relative strength of correlations in subjects from the 2011 to the 2012 testing cycle, showing that Level 2-CE scores had the highest correlation with scores for internal medicine (
r=0.62-0.65), followed by surgery, family medicine, obstetrics-gynecology, and pediatrics (
r=0.52-0.61) and then psychiatry (
r=0.50-0.52), with the lowest correlation to scores for OPP (
r=0.38-0.45).
In general, for 2011, high-stakes scores had higher correlations across subjects than low-stakes scores. However, COMAT OPP scores had lower correlations with COMLEX-USA Level 2-CE scores in the 2011 high-stakes group (r=0.38) than in the 2011 low-stakes group (r=0.45), probably because of a substantial difference in sample size, from 1556 to 749.
Table 4 provides results from regression analyses with scores from 7 COMAT subject examinations used to predict COMLEX-USA Level 2-CE scores for both low- and high-stakes groups in 2011 and the high-stakes group in 2012. All 3 regression analyses showed scores from internal medicine as the strongest predictors of Level 2-CE performance, whereas OPP scores were not a significant predictor in 2011 but became significant in 2012. Both years had similar amounts of variance explained by 7 COMAT subject examinations for the high-stakes groups (adjusted
R2=0.63 for both 2011 and 2012), substantially more than for the 2011 low-stakes group (adjusted
R2=0.47).
In the current study, we investigated the relationship between the COMAT subject examination scores and the COMLEX-USA Level 2-CE scores through correlational and regression analyses. We used data from the 2011 and 2012 COMAT cycles, focusing on students with scores in the high-stakes category whose schools used COMAT subject examinations as part of clerkship evaluations.
The results confirmed our hypotheses that intercorrelations between COMAT subject examination scores were lower than the correlations between those scores and scores for the comprehensive COMLEX-USA Level 2-CE. One reason for the lower intercorrelations might be the number of items in the COMAT subject examinations, which is a quarter of the number of items in the COMLEX-USA series, which could affect the internal consistency of the shorter examinations. The intercorrelations we found were similar to those found for comparable allopathic subject and licensure examination scores in a smaller sample.
12 Transferable skills may be important contributors to performance, as suggested elsewhere.
11 In light of the test blueprints, one would expect correlations between COMAT and COMLEX-USA Level 2-CE scores to be higher than correlations between any 2 distinct subject areas, because COMLEX-USA comprises several disciplines, some of which overlap with the COMAT subject examinations.
All samples showed moderate to high correlation between COMAT scores and Level 2-CE scores. Generally, primary care subjects, such as internal medicine and family medicine, had higher correlations with Level 2-CE scores than non–primary care subjects, such as psychiatry. This finding may arise from the fact that although all the test blueprints are osteopathically oriented, the content domain for psychiatry, for example, differs distinctly from those for the other COMAT subject examinations or COMLEX-USA Level 2-CE. Furthermore, to our knowledge, some schools have reported using the COMAT OPP examination either as a comprehensive examination for their OPP curriculum at the end of the second year or as a third-year end-of-clerkship examination. This variable use of COMAT OPP scores probably contributes to the lowest observed intercorrelations of the 7 subjects. In contrast, COMAT surgery scores showed higher correlation with Level 2-CE scores than did COMAT family medicine scores.
A comparable study investigated the relationship between scores on the National Board of Medical Examiners (NBME) subject examinations and the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge.
12 The researchers found that the NBME surgery examination scores had a higher correlation with USMLE Step 2 Clinical Knowledge scores than did the NBME family medicine scores but a lower correlation than the NBME internal medicine scores. Overall, the correlational results from the USMLE study
12 are similar to those in the current study.
Arguably, when students take COMAT for high-stakes purposes, as for use in grading and determining pass-fail status for clerkships and clinical rotations, their COMAT performance will probably better reflect their actual ability and skills. Subsequently, COMAT scores in a high-stakes group would probably better predict COMLEX-USA Level 2-CE scores. The comparison of results between high-stakes and low-stakes groups strongly support this supposition, especially for the 2011 sample, which included a sizeable proportion of students (47.0%) who were not in a high-stakes scenario. When we removed the low-stakes scores from the analysis, the results indicated generally higher correlations with Level 2-CE scores and explained more of the variance in Level 2-CE scores. These findings suggest that schools using the COMAT subject examinations for high-stakes purposes can better predict COMLEX-USA Level 2-CE performance.
Even though we found that COMAT scores had moderate-to-high correlations with COMLEX-USA Level 2-CE scores and that COMAT performance explained a substantial amount of variance in Level 2-CE performance, 37% of the variance was still unexplained by the current models. As a future study, a multilevel analysis with students nested in schools may be considered, wherein variation within schools (eg, rotation length, rotation timing, and various uses of COMAT scores) can be further analyzed. To determine why the lowest correlations were between COMLEX-USA Level 2-CE and COMAT OPP scores, a follow-up study investigating regional differences in OPP curricula and possible effects on item-level performances is forthcoming.
The results of the current study show that, as we hypothesized, COMAT subject examination scores were moderately intercorrelated, with higher correlations between the COMAT and the COMLEX-USA Level 2-CE scores. Further, COMAT performance was predictive of COMLEX-USA Level 2-CE performance.