Scores from COMLEX-USA Level 1, Level 2-CE, and Level 3 revealed statistically significant positive moderate correlations with scores from AOBEM Part I. Scores on COMLEX-USA Level 3 were most highly correlated with AOBEM Part I scores, followed by COMLEX-USA Level 2-CE and Level 1 scores in terms of strength of correlation. This evidence supported the discriminative validity of COMLEX-USA to some extent. During the time between the COMLEX-USA series and AOBEM Part I, the year in which candidates took COMLEX-USA Level 3 was closest to the time frame in which candidates took AOBEM Part I. By contrast, candidates typically took COMLEX-USA Level 1 seven years before they took AOBEM Part I. With a longer interval, more potential factors could change examinees' abilities and performances. In addition, by the time that candidates take COMLEX-USA Level 3, many are already training in postdoctoral programs (eg, emergency medicine residency programs for this sample) and are expected to demonstrate knowledge of clinical concepts and principles necessary for solving medical problems as independently practicing osteopathic generalists. In the present study, examinees' knowledge, skill, and ability to manage clinical problems in the unsupervised practice setting was most comparable to those of AOBEM Part I candidates. By contrast, COMLEX-USA Level 1 focuses more on scientific understanding of health and disease, often referred to as clinically applied foundational biomedical sciences and osteopathic principles. Therefore, we were not surprised that COMLEX-USA Level 1 scores were least correlated and Level 3 scores were most correlated with AOBEM Part I scores.
According to the multiple regression models, COMLEX-USA Level 1 and Level 2-CE scores, which some residency program directors use as part of the selection criteria for residents, together explained 30% of variance in AOBEM Part I scores for the sample of this study. This result is equivalent to .55 as a joint correlation between COMLEX-USA Levels 1 and 2-CE and AOBEM Part I performances. Adding COMLEX-USA Level 3 scores explained 7% of extra variation of AOBEM Part I scores. Considering that “the COMLEX-USA examination series is designed to assess the osteopathic medical knowledge and clinical skills considered essential for osteopathic generalist physicians to practice osteopathic medicine without supervision,”
9(p7) along with the fact that emergency medicine is not generally classified as a primary care discipline, it is understandable that the explained variance was not higher. These results also suggest that COMLEX-USA Level 1 and Level 2-CE could serve as effective and important partial criteria in predicting whether candidates pass or fail AOBEM Part I.
Differences in COMLEX-USA Level 1, 2-CE, and 3 scores were statistically significant among the examinees who passed and the examinees who failed AOBEM Part I. However, when all 3 COMLEX-USA scores were included in the logistic model to predict the odds of passing AOBEM Part I, Level 2-CE scores were not statistically significant. Even though COMLEX-USA Level 2-CE and Level 3 have different emphases, Level 3 was more similar in terms of clinical knowledge and application of principles and management of patient presentations to Level 2-CE relative to Level 1. Because of the similarity (ie, strong correlation) between Level 2-CE scores and Level 3 scores, the model picked the stronger predictor (Level 3). This logistic model may also apply to the individual level: residents can use their Level 3 scores to assess their chance of passing AOBEM Part I with a 95% confidence level. When the predicted chance of passing for a resident is low, the resident may take extra effort in preparing for AOBEM Part I.