Free
JAOA/AACOM Medical Education  |   February 2017
Comparison of Basic Science Knowledge Between DO and MD Students
Author Notes
  • From the Touro University College of Osteopathic Medicine-CA, in Vallejo, California. 
  •  *Address correspondence to Glenn E. Davis, MS, Touro University College of Osteopathic Medicine-CA, 1310 Club Dr, Vallejo, CA 94592-1159. E-mail: glenn.davis@tu.edu
     
Article Information
Medical Education / Medical School Admissions / COMLEX-USA
JAOA/AACOM Medical Education   |   February 2017
Comparison of Basic Science Knowledge Between DO and MD Students
The Journal of the American Osteopathic Association, February 2017, Vol. 117, 114-123. doi:10.7556/jaoa.2017.022
The Journal of the American Osteopathic Association, February 2017, Vol. 117, 114-123. doi:10.7556/jaoa.2017.022
Abstract

Context: With the coming single accreditation system for graduate medical education, medical educators may wonder whether knowledge in basic sciences is equivalent for osteopathic and allopathic medical students.

Objective: To examine whether medical students’ basic science knowledge is the same among osteopathic and allopathic medical students.

Methods: A dataset of the Touro University College of Osteopathic Medicine-CA student records from the classes of 2013, 2014, and 2015 and the national cohort of National Board of Medical Examiners Comprehensive Basic Science Examination (NBME-CBSE) parameters for MD students were used. Models of the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 1 scores were fit using linear and logistic regression. The models included variables used in both osteopathic and allopathic medical professions to predict COMLEX-USA outcomes, such as Medical College Admission Test biology scores, preclinical grade point average, number of undergraduate science units, and scores on the NBME-CBSE. Regression statistics were studied to compare the effectiveness of models that included or excluded NBME-CBSE scores at predicting COMLEX-USA Level 1 scores. Variance inflation factor was used to investigate multicollinearity. Receiver operating characteristic curves were used to show the effectiveness of NBME-CBSE scores at predicting COMLEX-USA Level 1 pass/fail outcomes. A t test at 99% level was used to compare mean NBME-CBSE scores with the national cohort.

Results: A total of 390 student records were analyzed. Scores on the NBME-CBSE were found to be an effective predictor of COMLEX-USA Level 1 scores (P<.001). The pass/fail outcome on COMLEX-USA Level 1 was also well predicted by NBME-CBSE scores (P<.001). No significant difference was found in performance on the NBME-CBSE between osteopathic and allopathic medical students (P=.322).

Conclusion: As an examination constructed to assess the basic science knowledge of allopathic medical students, the NBME-CBSE is effective at predicting performance on COMLEX-USA Level 1. In addition, osteopathic medical students performed the same as allopathic medical students on the NBME-CBSE. The results imply that the same basic science knowledge is expected for DO and MD students.

A collaboration between the JAOA and the American Association of Colleges of Osteopathic Medicine (AACOM) to recruit, peer review, publish, and distribute research and other scholarly articles related to osteopathic medical education.

Keywords: basic science, COMLEX-USA, medical school, USMLE

Basic science knowledge supports general physician competence. A foundational component of medical knowledge and competence as measured by medical licensing examinations, basic science knowledge is essential for clinical reasoning and is necessary to optimize patient outcomes.1-4 The American Osteopathic Association and the Accreditation Council for Graduate Medical Education both define medical knowledge as a core physician competency that the curricula of all accredited medical schools must address.5-7 It is also a major component relevant to Entrustable Professional Activities for Entering Residency.8 
Proficiency in basic science subjects such as anatomy, behavioral science, biochemistry, microbiology, pathology, pharmacology, and physiology constitutes a measurable underlying component of medical knowledge competence.1-4 The Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 1 comprises 70% to 85% of medical knowledge assessment, and the United States Medical Licensing Examination (USMLE) Step 1 comprises 55% to 65% of medical knowledge assessment.2-4 
The relevance of basic science to medical knowledge and Entrustable Professional Activities for Entering Residency—and its large representation on licensing examinations—has led to research correlating early indicators of basic science knowledge with licensing examination outcomes. For example, Medical College Admission Test (MCAT) scores and grade point average (GPA) in the first 2 years of medical school (preclinical GPA), when basic science knowledge is often prioritized, have been shown to predict COMLEX-USA Level 1 and USMLE Step 1 outcomes.9-20 The biology score on the MCAT was shown to predict COMLEX-USA Level 1 performance.12 Performance on the National Board of Medical Examiners Comprehensive Basic Sciences Examination (NBME-CBSE), a basic science examination focused on general principles, normal and abnormal processes, therapeutic principles, and behavioral medicine, has proven successful in predicting USMLE Step 1 performance.21,22 To our knowledge, no studies have compared NBME-CBSE performance between osteopathic (ie, DO) and allopathic (ie, MD) medical students or correlated performance on this examination to COMLEX-USA Level 1 outcomes. 
With the upcoming single accreditation system for graduate medical education, it is important to know whether physicians’ expectations for basic science knowledge are the same between DO and MD students. Comparing basic science knowledge among students attending DO and MD schools is challenging for many reasons, including distinctive philosophies and variations in curricula. To analyze basic science knowledge, we chose to compare NBME-CBSE performance between DO and MD students and to compare DO student performance on this examination with their COMLEX-USA Level 1 outcomes. We hypothesized that basic science knowledge is similar for DO and MD students. 
Methods
The Touro University California institutional review board found this study to be exempt under federal human research protection guidelines. Following federal guidelines, a deidentified dataset of Touro University College of Osteopathic Medicine-CA (TUCOM) student grades, COMLEX-USA Level 1 scores, and NBME-CBSE scores, was retrieved from an internally maintained student data warehouse. National NBME-CBSE parameters for MD students were obtained for the same period.23 
Multiple linear regression models were studied in a backward stepwise process to identify 2 optimal models of COMLEX-USA Level 1 scores: 1 using preadmission variables and 1 using preclinical medical school variables.24(pp662-663) Preclinical medical school variables were then added to the optimal preadmission model in a forward stepwise process.24(p662) Model and variable statistics were compared at each step. Optimal models were identified as those with a balance of low model P value, maximal adjusted r2 (percentage of variance in the dependent variable explained by the model), minimal root mean squared error (MSE), and low independent variable P values.24(pp662-663) Variance inflation factor (VIF) was calculated to determine whether collinearity (highly correlated independent variables, or those that are a linear transformation of one another) distorted results.25 
Multiple logistic regression was then applied in a forward stepwise process to study models of COMLEX-USA Level 1 pass/fail outcomes.24(p662) Model and variable statistics were compared at each step. Receiver operating characteristic (ROC) curves and areas under the curve were used to compare and quantify variable effectiveness at predicting COMLEX-USA Level 1 pass/fail outcomes.26 A 2-tailed t test was used to compare mean TUCOM student NBME-CBSE scores with the national cohort. 
All regression statistics and ROC analyses were calculated and compared at the 99% confidence level. All statistical analyses were performed in Stata/MP 13.1 (StataCorp LP). 
Results
The dataset included 392 records of TUCOM students from 2011 to 2013 (classes of 2013-2015). Students were required to take the NBME-CBSE at the end of their second-year courses, before additional board preparation and before attempting COMLEX-USA Level 1. Two records without a first-attempt COMLEX-USA Level 1 score were eliminated, yielding a sample with all data points accounted for (N=390). 
Modeling COMLEX-USA Level 1 Scores
Linear regression is used to predict the values of 1 variable in terms of 1 or more others. Stepwise linear regression is a process to identify an optimal predictive model by either adding or removing variables 1 at a time and comparing resulting changes statistically.24(p454) 
The optimal preadmission model of COMLEX-USA Level 1 scores (Table 1, model 1) consisted of MCAT biology scores and undergraduate science units. The latter was inversely correlated such that on average, each 0.8 unit decrease in undergraduate science units yielded a 1 unit increase in COMLEX-USA Level 1 score. Preadmission variables eliminated during optimization were: undergraduate science units and total GPA, total MCAT score, and additional MCAT component scores. 
Table 1.
Linear Regression Results for Scores on the Comprehensive Osteopathic Medical Licensing Examination-USA Level 1 (N=390)
Modela
1 2 3 4 5 6 7
Variable
MCAT biology scoreb P<.001 NS NS NS
Undergraduate science unitsc P<.001 NS NS NS
NBME-CBSE scored P<.001 P<.001 P<.001 P<.001
Preclinical GPAd P<.001 P<.001 P<.001 P<.001
Model Statistic
P valueb <.001 <.001 <.001 <.001 <.001 <.001 <.001
Adjusted r2c 0.0733 0.6639 0.5915 0.7420 0.7418 0.6635 0.5785
Root MSEd 80.571 48.519 53.497 42.51 42.528 48.551 54.337

a Models: 1, best preadmission model; 2, preadmission with NBME-CBSE (National Board of Medical Examiners Comprehensive Basic Science Examination); 3, preadmission with preclinical GPA (grade point average); 4, all variables; 5, best preclinical medical school model; 6, NBME-CBSE alone; 7, preclinical GPA alone.

b Statistical significance of the model measured as probability that independent variable coefficients equal 0.

c Fraction of variance explained by model.

d Smaller error indicates better fit to data.

Abbreviations: MSE, mean squared error; NS, not significant.

Table 1.
Linear Regression Results for Scores on the Comprehensive Osteopathic Medical Licensing Examination-USA Level 1 (N=390)
Modela
1 2 3 4 5 6 7
Variable
MCAT biology scoreb P<.001 NS NS NS
Undergraduate science unitsc P<.001 NS NS NS
NBME-CBSE scored P<.001 P<.001 P<.001 P<.001
Preclinical GPAd P<.001 P<.001 P<.001 P<.001
Model Statistic
P valueb <.001 <.001 <.001 <.001 <.001 <.001 <.001
Adjusted r2c 0.0733 0.6639 0.5915 0.7420 0.7418 0.6635 0.5785
Root MSEd 80.571 48.519 53.497 42.51 42.528 48.551 54.337

a Models: 1, best preadmission model; 2, preadmission with NBME-CBSE (National Board of Medical Examiners Comprehensive Basic Science Examination); 3, preadmission with preclinical GPA (grade point average); 4, all variables; 5, best preclinical medical school model; 6, NBME-CBSE alone; 7, preclinical GPA alone.

b Statistical significance of the model measured as probability that independent variable coefficients equal 0.

c Fraction of variance explained by model.

d Smaller error indicates better fit to data.

Abbreviations: MSE, mean squared error; NS, not significant.

×
The optimal preclinical medical school model of COMLEX-USA Level 1 scores (Table 1, model 5) consisted of preclinical GPA and NBME-CBSE scores. A model with preclinical basic science GPA was also statistically significant but was eliminated because of collinearity with preclinical GPA, which could inflate variance and distort results. Preclinical medical school variables eliminated during optimization were first- and second-year total and basic science GPAs and preclinical basic science GPA. 
Model 1 (P<.001), the optimal preadmission model, was a statistically significant predictor of COMLEX-USA Level 1 scores and consisted of only MCAT biology score and undergraduate science units. The model explained just 7% of the variance in COMLEX-USA Level 1 scores (adjusted r2=0.0733). Error in the model (root MSE=80.571) was approximately 17% of the range (469 [304-773]) of sampled COMLEX-USA Level 1 scores. 
Model 2 (P<.001) showed that adding NBME-CBSE scores (P<.001) to optimal preadmission variables increased the explained variance to 66% (adjusted r2=0.6639, more than 9 times that of model 1) and dropped the error to 10% of the COMLEX-USA Level 1 score range (root MSE=48.519). Preadmission variables were not statistically significant in the presence of NBME-CBSE scores. 
Model 3 (P<.001) showed that substituting preclinical GPA (P<.001) for NBME-CBSE score with optimal preadmission variables dropped explained variance to 59% (adjusted r2=0.5915, less than model 2 but 8 times more than model 1), and increased the error to 11% of the COMLEX-USA Level 1 score range (root MSE=53.497). Preadmission variables were not statistically significant in the presence of preclinical GPA. 
Model 4 (P<.001), which contained all variables, showed that including both NBME-CBSE scores (P<.001) and preclinical GPA (P<.001) with optimal preadmission variables increased the explained variance to 74% (adjusted r2=0.7420) and dropped the error to 9% of the COMLEX-USA Level 1 score range (root MSE=42.51, approximately half of model 1 and less than models 2 and 3). Preadmission variables were not statistically significant. 
Model 5 (P<.001), the optimal preclinical medical school model, showed that NBME-CBSE scores (P<.001) and preclinical GPA (P<.001) both remained statistically significant in the presence of one another. Compared with model 4, explained variance (adjusted r2=0.7418) and error (root MSE=42.52) remained constant. 
Model 6 (P<.001) showed that NBME-CBSE scores (P<.001) alone were statistically significant. The explained variance of 66% (adjusted r2=0.6635) and error at 10% of the COMLEX-USA Level 1 score range (root MSE=48.551) were nearly identical to model 2 (adjusted r2=0.6639; root MSE=48.519), suggesting that the effectiveness of model 2 derived mostly from NBME-CBSE scores. 
Model 7 (P<.001) showed that preclinical GPA (P<.001) alone was statistically significant. The explained variance of 58% (adjusted r2=0.5785) and error at 11% of the COMLEX-USA Level 1 score range (root MSE=54.337) were similar to model 3 (adjusted r2=0.5915; root MSE=53.497), suggesting that the effectiveness of model 3 derived mostly from preclinical GPA. Compared with the NBME-CBSE score in model 6 (adjusted r2=0.6635; root MSE=48.551), preclinical GPA alone (adjusted r2=0.5785; root MSE=54.337) explained slightly less variance with slightly more error. 
Overall, Table 1shows that NBME-CBSE score is a statistically significant predictor of COMLEX-USA Level 1 scores. This finding holds that whether analyzed alone or in the presence of other statistically significant variables, such as MCAT biology score, undergraduate science units, and preclinical GPA. 
Modeling COMLEX-USA Level 1 Pass/Fail Outcome
Logistic regression uses real-number independent variables (eg, GPA) to model a categorical dependent variable like COMLEX-USA Level 1 pass/fail outcome.27(pp1057,1078) We used multiple logistic regression in a stepwise process to determine whether variables predicting COMLEX-USA Level 1 scores also predict the pass/fail outcome. 
Model 1 shows that MCAT biology score and undergraduate science units are not statistically significant predictors of COMLEX-USA Level 1 pass/fail outcome. This finding validates the investigation of pass/fail outcomes in addition to scores because these variables were statistically significant predictors of COMLEX-USA Level 1 scores. Score on the NBME-CBSE (P<.001) was the only significant variable in model 2 (P<.001). Preclinical GPA (P<.001) was the only significant variable when replacing NBME-CBSE scores in model 3 (P<.001). Model 4 (P<.001) showed that NBME-CBSE score (P<.001) and preclinical GPA (P<.001) remained significant in the presence of one another. Model 5 (P<.001) showed that without preadmission variables, NBME-CBSE scores (P<.001) and preclinical GPA (P<.001) comprised a statistically significant model, and both variables remained statistically significant in the presence of one another. Model 6 (P<.001) showed that NBME-CBSE score alone (P<.001) is a statistically significant predictor of COMLEX-USA Level 1 pass/fail outcome. Model 7 (P<.001) showed that preclinical GPA (P<.001) alone is a significant predictor of COMLEX-USA Level 1 pass/fail outcome. 
Overall, the logistic regression results in Table 2 show that NBME-CBSE score is a statistically significant variable in predicting COMLEX-USA Level 1 pass/fail outcomes. This finding holds whether analyzed alone or in the presence of preclinical GPA, another statistically significant variable. 
Table 2.
Logistic Regression Results for Comprehensive Osteopathic Medical Licensing Examination USA Level 1 Pass/Fail Outcome
Modela
1 2 3 4 5 6 7
Variable
MCAT biology score NS NS NS NS
Undergraduate science units NS NS NS NS
NBME-CBSE score P<.001 P<.001 P<.001 P<.001
Preclinical GPA P<.001 P<.001 P<.001 P<.001
Model Statistic
P valueb NS <.001 <.001 <.001 <.001 <.001 <.001

a Models: 1, preadmission model; 2, preadmission with NBME-CBSE (National Board of Medical Examiners Comprehensive Basic Science Examination); 3, preadmission with preclinical GPA (grade point average); 4, all variables; 5, best preclinical medical school model; 6, NBME-CBSE alone; 7, preclinical GPA alone.

b Statistical significance of the model measured as probability that there is no effect of independent variables on the dependant variables.

Abbreviation: NS, not significant

Table 2.
Logistic Regression Results for Comprehensive Osteopathic Medical Licensing Examination USA Level 1 Pass/Fail Outcome
Modela
1 2 3 4 5 6 7
Variable
MCAT biology score NS NS NS NS
Undergraduate science units NS NS NS NS
NBME-CBSE score P<.001 P<.001 P<.001 P<.001
Preclinical GPA P<.001 P<.001 P<.001 P<.001
Model Statistic
P valueb NS <.001 <.001 <.001 <.001 <.001 <.001

a Models: 1, preadmission model; 2, preadmission with NBME-CBSE (National Board of Medical Examiners Comprehensive Basic Science Examination); 3, preadmission with preclinical GPA (grade point average); 4, all variables; 5, best preclinical medical school model; 6, NBME-CBSE alone; 7, preclinical GPA alone.

b Statistical significance of the model measured as probability that there is no effect of independent variables on the dependant variables.

Abbreviation: NS, not significant

×
Assessment of Collinearity
Using highly correlated explanatory variables (collinearity) in regression models can reduce the reliability of findings by inflating variance and distorting results. Inflated variance can prevent the detection of statistically significant relationships (type II error), inflate SE, or cause large coefficient changes when adding or removing variables.28 Because our study design called for comparing SEs and we observed some large coefficient changes as variables were introduced, we calculated VIFs to assess whether our results were distorted by collinearity. The distortion caused by correlated variables is indexed by VIF against the situation in which no correlation exists. Therefore, a VIF of 1 means that the distortion is the same as if there were no correlation (ie, no distortion), whereas higher values indicate greater distortion.28 Small amounts of distortion do not affect the validity of conclusions drawn from linear regression.25 Collinearity is often deemed problematic if a VIF for a single variable is greater than 10 (some suggest a more conservative 30) and the mean of all VIFs is considerably larger than 1.27 (p1892) None of the values, including the mean, was greater than 2, so we determined that our regression results were likely not distorted by collinearity. 
Visualizing and Quantifying Diagnostic Effectiveness for Passing COMLEX-USA Level 1
Previous studies show that MCAT biology score and preclinical GPA are successful predictors of COMLEX-USA Level 1 or USMLE Step 1 pass/fail outcomes.9-13 Score on the NBME-CBSE is an effective predictor of USMLE Step 1 scores and pass/fail outcomes.21 To visually and statistically compare the effectiveness of NBME-CBSE scores, MCAT biology scores, and preclinical GPA at predicting passing COMLEX-USA Level 1 in our sample, ROC curves with area under the curve (AUC) were constructed. 
For a binary outcome (passing COMLEX-USA Level 1), ROC curves are generated by plotting the true positive fraction (sensitivity, or the count of those who actually passed after being classified by the diagnostic variable as likely to pass, divided by all who passed) against the false-positive fraction (1−specificity, or the count of actual failures after being identified by the diagnostic variable as likely to pass, divided by the total that actually passed) produced by each observed score of a continuous predictor variable.26,29 Area under the curve serves as a gross indicator of the effectiveness of the variable across its range of values at correctly predicting the binary outcome.26,29 A perfectly useless variable would produce an even percentage of true- and false-positives at each score, as indicated by the Reference line in the Figure, yielding AUC of 0.50.29 Successful predictors produce high true- to false-positive ratios as variable values increase, causing the plot to approximate the left and top boundaries of the plot area and the AUC to approach 1.29 
Figure
Receiver operating characteristic curves and area under the curve (AUC) for 3 predictors of passing the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA). Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test; NBME-CBSE, National Board of Medical Examiners Comprehensive Basic Science Examination.
Figure
Receiver operating characteristic curves and area under the curve (AUC) for 3 predictors of passing the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA). Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test; NBME-CBSE, National Board of Medical Examiners Comprehensive Basic Science Examination.
Receiver operating characteristic curves and AUCs found 3 predictors of passing COMLEX-USA Level 1. Score on the NBME-CBSE (AUC=0.88) predicts COMLEX-USA Level 1 outcome with success similar to preclinical GPA (AUC=0.86), and both are more successful than MCAT biology score (AUC=0.58). Although not formally defined, an AUC greater than 0.75 is often considered the threshold of a moderately useful diagnostic test.29 By this criterion, both NBME-CBSE score (AUC=0.8817) and preclinical GPA (AUC=0.8643) are similarly effective and more than moderately useful for predicting COMLEX-USA Level 1 outcomes. 
Comparing DO and MD Student Achievement on the NBME-CBSE
The mean scores on the NBME-CBSE were compared between the TUCOM sample and the national MD student cohort using a t test at a 99% confidence level. 
The mean NBME-CBSE score for our sample was compared with that of the national cohort of MD students, who took the examination during the same period. The t test indicated that it was unlikely the TUCOM sample mean was significantly different from the national cohorts’ (mean [SD] score, 63.5 [9.76] vs 64 [11.0], respectively; t=−0.9915, P=.322). 
Discussion
Residency program directors of DO and MD programs must be confident that incoming residents have the necessary education and training in the basic sciences, particularly with the upcoming single accreditation system for graduate medical education. Residency program directors may be more comfortable interpreting medical school grades and board examination scores from their own tradition and may be less certain regarding the other. We believe that the findings of the current study imply that residency program directors should be confident in the basic science preparation of students from either tradition. 
It would be logical to directly compare COMLEX-USA Level 1 with USMLE Step 1 results as proxies for professional expectations. By heavily weighting each toward basic science, the National Board of Osteopathic Medical Examiners and the NBME have defined minimum expectations for the breadth, depth, and level of basic science knowledge in each pathway to licensure.2-4 However, the examinations are not the same. The COMLEX-USA Level 1 assesses unique philosophy, principles, and practice patterns of DOs, and the USMLE Step 1 does not include distinctively osteopathic components, such as osteopathic principles or osteopathic manipulative treatment. Neither test exclusively measures basic science knowledge, and whereas COMLEX-USA Level 1 reports basic science subject area scores, neither test reports an overall basic science achievement score. Therefore, directly comparing their underlying constructs of basic science would require controlling for all additional variance components. We believe that we overcame that challenge by comparing COMLEX-USA Level 1 results with NBME-CBSE scores rather than to USMLE Step 1. We studied the NBME-CBSE because it is an intermediate measure, available to both populations of students, validated relative to MD licensing examination outcomes, and precisely focused on basic science knowledge. We do not believe that it completely measures medical knowledge competence, nor are we suggesting that it be required by all osteopathic medical schools. 
To our knowledge, this is the first study to document that NBME-CBSE scores predict COMLEX-USA Level 1 scores and the likelihood of passing the test. Chick et al30 showed a statistically significant correlation between USMLE Step 1 and COMLEX-USA Level 1 by directly comparing scores. However, they did not study the basic science component in isolation. Moreover, they studied applicants to an internal medicine residency, a population that may not be similar to the population in the current study. Our findings that NBME-CBSE scores explained variance and successfully predicted pass/fail outcomes of COMLEX-USA Level 1 alone and in the presence of all other study variables extend the findings of Glew et al21 that NBME-CBSE score alone explains variance and predicts pass/fail outcomes of USMLE Step 1. Our finding that preclinical GPA explained variance and successfully predicted pass/fail outcomes of COMLEX-USA Level 1 alone and in the presence of all other variables confirms the findings of Baker et al,11 Dixon,12 and Vora et al.13 We also observed an inverse correlation between undergraduate science units and COMLEX-USA Level 1 scores, which merits further study. 
The study also provides evidence from a large sample (N=390) that TUCOM students achieve at the same level on the NBME-CBSE as MD students. Administering the NBME-CBSE immediately after year-2 courses suggests that medical school curriculum, not additional board preparation, explains the performance. It is possible that TUCOM prepares students differently from other osteopathic institutions in the area of basic science. Data reported to the American Association of Colleges of Osteopathic Medicine indicate that the number of basic science hours at TUCOM in anatomy, microbiology, pathology, pharmacology, and physiology are not very different from the requirements of other colleges of osteopathic medicine; in biochemistry and “other basic sciences,” they may be lower.31 
The study design included statistical methods to promote reliability in the data and rule out competing explanations. Although linear regression identified statistically significant relationships between independent variables and COMLEX-USA Level 1 scores, it did not reveal whether the distribution was skewed to 1 side of the pass line. Therefore, we investigated the pass/fail outcome as well as scores. Receiver operating characteristic curve analysis showed that NBME-CBSE score and preclinical GPA alone were similarly sensitive and specific and more than moderately effective at predicting COMLEX-USA Level 1 pass/fail outcomes. Using multiple linear and logistic regressions in stepwise processes was also important because it established that the predictive capacity of NBME-CBSE score for passing COMLEX-USA Level 1 was not influenced by other variables. Investigating VIF confirmed the absence of additional statistically significant correlations in the regression models by statistically ruling out the possibility that observed regression statistics were biased by inflated variance. Including preclinical GPA in our analyses allowed comparison of NBME-CBSE scores with a widely used statistic. More importantly, we discovered that overall preclinical GPA at TUCOM is highly correlated with preclinical basic science GPA. Therefore, the finding that NBME-CBSE score is at least as effective as preclinical GPA in predicting COMLEX-USA Level 1 scores and pass/fail outcomes constitutes evidence that the construct of the NBME-CBSE is consistent with the construct of preclinical basic science GPA. 
Our hypothesis that basic science knowledge is similar for DO and MD students implies a similarity in both the underlying construct and level of expected basic science knowledge. The NBME-CBSE comprises systems (25%-35% general principles, 65%-75% individual organ systems) and processes (normal 25%-45%; abnormal 30%-50%; principles of therapeutics 15%-25%; and psychosocial, cultural, occupational, and environmental considerations 5%-10%) and has previously been shown to predict scores and the likelihood of passing USMLE Step 1, implying similarity among their constructs of basic science.21,22 Our demonstration that the mean NBME-CBSE score of TUCOM students cannot be statistically distinguished from that of the national cohort of MD students, together with evidence that the TUCOM basic science curriculum may not differ from other DO schools, supports our opinion that the current study provides preliminary evidence that the same level of basic science knowledge is expected within both philosophical approaches. However the current study is inferential in nature, relying on indirect analysis of a component of the underlying construct of several measures. Although a large portion of the variance in COMLEX-USA Level 1 scores, approximately 74%, was explained by the linear relationship to NBME-CBSE score and preclinical GPA (Table 1, model 5), it is possible that even more of the variance would be explained by including additional variables. For example, future studies comparing Comprehensive Osteopathic Medical Self-Assessment Examination Phase 1 (a self-assessment to gauge readiness for COMLEX-USA Level 1) score to COMLEX-USA Level 1 score may contribute to the explained variance. Furthermore, the generalizability of our findings is certainly limited by our reliance on a sample from a single institution. Although further study using data from additional measures and multiple colleges of osteopathic medicine would provide more generalizable evidence, the current study provides preliminary evidence that expectations of DO and MD students’ basic science knowledge may be the same. 
Conclusion
Score on the NBME-CBSE alone and in the presence of other variables predicts COMLEX-USA Level 1 pass/fail outcome. The USMLE Step 1, the NBME-CBSE, and COMLEX-USA Level 1 are constructed according to expectations for basic science knowledge within the respective philosophical approaches. Furthermore, the mean score of TUCOM students on the NBME-CBSE was not different from that of the national cohort of MD students. These findings may constitute preliminary evidence that the construct and level of basic science knowledge of DO and MD students is the same. 
Acknowledgments
We acknowledge the contributions of Donna Fyfe, MLIS, administrative assistant, and Walter Hartwig, PhD, associate dean for Academic Affairs, at TUCOM. 
References
Osteopathic Core Competencies for Medical Students. Chevy Chase, MD: American Association of Colleges of Osteopathic Medicine; 2012.
Comprehensive Osteopathic Medical Licensing Examination: the COMLEX-USA examination blueprint. National Board of Osteopathic Medical Examiners website. http://www.nbome.org/comlex-cbt.asp?m=can#a1. Accessed December 14, 2016.
USMLE Step 1 content description. United States Medical Licensing Examination website. http://www.usmle.org/step-1/#content-description. Accessed October 22, 2015.
Examination content Step 1. United States Medical Licensing Examination website. http://www.usmle.org/bulletin/exam-content/. Accessed December 14, 2016.
American Osteopathic Association Commission on Osteopathic College Accreditation. Accreditation of Colleges of Osteopathic Medicine: COM Accreditation Standards and Procedures (Effective: August 29, 2016). http://www.osteopathic.org/inside-aoa/accreditation/COM-accreditation/Documents/com-accreditation-standards-8-29-2016.pdf. Accessed January 6, 2017.
Accreditation Council for Graduate Medical Education. ACGME Common Program Requirements. 2016. http://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRs_07012016.pdf. Accessed January 6, 2017.
Liaison Committee on Medical Education. Guide tothe Institutional Self-Study For Medical Education Programs with Full Accreditation Surveys in 2017-2018. http://lcme.org/wp-content/uploads/filebase/self-study_guide/2017-18_Self-Study-Guide_Full_2016-07-27.docx. Accessed January 6, 2017.
Core Entrustable Professional Activities for Entering Residency: Curriculum Developers’ Guide. Washington, DC: Association of American Medical Colleges; 2014. http://members.aamc.org/eweb/upload/Core%20EPA%20Curriculum%20Dev%20Guide.pdf. Accessed December 14, 2016.
Coumarbatch J, Robinson L, Thomas R, Bridge P. Strategies for identifying students at risk for USMLE Step 1 failure. Fam Med. 2010;42(2):105-110. [PubMed]
Paolo AM, Bonaminio GA, Durham D, Stites SW. Comparison and cross-validation of simple and multiple logistic regression models to predict USMLE Step 1 performance. Teach Learn Med. 2004;16(1):69-73. doi:10.1207/s15328015tlm1601_14 [CrossRef] [PubMed]
Baker HH, Foster RW, Bates BP, et al Relationship between academic achievement and COMLEX-USA Level 1 performance: a multisite study. J Am Osteopath Assoc. 2000;100(4):238-242. [PubMed]
Dixon D. Prediction of osteopathic medical school performance on the basis of MCAT score, GPA, sex, undergraduate major, and undergraduate institution. J Am Osteopath Assoc. 2012;112(4):177.
Vora A, Maltezos N, Alfonzo L, Hernandez N, Calix E, Fernandez MI. Predictors of scoring at least 600 on COMLEX-USA Level 1: successful preparation strategies. J Am Osteopath Assoc. 2013;113(2):164-173. [CrossRef] [PubMed]
Dixon D. Relation between variables of preadmission, medical school performance, and COMLEX-USA Levels 1 and 2 performance. J Am Osteopath Assoc. 2004;104(8):332-336. [PubMed]
Meoli F, Wallace W, Kaiser-Smith J, Shen L. Relationship of osteopathic medical licensure examinations with undergraduate admission measures and predictive value of identifying future performance in osteopathic principles and practice/osteopathic manipulative medicine courses and rotations. J Am Osteopath Assoc. 2002;102(11):615-620. [PubMed]
Wong SK, Ramirez JR, Helf SC. Student performance on Levels 1 and 2-CE of COMLEX-USA: do elective upper-level undergraduate science courses matter? J Am Osteopath Assoc. 2009;109(11):592-598. [PubMed]
Basco WT, Way DP, Gilbert GE, Hudson A. Undergraduate institutional MCAT scores as predictors of USMLE Step 1 performance. Acad Med. 2002;77(10 suppl):S13-S16. [CrossRef] [PubMed]
Donnon T, Paolucci EO, Violato C. The predictive validity of the MCAT for medical school performance and medical board licensing examinations: a meta-analysis of the published research. Acad Med. 2007;82(1):100-106. doi:10.1097/01.ACM.0000249878.25186.b7 [CrossRef] [PubMed]
Silver B, Hodgson CS. Evaluating GPAs and MCAT scores as predictors of NBME I and clerkship performances based on students’ data from one undergraduate institution. Acad Med. 1997;72(5):394-396. [CrossRef] [PubMed]
Veloski JJ, Callahan CA, Xu G, Hojat M, Nash DB. Prediction of students’ performances on licensing examinations using age, race, sex, undergraduate GPAs, and MCAT scores. Acad Med. 2000;75(10 suppl):S28-S30. [CrossRef] [PubMed]
Glew RH, Ripkey DR, Swanson DB. Relationship between students’ performances on the NBME Comprehensive Basic Science Examination and the USMLE Step 1: a longitudinal investigation at one school. Acad Med. 1997;72(12):1097-1102. [CrossRef] [PubMed]
Comprehensive Basic Science Examination. National Board of Medical Examiners website. http://www.nbme.org/Schools/Subject-Exams/Subjects/comp_basicsci.html. Accessed December 14, 2016.
National Board of Medical Examiners. Norms for examinee performance. Comprehensive Basic Science Examination Score Interpretation Guide. Score Report for TUCOM; April 2015. http:\\NBME-CBSE_2015_ScoreReport_Direct Source.pdf. Accessed January 6, 2017.
Sokal RR, Rohlf FJ. Biometry: The Principles and Practices of Statistics in Biological Research. 2nd ed. New York, NY: W.H. Freeman; 1981:661-671.
Marquardt DW. Generalized inverses, ridge regression, biased linear estimation, and nonlinear estimation. Technometrics. 1970;12(3):591-612. [CrossRef]
Metz CE. Receiver operating characteristic analysis: a tool for the quantitative evaluation of observer performance and imaging systems. J Am Coll Radiol. 2006;3(6):413-422. doi:10.1016/j.jacr.2006.02.021 [CrossRef] [PubMed]
Stata Corporation. Base Reference Manual, Release 13. College Station, Texas: Stata Press; 2013:1892.
O’Brien RM. A caution regarding rules of thumb for variance inflation factors. Qual Quant. 2007;41(5):673-690. [CrossRef]
Grzybowski M, Younger JG. Statistical methodology III: receiver operating characteristic (ROC) curves. Acad Emerg. Med. 1997;4(8):818-826. doi:10.1111/j.1553-2712.1997.tb03793.x [CrossRef] [PubMed]
Chick DA, Friedman HP, Young VB, Solomon D. Relationship Between COMLEX and USMLE Scores Among Osteopathic Medical Students who Take Both Examinations [published correction appears in Teach Learn Med. 2010;22(2):165]. Teach Learn Med. 2010;22(1):3-7. doi:10.1080/10401330903445422 [CrossRef] [PubMed]
2012-13 Contact hours of required basic science instruction in first and second year curricula, by osteopathic medical college. American Association of Colleges of Osteopathic Medicine website. http://www.aacom.org/docs/default-source/data-and-trends/2012-13-contacthrs-reqbasicsciinstr-byCOM.pdf?sfvrsn=8. Accessed December 14, 2016.
Figure
Receiver operating characteristic curves and area under the curve (AUC) for 3 predictors of passing the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA). Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test; NBME-CBSE, National Board of Medical Examiners Comprehensive Basic Science Examination.
Figure
Receiver operating characteristic curves and area under the curve (AUC) for 3 predictors of passing the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA). Abbreviations: GPA, grade point average; MCAT, Medical College Admission Test; NBME-CBSE, National Board of Medical Examiners Comprehensive Basic Science Examination.
Table 1.
Linear Regression Results for Scores on the Comprehensive Osteopathic Medical Licensing Examination-USA Level 1 (N=390)
Modela
1 2 3 4 5 6 7
Variable
MCAT biology scoreb P<.001 NS NS NS
Undergraduate science unitsc P<.001 NS NS NS
NBME-CBSE scored P<.001 P<.001 P<.001 P<.001
Preclinical GPAd P<.001 P<.001 P<.001 P<.001
Model Statistic
P valueb <.001 <.001 <.001 <.001 <.001 <.001 <.001
Adjusted r2c 0.0733 0.6639 0.5915 0.7420 0.7418 0.6635 0.5785
Root MSEd 80.571 48.519 53.497 42.51 42.528 48.551 54.337

a Models: 1, best preadmission model; 2, preadmission with NBME-CBSE (National Board of Medical Examiners Comprehensive Basic Science Examination); 3, preadmission with preclinical GPA (grade point average); 4, all variables; 5, best preclinical medical school model; 6, NBME-CBSE alone; 7, preclinical GPA alone.

b Statistical significance of the model measured as probability that independent variable coefficients equal 0.

c Fraction of variance explained by model.

d Smaller error indicates better fit to data.

Abbreviations: MSE, mean squared error; NS, not significant.

Table 1.
Linear Regression Results for Scores on the Comprehensive Osteopathic Medical Licensing Examination-USA Level 1 (N=390)
Modela
1 2 3 4 5 6 7
Variable
MCAT biology scoreb P<.001 NS NS NS
Undergraduate science unitsc P<.001 NS NS NS
NBME-CBSE scored P<.001 P<.001 P<.001 P<.001
Preclinical GPAd P<.001 P<.001 P<.001 P<.001
Model Statistic
P valueb <.001 <.001 <.001 <.001 <.001 <.001 <.001
Adjusted r2c 0.0733 0.6639 0.5915 0.7420 0.7418 0.6635 0.5785
Root MSEd 80.571 48.519 53.497 42.51 42.528 48.551 54.337

a Models: 1, best preadmission model; 2, preadmission with NBME-CBSE (National Board of Medical Examiners Comprehensive Basic Science Examination); 3, preadmission with preclinical GPA (grade point average); 4, all variables; 5, best preclinical medical school model; 6, NBME-CBSE alone; 7, preclinical GPA alone.

b Statistical significance of the model measured as probability that independent variable coefficients equal 0.

c Fraction of variance explained by model.

d Smaller error indicates better fit to data.

Abbreviations: MSE, mean squared error; NS, not significant.

×
Table 2.
Logistic Regression Results for Comprehensive Osteopathic Medical Licensing Examination USA Level 1 Pass/Fail Outcome
Modela
1 2 3 4 5 6 7
Variable
MCAT biology score NS NS NS NS
Undergraduate science units NS NS NS NS
NBME-CBSE score P<.001 P<.001 P<.001 P<.001
Preclinical GPA P<.001 P<.001 P<.001 P<.001
Model Statistic
P valueb NS <.001 <.001 <.001 <.001 <.001 <.001

a Models: 1, preadmission model; 2, preadmission with NBME-CBSE (National Board of Medical Examiners Comprehensive Basic Science Examination); 3, preadmission with preclinical GPA (grade point average); 4, all variables; 5, best preclinical medical school model; 6, NBME-CBSE alone; 7, preclinical GPA alone.

b Statistical significance of the model measured as probability that there is no effect of independent variables on the dependant variables.

Abbreviation: NS, not significant

Table 2.
Logistic Regression Results for Comprehensive Osteopathic Medical Licensing Examination USA Level 1 Pass/Fail Outcome
Modela
1 2 3 4 5 6 7
Variable
MCAT biology score NS NS NS NS
Undergraduate science units NS NS NS NS
NBME-CBSE score P<.001 P<.001 P<.001 P<.001
Preclinical GPA P<.001 P<.001 P<.001 P<.001
Model Statistic
P valueb NS <.001 <.001 <.001 <.001 <.001 <.001

a Models: 1, preadmission model; 2, preadmission with NBME-CBSE (National Board of Medical Examiners Comprehensive Basic Science Examination); 3, preadmission with preclinical GPA (grade point average); 4, all variables; 5, best preclinical medical school model; 6, NBME-CBSE alone; 7, preclinical GPA alone.

b Statistical significance of the model measured as probability that there is no effect of independent variables on the dependant variables.

Abbreviation: NS, not significant

×