Free
JAOA/AACOM Medical Education  |   April 2020
Success Predictors For Third-Year Osteopathic Medical Students on National Standardized Examinations: A Family Medicine Clerkship Course Study
Author Notes
  • From the Department of Clinical Science Education at the A.T. Still University School of Osteopathic Medicine in Mesa, Arizona (Drs Glaser and Sackett); and the Department of Research Support at A.T. Still University in Kirksville, Missouri (Ms Pazdernik). 
  • Financial Disclosures: None reported. 
  • Support: None reported. 
  •  *Address correspondence to Kelli Glaser, DO, Department of Clinical Science Education, A.T. Still University School of Osteopathic Medicine, 5850 E Still Circle, Mesa, AZ 85206-3618. Email: kglaser@atsu.edu
     
Article Information
Medical Education
JAOA/AACOM Medical Education   |   April 2020
Success Predictors For Third-Year Osteopathic Medical Students on National Standardized Examinations: A Family Medicine Clerkship Course Study
The Journal of the American Osteopathic Association, April 2020, Vol. 120, 253-262. doi:https://doi.org/10.7556/jaoa.2020.042
The Journal of the American Osteopathic Association, April 2020, Vol. 120, 253-262. doi:https://doi.org/10.7556/jaoa.2020.042
Abstract

Context: Medical education institutions often use community-based sites and preceptors during students' third and fourth years for clinical training. However, differences in the sites, preceptors, assessment methods, and students may result in variations in clinical training, potentially affecting educational outcomes. During clerkships at A.T. Still University School of Osteopathic Medicine in Arizona, all students are evaluated by several methods for each required clerkship course. Required assessments include the clinical preceptor's evaluation, online coursework specific to each clerkship, patient log documentation, and the Comprehensive Osteopathic Medical Achievement Test (COMAT) relevant to that clerkship.

Objective: To evaluate which methods of student assessment in a family medicine clerkship course were most predictive of the future success of students on national standardized examinations.

Methods: Third-year osteopathic medical students from a single class who had completed the Comprehensive Osteopathic Medical Licensing Examination (COMLEX)-USA Level 2-Cognitive Evaluation (CE) and 2-Performance Evaluation (PE) and the COMAT were included in the study. Scores on the examinations were used as success benchmarks. Analysis of 4 categories of predictor variables—clerkship site, previous student performance, preceptor evaluation, and clerkship coursework assessment (ie, assignment scores and log numbers)—was used to predict success on the national standardized examinations.

Results: Ninety-nine of 105 students were eligible for inclusion. No associations were found between examination scores and clerkship site or log numbers (all P≥.10). Correlations were found for previous student performance (ie, grade point average for first-year and second-year coursework) and all examinations except COMLEX-USA Level 2-PE (r=0.56-0.74, all P<.001), and between total score in family medicine clerkship coursework and COMLEX-USA Level 2-CE and COMAT scores (r=0.28-0.39, all P≤.006). Correlations were also found between preceptor evaluation (total score and subscore on medical knowledge) and all assessed national standardized examinations (r=0.20-0.34, all P<.049).

Conclusion: Our results suggest that analysis of predictor variables in clerkship courses can reasonably predict success on national standardized examinations and may be useful for early identification of struggling students who may need additional support to perform well on the examinations.

Medical education institutions often use community-based sites and preceptors for clinical training during students' third and fourth years. However, differences in the sites, preceptors, assessment methods, and students may result in variations in clinical training within an institution, potentially affecting educational outcomes.1-3 In 2014, A.T. Still University School of Osteopathic Medicine in Arizona (ATSU-SOMA) addressed this variability by creating mandatory coursework for all required clerkship courses. This coursework is concurrent with students’ clerkship rotation with an individual preceptor and is assessed by faculty on the ATSU-SOMA main campus. Students are evaluated by several methods, including the clinical preceptor's evaluation, online coursework specific to each clerkship and graded by the clerkship course director, patient log documentation, and the Comprehensive Osteopathic Medical Achievement Test (COMAT), a nationally standardized subject examination. 
Although evidence suggests that discipline-specific COMAT and National Board of Medical Examiners (NBME) shelf examinations and preceptor evaluations are predictive of student performance on the United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge and Comprehensive Osteopathic Medical Licensing Examination (COMLEX)-USA Level 2-Cognitive Evaluation (CE),4-7 it is unclear whether other methods of evaluation during third-year clerkships are predictive of a student's success on the COMLEX-USA Level 2-CE. Because the success of medical students is difficult to define, national standardized examinations are commonly used to predict future success (eg, COMLEX-USA Level 2-CE, COMLEX-USA Level 2-Performance Evaluation [PE], and discipline-specific COMAT or NBME shelf examinations).6,8-10 Furthermore, COMLEX-USA Levels 1 and 2 (CE and PE) are required for graduation. Studies investigating student learning and success have used scores from the COMLEX-USA examinations,10-14 but use of COMAT examination scores to assess student success has been less frequently reported. Li et al15 showed that COMAT examination scores correlated well with COMLEX-USA Level 2-CE scores. 
The purpose of the current study was to evaluate which methods of student assessment in a family medicine (FM) clerkship course were most predictive of future success of students on national standardized examinations. Specifically, we assessed how clerkship site, previous student performance, FM clerkship coursework assessment, and preceptor evaluation were related to results on the COMLEX-USA Level 2-CE, COMLEX-USA Level 2-PE, and COMAT examinations. We believe that analysis of these data will allow early identification of students who may be struggling so we can help them improve performance before they take the national examinations. 
Methods
The current quantitative study used a nonexperimental design with a convenience sample of 105 third-year osteopathic medical students from a single class at ATSU-SOMA. Inclusion criteria required students to have completed all COMLEX-USA Level 2-CE and Level 2-PE and COMAT examinations. De-identified data were collected from existing databases, namely Blackboard, E*value, and Progress IQ systems. The A.T. Still University-Arizona institutional review board considered the study exempt. 
The FM clerkship at ATSU-SOMA is 8 weeks and includes online coursework consisting of 7 graded assignments, 1 each week for the first 7 weeks. Types of graded assignments included subjective, objective, assessment, and plan (SOAP) notes; multiple-choice quizzes; reflective essays; and patient case analyses. Students are provided a list of reading assignments and learning objectives that correlate with each of the graded assignments. 
The COMLEX-USA Level 2-CE, COMLEX-USA Level 2-CE FM subscore, COMLEX-USA Level 2- PE, COMAT examination for family medicine (COMAT FM), and mean COMAT examination scores from the FM, internal medicine, obstetrics and gynecology, pediatrics, psychiatry, and general surgery clerkships were used as success benchmarks. The current study focused specifically on the FM clerkship, but we included data from all required clerkships to calculate a mean across courses for comparisons of some measures. We included COMAT scores because they are an important and substantial component of the grade for required clerkships at our institution. Clerkship site, previous student performance, FM clerkship coursework assessment, and preceptor evaluation were considered in relation to results on national standardized examinations in the clerkship years. 
At ATSU-SOMA, students are assigned to 1 of 12 community health centers (CHCs) to complete third- and fourth-year clerkships. Although our previous internal evaluations of student performance at our CHC clerkship sites showed no differences between sites in performance on national examinations, we continue to evaluate this outcome to ensure equivalency across sites. Therefore, this predictor variable was included in the current study. Because students with a higher grade point average (GPA) during the first and second year tend to perform better on national standardized tests,4,6,12,13 we used GPA as a predictor variable of student performance (Figure 1). 
Figure 1.
Predictor and outcome variables used in the current study to determine the methods of student assessment most predictive of student success on national standardized examinations. Abbreviations: CE, cognitive evaluation; CHC, community health center; COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination; CRE, clinical rotation evaluation; FM, family medicine; GPA, grade point average; IM, internal medicine; OBGYN, obstetrics and gynecology; PE, performance evaluation; PEDS, pediatrics; PSYCH, psychiatry; SOAP, subjective, objective, assessment, and plan; SURGERY, general surgery.
Figure 1.
Predictor and outcome variables used in the current study to determine the methods of student assessment most predictive of student success on national standardized examinations. Abbreviations: CE, cognitive evaluation; CHC, community health center; COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination; CRE, clinical rotation evaluation; FM, family medicine; GPA, grade point average; IM, internal medicine; OBGYN, obstetrics and gynecology; PE, performance evaluation; PEDS, pediatrics; PSYCH, psychiatry; SOAP, subjective, objective, assessment, and plan; SURGERY, general surgery.
For the FM clerkship coursework assessment predictor variables, we evaluated the total score for graded clerkship assignments and a few individual assignments, such as quiz scores and SOAP notes. We also evaluated whether students accessed learning objectives, the time spent on assignments and reading, and the number of clinical activity logs. We suspected that better scores on course content and student completion of learning objectives, assigned reading, and clinical activity logs would correlate with better examination outcomes. 
Preceptor evaluation was included as a predictor variable and was assessed with a clinical rotation evaluation (CRE) score. We evaluated 3 CRE subscores that assessed student performance in addition to the total CRE score. Subscores determined whether the student (1) demonstrated knowledge of common acute and chronic conditions, such as their main signs and symptoms, diagnostic tests, and treatment, including medication appropriate to the level of training; (2) demonstrated accurate interviewing skills, such as taking a history pertinent to the clinical condition; and (3) accurately performed systematic physical examinations pertinent to the clinical presentation. Each item was scored by the preceptor using a Likert-style scale from 1 to 5 points (1, consistently failed to meet expectations and 5, consistently exceeded expectations). Because of the subjective nature of preceptor evaluations, we were uncertain whether they would correlate with national examination outcomes and whether these items could individually serve as an early warning sign of poor student performance. 
Analysis of variance was used to assess differences between national standardized examination scores (COMLEX-USA Level 2-CE score, COMLEX-USA Level 2-CE FM subscore, COMAT FM score, and mean COMAT score across all required clerkships) and CHC site predictor variable. Linear regression analyses were used to assess the correlation between national performance examination outcomes and remaining predictor variables (previous student performance, FM clerkship coursework assessment, and preceptor evaluation). The Fisher exact test was used to assess the association between COMLEX-USA Level 2-PE score and CHC site predictor variable. Logistic regression analyses were used to assess the relationship between passing COMLEX-USA Level 2-PE and remaining predictor variables. The proportion of variance in student performances on a national examination explained by the predictor variable are reported as R2 from analysis of variance and linear regressions, McFadden R2 from logistic regressions, and the uncertainty coefficient from the Fisher exact test. For linear regressions, the unsigned correlation, r, was determined by R2. We used SAS 9.4 (SAS Institute, Inc) and R version 3.5.2 (Revolutions Analytics) to conduct the analysis. P<.05 was considered statistically significant. 
Results
Of the 105 osteopathic medical students in the class cohort, performance data from 99 (94%) students were eligible for inclusion and analysis. Six students did not meet inclusion criteria at the time of analysis. Comparisons of national standardized examination scores and predictor variables are reported in the Table. 
Table.
Proportion of Variance in Student Performances on National Examination Explained by Predictor Variablea
COMLEX-USA Level 2 COMAT
Predictor Variable CE P Value CE FM P Value PE P Value FM P Value Meanb P Value
CHC site 0.11 .51 0.116 .42 0.280 .50 0.062 .88 0.065 .87
GPA years 1-2 0.50 <.001 0.318 <.001 0.006 .62 0.372 <.001 0.553 <.001
FM clerkship coursework 0.09 .003 0.029 .09 0.004 .68 0.076 .006 0.151 <.001
FM quiz 1 0.04 .06 0.003 .58 NA NA 0.021 .16 NA NA
FM quiz 2c 0.09 .002 0.019 .17 NA NA 0.048 .03 NA NA
FM quiz 2d 0.05 .049 0.010 .33 NA NA 0.013 .27 NA NA
FM SOAP note NA NA NA NA −0.005 .67 NA NA NA NA
FM assignment time −0.05 .03 −0.026 .11 −0.001 .86 −0.011 .30 NA NA
Meanb assignment time 0.04 .06 NA NA NA NA NA NA 0.013 .27
FM reading time −0.01 .36 −0.016 .21 −0.001 .81 −0.011 .29 NA NA
Meanb reading time −0.01 .32 NA NA NA NA NA NA −0.002 .67
Access LO week 1 FM 0.01 .41 0.006 .44 NA NA <0.001 .94 0.007 .42
Access LO week 8 FM 0.01 .49 0.004 .53 NA NA <0.001 .98 <0.001 .80
FM logs −0.02 .13 −0.026 .11 −0.041 .14 −0.027 .10 NA NA
Meanb logs 0.01 .27 NA NA <0.001 .97 NA NA −0.015 .23
FM CRE total 0.12 <.001 0.044 .04 NA NA 0.040 .049 0.067 .01
FM CRE knowledge 0.12 <.001 0.072 .007 NA NA 0.079 .005 0.074 .007
FM CRE history-taking NA NA NA NA 0.026 .26 NA NA NA NA
FM CRE physical examination NA NA NA NA 0.055 .10 NA NA NA NA

a A negative proportion indicates a negative relationship. Proportion of variance explained was reported as R2 from analysis of variance and linear regressions, McFadden R2 from logistic regressions, and uncertainty coefficient from Fisher exact test.

b Mean across all required clerkships (family medicine, internal medicine, pediatrics, obstetrics and gynecology, psychiatry, general surgery).

c Two incomplete quizzes (score of 0) were included in the analysis.

d Two incomplete quizzes were excluded from the analysis.

Abbreviations: CE, cognitive evaluation; CHC, community health center; COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination; CRE, clinical rotation evaluation; FM, family medicine; GPA, grade point average; LO, learning objective; PE, performance evaluation; NA, not applicable; SOAP, subjective, objective, assessment, and plan.

Table.
Proportion of Variance in Student Performances on National Examination Explained by Predictor Variablea
COMLEX-USA Level 2 COMAT
Predictor Variable CE P Value CE FM P Value PE P Value FM P Value Meanb P Value
CHC site 0.11 .51 0.116 .42 0.280 .50 0.062 .88 0.065 .87
GPA years 1-2 0.50 <.001 0.318 <.001 0.006 .62 0.372 <.001 0.553 <.001
FM clerkship coursework 0.09 .003 0.029 .09 0.004 .68 0.076 .006 0.151 <.001
FM quiz 1 0.04 .06 0.003 .58 NA NA 0.021 .16 NA NA
FM quiz 2c 0.09 .002 0.019 .17 NA NA 0.048 .03 NA NA
FM quiz 2d 0.05 .049 0.010 .33 NA NA 0.013 .27 NA NA
FM SOAP note NA NA NA NA −0.005 .67 NA NA NA NA
FM assignment time −0.05 .03 −0.026 .11 −0.001 .86 −0.011 .30 NA NA
Meanb assignment time 0.04 .06 NA NA NA NA NA NA 0.013 .27
FM reading time −0.01 .36 −0.016 .21 −0.001 .81 −0.011 .29 NA NA
Meanb reading time −0.01 .32 NA NA NA NA NA NA −0.002 .67
Access LO week 1 FM 0.01 .41 0.006 .44 NA NA <0.001 .94 0.007 .42
Access LO week 8 FM 0.01 .49 0.004 .53 NA NA <0.001 .98 <0.001 .80
FM logs −0.02 .13 −0.026 .11 −0.041 .14 −0.027 .10 NA NA
Meanb logs 0.01 .27 NA NA <0.001 .97 NA NA −0.015 .23
FM CRE total 0.12 <.001 0.044 .04 NA NA 0.040 .049 0.067 .01
FM CRE knowledge 0.12 <.001 0.072 .007 NA NA 0.079 .005 0.074 .007
FM CRE history-taking NA NA NA NA 0.026 .26 NA NA NA NA
FM CRE physical examination NA NA NA NA 0.055 .10 NA NA NA NA

a A negative proportion indicates a negative relationship. Proportion of variance explained was reported as R2 from analysis of variance and linear regressions, McFadden R2 from logistic regressions, and uncertainty coefficient from Fisher exact test.

b Mean across all required clerkships (family medicine, internal medicine, pediatrics, obstetrics and gynecology, psychiatry, general surgery).

c Two incomplete quizzes (score of 0) were included in the analysis.

d Two incomplete quizzes were excluded from the analysis.

Abbreviations: CE, cognitive evaluation; CHC, community health center; COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination; CRE, clinical rotation evaluation; FM, family medicine; GPA, grade point average; LO, learning objective; PE, performance evaluation; NA, not applicable; SOAP, subjective, objective, assessment, and plan.

×
For clerkship site, the number of students per CHC ranged from 7 to 12 students, except for Waianae, Hawaii, which had 3 students. No associations were found between national examination outcomes and site (all P≥.42) (Table). 
For previous student performance, students with higher GPAs at the end of the second year performed better on COMLEX-USA Level 2-CE, COMLEX-USA Level 2-CE FM subscore, COMAT FM, and mean COMAT (all P<.001) (Table). Depending on the examination, GPA accounted for 32% (r=0.56) to 55% (r=0.74) of variation in student performance. 
For the FM clerkship coursework assessment, there was a positive correlation between total coursework score and COMLEX-USA Level 2-CE (r=0.30, P=.003), COMAT FM (r=0.28, P=.006), and mean COMAT scores for all required clerkship courses (r=0.39, P<.001) (Table). Regarding individual assignments, comparison of the first quiz and COMLEX-USA Level 2-CE score was not statistically significant but suggested a positive correlation (r=0.19, P=.06). Significant correlation was found between the second quiz and COMLEX-USA Level 2-CE score (r=0.22, P=.049). Two students did not complete the second quiz, were given scores of 0, and were initially excluded from analysis. Positive correlations were found for COMLEX-USA 2-CE (r=0.30, P=.002) and COMAT FM (r=0.22, P=.03) scores when the 0 scores were included. Time spent on FM clerkship coursework assignments was negatively correlated with performance on COMLEX-USA Level 2-CE (r=–0.21, P=.03). No correlation was found between performance on examinations and mean number of clinical activity logs across all required clerkships (all P≥.23) (Figure 2). Although students accessed learning objectives more frequently in week 1 of the clerkship than in week 8 (78 [79%] students compared with 7 students [17%], respectively), no correlation with performance on examinations was found (all P≥.41). 
Figure 2.
Clinical activity log numbers and student performance on the Comprehensive Osteopathic Medical Licensing Examination (COMLEX)-USA Level 2-Cognitive Evaluation (CE). Mean across all required clerkships. Abbreviations: FM, family medicine; IM, internal medicine; OBGYN, obstetrics and gynecology; PEDS, pediatrics; PSYCH, psychiatry; SURGERY, general surgery.
Figure 2.
Clinical activity log numbers and student performance on the Comprehensive Osteopathic Medical Licensing Examination (COMLEX)-USA Level 2-Cognitive Evaluation (CE). Mean across all required clerkships. Abbreviations: FM, family medicine; IM, internal medicine; OBGYN, obstetrics and gynecology; PEDS, pediatrics; PSYCH, psychiatry; SURGERY, general surgery.
For preceptor evaluation, a significant correlation was found between FM CRE total score and COMLEX-USA Level 2-CE (r=0.34, P<.001), COMLEX-USA Level 2-CE FM subscore (r=0.21, P=.04), COMAT FM (r=0.20, P=.049), and mean COMAT score for all required clerkship courses (r=0.26, P=.01) (Table). A significant correlation was also found between CRE subscore on medical knowledge and COMLEX-USA Level 2-CE (r=0.34, P<.001), COMLEX-USA Level 2-CE FM subscore (r=0.27, P=.007), COMAT FM score (r=0.28, P=.005), and the mean COMAT score for all required clerkship courses (r=0.27, P=.007). There was no clear relationship between CRE subscores for history-taking or physical examination skills and passing COMLEX-USA Level 2-PE (both P≥.10). 
Discussion
The current study evaluated which methods of student assessment in an FM clerkship course were most predictive of future success of students on national standardized COMLEX-USA Level 2-CE, COMLEX-USA Level 2-PE, and COMAT examinations. We used 4 categories of predictor variables—clerkship site, previous student performance, clerkship coursework assessment, and preceptor evaluation—to determine success on the examinations. No differences were found between clerkship site and examination scores. The most significant correlations were found for previous student performance (GPA for first-and second-year coursework) and all examination results, except for COMLEX-USA Level 2-PE. Significant correlations were found between total score in FM clerkship coursework and COMLEX-USA Level 2-CE, COMAT FM, and mean COMAT scores, indicating that student performance on the FM clerkship coursework could be used as a predictor of their future success on the COMLEX Level 2-CE and COMAT examinations. Significant correlations were also found between preceptor evaluation (CRE total score and subscore on medical knowledge) and all examinations, which supports the value of preceptor ratings of student performance. 
The current study included site as a predictor variable because a student's placement at a clerkship site may affect success on examinations. This variable is particularly relevant for ATSU-SOMA because of our distributed model of education. However, student performance on national standardized examinations was not associated with CHC site assignment. 
Similar to previous studies,4,12,16 a higher GPA from the first and second years of medical education correlated with stronger performance on the COMLEX-USA Level 2-CE examination, which suggests that it is still the best predictor of student performance on this examination. Scores on the COMAT FM and mean COMAT were also correlated with GPA. To our knowledge, no previous studies have investigated this correlation. However, similar findings have been reported for NBME subject examinations, where higher performers had higher GPAs and performed better on USMLE Step 1 and Step 2 Clinical Knowledge examinations.17,18 
As expected, students in the current study who performed well overall on clerkship coursework performed well on the COMLEX-USA Level 2-CE, COMAT FM, and mean COMAT. Similarly, Myles and Galvez-Myles19 found that course final examination scores correlated with USMLE Step 2 scores. Our finding suggests that poor performance on total coursework score and, specifically, the second FM multiple-choice quiz can be used to identify students at risk for poor performance on national standardized examinations. 
An unexpected result was the negative correlation between time students reported they spent on coursework and performance on COMLEX-USA Level 2-CE. Perhaps students with stronger skills spent less time on graded assignments because their skills were already at a higher level. Therefore, they could complete assignments in less time. Another explanation is that some assignments may have taken time away from studying other content that is necessary to do well on national examinations. For example, some graded assignments have learning goals that focus on the development of other essential skills for physicians, such as interprofessional teamwork. Therefore, those assignments would be less likely to positively affect COMLEX-USA Level 2-CE scores because content is more heavily based on scientific and diagnostic information. 
We expected that students who spent more time reading during the clerkship would perform better on national standardized examinations. However, we found no correlation between student-reported reading time and performance on examinations. However, students may have misestimated the time they spent reading. We also suspect that students may have overreported reading time to create a positive perception of their effort. 
Although we expected that students who consistently accessed course learning objectives would perform better than those who did not, no correlations were found with national standardized examinations. One explanation for this result is that accessing learning objectives does not equate to using them or mastering their content. We measured accessing learning objectives in the first and last week of the FM clerkship course, and the number of students who accessed them in the final week was noticeably lower. Thus, it seems that most students did not consistently use them to guide learning. The academic breadth of FM is difficult to cover in an 8-week course, and the large volume of learning objectives and reading assignments provided to the students may have resulted in them being underused. Another explanation could be that each quiz from assigned readings was worth only 5% of the final grade, so students preferred to focus on other requirements that contributed more to the final grade. However, this approach would be shortsighted given the correlation between the FM clerkship quizzes and the national examination scores. 
We expected that students would perform better on the national examinations if they had more clinical activity logs because of exposure to more patient cases or more diligence in logging cases, but this finding was not supported by our data. The use of logs to assess rotations may have been flawed if students were not accurately reporting what they saw. Technical problems with data entry or other unreported issues may also contribute to inaccurate reporting. Another possible explanation could be that the experience of observing and working with a larger volume of patients does not translate to deeper understanding and increased knowledge. As such, students who spend more time with fewer patients may develop better understanding because they have increased cognitive processing time. 
Regarding preceptor evaluation, we expected that CRE scores would not correlate with national standardized examination outcomes because, as clerkship course directors, we have anecdotally noticed that some preceptor CREs of students in the FM clerkship lacked discrimination of learner abilities, similar to another study.20 Although a Likert-style scale was used for the CRE, it is still prone to subjective evaluation, and preceptors may give students the benefit of the doubt. Furthermore, some preceptors may not understand how to assess student abilities. However, preceptor evaluations, specifically total CRE and CRE subscore on medical knowledge, were strongly positively correlated with all examination results. Other studies17-19 investigating allopathic programs found similar correlations between faculty preceptor evaluations and NBME subject examination scores or USMLE Step 2 Clinical Knowledge scores. Taken together, these findings suggest that identifying students with poor preceptor evaluations for early intervention could affect examination outcomes. For example, early intervention can be used to help those students address areas of deficiency before taking the COMLEX-USA Level 2-CE. 
We expected other items of the CRE, including history-taking and physical examination skills, to be related to the COMLEX-USA Level 2-PE examination, but we found no relationship. One reason for this outcome may be that preceptor teaching and assessment of students is rarely standardized and may be performed differently from the COMLEX-USA Level 2-PE examination. We had only a handful of failing students in our study, so the power to detect a true positive relationship for this outcome was low. 
When considering COMLEX-USA Level 2-PE performance and all other predictor variables, such as GPA, FM clerkship coursework total score, and SOAP note assignment, no correlations were found. Future research should focus on factors known to correlate with COMLEX-USA Level 2-PE. For example, objective structured clinical examination scores have been correlated with student performance on the USMLE Step 2 Clinical Knowledge.21,22 
The current study had several limitations. Although our results showed correlations between national standardized examination scores and some of our course assessment methods, it is unknown whether these results can be generalized to different years or classes of students since only a single class at a single institution was assessed, and other institutions would likely have different methods of assessment for their students. Another limitation was the self-reported nature of the data for reading time, time spent on graded assignments, and volume of diagnoses reported in clinical activity logs. Such data may be prone to error, misunderstanding, and bias. 
Future studies should investigate our predictor variables for determining success in other courses and at other institutions to evaluate whether results are similar. Studies could also assess consistency of results for different student cohorts or identify additional factors that predict student performance on the COMLEX-USA Level 2-PE examination. Given the osteopathic manipulative treatment component of the COMLEX-USA Level 2-PE, future studies could determine whether preceptor evaluation scores of students from osteopathic or allopathic programs are better able to predict success on the examination. Ultimately, as medical educators, it is important that we determine the factors that predict success in residency and future practice to ensure that patients receive the best care. 
Conclusion
We determined that several variables of student performance in an FM clerkship course were predictors of later performance on the COMLEX-USA Level 2-CE and COMAT examinations. We also found that preceptor evaluation of student performance was a reasonable predictor of success on those examinations. Therefore, these predictor variables may be a useful measure to identify struggling students for early intervention. Our findings also suggested that clinical activity log volume has limited value as a measure of student learning. Other institutions should perform a similar analysis of evaluation methods to identify specific items that predict a student's future performance on national standardized examinations to improve students’ likelihood of succeeding. 
Acknowledgements
We thank Nichole Goldman, instructional coordinator at A.T. Still University's School of Osteopathic Medicine in Arizona, for compiling the data used in the current study; and Deborah Goggin, MA, ELS, scientific writer in the Department of Research Support at A.T. Still University, for her editorial assistance. 
References
Rassie K. The apprenticeship model of clinical medical education: time for structural change. N Z Med J. 2017;130(1461):66-72. [PubMed]
Stalmeijer RE, Dolmans DHJM, Wolfhagen IHAP, Scherpbier AJJA. Cognitive apprenticeship in clinical practice: can it stimulate learning in the opinion of students? Adv Health Sci Educ Theory Pract. 2009;14(4):535-546. doi: 10.1007/s10459-008-9136-0 [CrossRef] [PubMed]
Steven K, Wenger E, Boshuizen H, Scherpbier A, Dornan T. How clerkship students learn from real patients in practice settings. Acad Med. 2014;89(3):469-476. doi: 10.1097/ACM.0000000000000129 [CrossRef] [PubMed]
Dixon D. Prediction of osteopathic medical school performance on the basis of MCAT score, GPA, sex, undergraduate major, and undergraduate institution. J Am Osteopath Assoc. 2012;112(4):175-181. [PubMed]
Dong T, Swygert KA, Durning SJ, et al. Is poor performance on NBME clinical subject examinations associated with a failing score on the USMLE step 3 examination? Acad Med. 2014;89(5):762-766. doi: 10.1097/ACM.0000000000000222 [CrossRef] [PubMed]
Zahn CM, Saguil A, Jr Artino AR, et al. Correlation of National Board of Medical Examiners scores with United States Medical Licensing Examination Step 1 and Step 2 scores. Acad Med. 2012;87(10):1348-1354. doi: 10.1097/ACM.0b013e31826a13bd [CrossRef] [PubMed]
Hudson KM, Tsai TH, Finch C, Dickerman JL, Liu S, Shen L. A validity study of COMLEX-USA Level 2-CE and COMAT clinical subjects: concurrent and predictive evidence. J Grad Med Educ. 2019;11(5):521-526. doi: 10.4300/JGME-D-19-00157.1 [CrossRef] [PubMed]
American Association of Colleges of Osteopathic Medicine. Osteopathic Medical College Information Book: 2018-2019. American Association of Colleges of Osteopathic Medicine; 2018. https://www.aacom.org/docs/default-source/cib/aacom-cib-2019-all-web.pdf. Accessed June 10, 2019.
National Board of Medical Examiners. 2016 NBME Clinical Clerkship Subject Examination Survey: Summary of Results. National Board of Medical Examiners; 2016. https://www.nbme.org/PDF/SubjectExams/Clerkship_Survey_Summary.pdf. Accessed June 10, 2019.
O'Neill TR, Peabody MR, Song H. The predictive validity of the National Board of Osteopathic Medical Examiners' COMLEX-USA examinations with regard to outcomes on American Board of Family Medicine examinations. Acad Med. 2016;91(11):1568-1575. doi: 10.1097/ACM.0000000000001254 [CrossRef] [PubMed]
Agahi F, Speicher MR, Cisek G. Association between undergraduate performance predictors and academic and clinical performance of osteopathic medical students. J Am Osteopath Assoc. 2018;118(2):106-114. doi: 10.7556/jaoa.2018.025 [CrossRef] [PubMed]
Baker HH, Cope MK, Fisk R, Gorby JN, Foster RW. Relationship of preadmission variables and first- and second-year course performance to performance on the National Board of Osteopathic Medical Examiners' COMLEX-USA Level 1 examination. J Am Osteopath Assoc. 2000;100(3):153-161. [PubMed]
Cope MK, Baker HH, Fisk R, Gorby JN, Foster RW. Prediction of student performance on the Comprehensive Osteopathic Medical Licensing Examination Level I based on admission data and course performance. J Am Osteopath Assoc. 2001;101(2):84-85, 89-90. [PubMed]
Hudson KM, Feinberg G, Hempstead L, Zipp C, Gimpel JR, Wang Y. Association between performance on COMLEX-USA and the American College of Osteopathic Family Physicians in-service examination. J Grad Med Educ. 2018;10(5):543-547. doi: 10.4300/JGME-D-17-00997.1 [CrossRef] [PubMed]
Li F, Kalinowski KE, Song H, Bates BP. Relationships between the Comprehensive Osteopathic Medical Achievement Test (COMAT) subject examinations and the COMLEX-USA Level 2-Cognitive Evaluation. J Am Osteopath Assoc. 2014;114(9):714-721. doi: 10.7556/jaoa.2014.140 [CrossRef] [PubMed]
Cope MK, Baker HH, Foster RW, Boisvert CS. Relationships between clinical rotation subscores, COMLEX-USA examination results, and school-based performance measures. J Am Osteopath Assoc. 2007;107(11):502-510. [PubMed]
Casey PM, Palmer BA, Thompson GB, et al. Predictors of medical school clerkship performance: a multispecialty longitudinal analysis of standardized examination scores and clinical assessments. BMC Med Educ. 2016;16(128). doi: 10.1186/s12909-016-0652-y
Ogunyemi D, Taylor-Harris D. Factors that correlate with the U.S. Medical Licensure Examination Step-2 scores in a diverse medical student population. . J Natl Med Assoc. . 2005;97(9):1258-1262. [PubMed]
Myles T, Galvez-Myles R. USMLE Step 1 and 2 scores correlate with family medicine clinical and examination scores. Fam Med. 2003;35(7):510-513. [PubMed]
Pulito AR, Donnelly MB, Plymale M. Factors in faculty evaluation of medical students' performance. Med Educ. 2007;41(7):667-675. doi: 10.1111/j.1365-2923.2007.02787.x [CrossRef] [PubMed]
Lee M, Vermillion M. Comparative values of medical school assessments in the prediction of internship performance. Med Teach. 2018;40(12):1287-1292. doi: 10.1080/0142159X.2018.1430353 [CrossRef] [PubMed]
Simon SR, Bui A, Day S, Berti D, Volkan K. The relationship between second-year medical students' OSCE scores and USMLE Step 2 scores. J Eval Clin Pract. 2007;13(6):901-905. doi: 10.1111/j.1365-2753.2006.00768.x [PubMed]
Figure 1.
Predictor and outcome variables used in the current study to determine the methods of student assessment most predictive of student success on national standardized examinations. Abbreviations: CE, cognitive evaluation; CHC, community health center; COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination; CRE, clinical rotation evaluation; FM, family medicine; GPA, grade point average; IM, internal medicine; OBGYN, obstetrics and gynecology; PE, performance evaluation; PEDS, pediatrics; PSYCH, psychiatry; SOAP, subjective, objective, assessment, and plan; SURGERY, general surgery.
Figure 1.
Predictor and outcome variables used in the current study to determine the methods of student assessment most predictive of student success on national standardized examinations. Abbreviations: CE, cognitive evaluation; CHC, community health center; COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination; CRE, clinical rotation evaluation; FM, family medicine; GPA, grade point average; IM, internal medicine; OBGYN, obstetrics and gynecology; PE, performance evaluation; PEDS, pediatrics; PSYCH, psychiatry; SOAP, subjective, objective, assessment, and plan; SURGERY, general surgery.
Figure 2.
Clinical activity log numbers and student performance on the Comprehensive Osteopathic Medical Licensing Examination (COMLEX)-USA Level 2-Cognitive Evaluation (CE). Mean across all required clerkships. Abbreviations: FM, family medicine; IM, internal medicine; OBGYN, obstetrics and gynecology; PEDS, pediatrics; PSYCH, psychiatry; SURGERY, general surgery.
Figure 2.
Clinical activity log numbers and student performance on the Comprehensive Osteopathic Medical Licensing Examination (COMLEX)-USA Level 2-Cognitive Evaluation (CE). Mean across all required clerkships. Abbreviations: FM, family medicine; IM, internal medicine; OBGYN, obstetrics and gynecology; PEDS, pediatrics; PSYCH, psychiatry; SURGERY, general surgery.
Table.
Proportion of Variance in Student Performances on National Examination Explained by Predictor Variablea
COMLEX-USA Level 2 COMAT
Predictor Variable CE P Value CE FM P Value PE P Value FM P Value Meanb P Value
CHC site 0.11 .51 0.116 .42 0.280 .50 0.062 .88 0.065 .87
GPA years 1-2 0.50 <.001 0.318 <.001 0.006 .62 0.372 <.001 0.553 <.001
FM clerkship coursework 0.09 .003 0.029 .09 0.004 .68 0.076 .006 0.151 <.001
FM quiz 1 0.04 .06 0.003 .58 NA NA 0.021 .16 NA NA
FM quiz 2c 0.09 .002 0.019 .17 NA NA 0.048 .03 NA NA
FM quiz 2d 0.05 .049 0.010 .33 NA NA 0.013 .27 NA NA
FM SOAP note NA NA NA NA −0.005 .67 NA NA NA NA
FM assignment time −0.05 .03 −0.026 .11 −0.001 .86 −0.011 .30 NA NA
Meanb assignment time 0.04 .06 NA NA NA NA NA NA 0.013 .27
FM reading time −0.01 .36 −0.016 .21 −0.001 .81 −0.011 .29 NA NA
Meanb reading time −0.01 .32 NA NA NA NA NA NA −0.002 .67
Access LO week 1 FM 0.01 .41 0.006 .44 NA NA <0.001 .94 0.007 .42
Access LO week 8 FM 0.01 .49 0.004 .53 NA NA <0.001 .98 <0.001 .80
FM logs −0.02 .13 −0.026 .11 −0.041 .14 −0.027 .10 NA NA
Meanb logs 0.01 .27 NA NA <0.001 .97 NA NA −0.015 .23
FM CRE total 0.12 <.001 0.044 .04 NA NA 0.040 .049 0.067 .01
FM CRE knowledge 0.12 <.001 0.072 .007 NA NA 0.079 .005 0.074 .007
FM CRE history-taking NA NA NA NA 0.026 .26 NA NA NA NA
FM CRE physical examination NA NA NA NA 0.055 .10 NA NA NA NA

a A negative proportion indicates a negative relationship. Proportion of variance explained was reported as R2 from analysis of variance and linear regressions, McFadden R2 from logistic regressions, and uncertainty coefficient from Fisher exact test.

b Mean across all required clerkships (family medicine, internal medicine, pediatrics, obstetrics and gynecology, psychiatry, general surgery).

c Two incomplete quizzes (score of 0) were included in the analysis.

d Two incomplete quizzes were excluded from the analysis.

Abbreviations: CE, cognitive evaluation; CHC, community health center; COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination; CRE, clinical rotation evaluation; FM, family medicine; GPA, grade point average; LO, learning objective; PE, performance evaluation; NA, not applicable; SOAP, subjective, objective, assessment, and plan.

Table.
Proportion of Variance in Student Performances on National Examination Explained by Predictor Variablea
COMLEX-USA Level 2 COMAT
Predictor Variable CE P Value CE FM P Value PE P Value FM P Value Meanb P Value
CHC site 0.11 .51 0.116 .42 0.280 .50 0.062 .88 0.065 .87
GPA years 1-2 0.50 <.001 0.318 <.001 0.006 .62 0.372 <.001 0.553 <.001
FM clerkship coursework 0.09 .003 0.029 .09 0.004 .68 0.076 .006 0.151 <.001
FM quiz 1 0.04 .06 0.003 .58 NA NA 0.021 .16 NA NA
FM quiz 2c 0.09 .002 0.019 .17 NA NA 0.048 .03 NA NA
FM quiz 2d 0.05 .049 0.010 .33 NA NA 0.013 .27 NA NA
FM SOAP note NA NA NA NA −0.005 .67 NA NA NA NA
FM assignment time −0.05 .03 −0.026 .11 −0.001 .86 −0.011 .30 NA NA
Meanb assignment time 0.04 .06 NA NA NA NA NA NA 0.013 .27
FM reading time −0.01 .36 −0.016 .21 −0.001 .81 −0.011 .29 NA NA
Meanb reading time −0.01 .32 NA NA NA NA NA NA −0.002 .67
Access LO week 1 FM 0.01 .41 0.006 .44 NA NA <0.001 .94 0.007 .42
Access LO week 8 FM 0.01 .49 0.004 .53 NA NA <0.001 .98 <0.001 .80
FM logs −0.02 .13 −0.026 .11 −0.041 .14 −0.027 .10 NA NA
Meanb logs 0.01 .27 NA NA <0.001 .97 NA NA −0.015 .23
FM CRE total 0.12 <.001 0.044 .04 NA NA 0.040 .049 0.067 .01
FM CRE knowledge 0.12 <.001 0.072 .007 NA NA 0.079 .005 0.074 .007
FM CRE history-taking NA NA NA NA 0.026 .26 NA NA NA NA
FM CRE physical examination NA NA NA NA 0.055 .10 NA NA NA NA

a A negative proportion indicates a negative relationship. Proportion of variance explained was reported as R2 from analysis of variance and linear regressions, McFadden R2 from logistic regressions, and uncertainty coefficient from Fisher exact test.

b Mean across all required clerkships (family medicine, internal medicine, pediatrics, obstetrics and gynecology, psychiatry, general surgery).

c Two incomplete quizzes (score of 0) were included in the analysis.

d Two incomplete quizzes were excluded from the analysis.

Abbreviations: CE, cognitive evaluation; CHC, community health center; COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination; CRE, clinical rotation evaluation; FM, family medicine; GPA, grade point average; LO, learning objective; PE, performance evaluation; NA, not applicable; SOAP, subjective, objective, assessment, and plan.

×