Free
JAOA/AACOM Medical Education  |   April 2020
Relationship of Clinical Skills Performance in Medical School With COMLEX-USA Level 2-Performance Evaluation
Author Notes
  • From the Rowan University School of Osteopathic Medicine in Stratford, New Jersey. 
  • Financial Disclosures: None reported. 
  • Support: None reported. 
  •  *Address correspondence to Shiyuan Wang, PhD, Rowan University School of Osteopathic Medicine, One Medical Center Drive, Suite 217, Stratford, NJ 08084-1501. Email: wangs1@rowan.edu
     
Article Information
Medical Education / Professional Issues / COMLEX-USA
JAOA/AACOM Medical Education   |   April 2020
Relationship of Clinical Skills Performance in Medical School With COMLEX-USA Level 2-Performance Evaluation
The Journal of the American Osteopathic Association, April 2020, Vol. 120, 245-252. doi:https://doi.org/10.7556/jaoa.2020.041
The Journal of the American Osteopathic Association, April 2020, Vol. 120, 245-252. doi:https://doi.org/10.7556/jaoa.2020.041
Abstract

Context: Osteopathic medical students are required to pass the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Levels 1- and 2-Cognitive Evaluation and COMLEX-USA Level 2-Performance Evaluation (PE) to graduate. Predictors of COMLEX-USA cognitive exam performance are well established, but relatively few studies have explored factors associated with performance on the Level 2-PE.

Objective: To evaluate the relationship between school-based clinical competency assessments (written, simulation, and workplace evaluations) and Level 2-PE performance to ensure that these assessment efforts are effective and grounded in evidence to support student readiness.

Methods: School-based performance measures for 451 first-time takers of COMLEX-USA Level 2-PE were analyzed. A series of Mann-Whitney analyses were applied to compare 3 types of clinical performance assessments against COMLEX-USA Level 2-PE pass-fail performance: (1) internal objective structured clinical examinations (OSCE; average discipline-specific OSCE score and a comprehensive OSCE); (2) national examination performance (average clinical subject Comprehensive Osteopathic Medical Achievement Test, or COMAT, scores and Comprehensive Osteopathic Medical Self-Assessment Examination, or COMSAE, Phase 2 scores); and (3) a workplace-based clinical evaluation by preceptors.

Results: Students who passed the Level 2-PE had a significantly higher average discipline-specific OSCE score, COMSAE Phase 2 performance score, average COMAT score, and individual subject COMAT scores in all subjects except Psychiatry. Students who passed the Level 2-PE humanistic domain also had significantly higher scores in the communication skill component of the school's comprehensive OSCE. Similarly, students who passed the Level 2-PE biomedical domain had significantly higher scores in the data gathering and subjective, objective, assessment, and plan documentation scores on the internal OSCE. The size of these differences (η2) was relatively small. Student performance on the competency-based preceptor evaluation showed no significant relationship with pass-fail performance on the Level 2-PE.

Conclusion: High-stakes OSCEs aligned with the level 2-PE blueprint are effective predictors of performance and are an important way to support student readiness for the Level 2-PE. Other assessments, such as subject-based COMATs and COMSAE Phase 2, add value to school-based assessments over workplace-based assessments.

Osteopathic medical students are required to pass Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Levels 1 and 2-Cognitive Evaluation (CE) and COMLEX-USA Level 2-Performance Evaluation (PE) to graduate medical school in accordance with the Commission on Osteopathic College Accreditation (COCA) standards.1 Numerous studies have shown a predictive correlation between performance on COMLEX-USA Levels 1 and 2-CE,2-4 and preadmission variables such as Medical College Admission Test scores (total or Biology) and preclerkship year performance. Few studies, however, have explored predictive factors correlating to COMLEX-USA Level 2-PE performance, despite the importance of the exam for successful graduation and residency match.5 What is the relationship between school-administered clinical performance assessments and COMLEX-USA Level 2-PE performance? What factors are most effective at predicting COMLEX-USA Level 2-PE success? A better understanding of both answers could help guide medical schools in preparing students for COMLEX-USA Level 2-PE and supporting students’ overall success. In addition, with increased emphasis on competency-based assessments to demonstrate resident readiness at the undergraduate level, determination of student competencies and entrustment has emerged as a new area of focus during the clerkship years.7,8 The relationship between school-administered assessments of clinical skills and COMLEX-USA Level 2-PE performance is important to understand within this context, as it could provide insight on how best to prepare students for future success. 
Since its inception in 2004, the COMLEX-USA Level 2-PE has been a required clinical skills exam that uses standard patients (SP) cases, administered in a time-restricted manner, to assess fundamental core competencies and patient care skills.9,10 Previous studies of preparatory strategies for Level 2-PE demonstrate that a significantly larger percentage of students who passed the PE reported having prerequisite SP encounters at their schools.11 Other studies confirmed this association between school SP-based exams (eg, objective structured clinical examinations or OSCE) and Level 2-PE with actual performance data.5 The use of SP encounters in school curricula help students integrate and apply medical knowledge to clinical contexts and facilitate development of core competencies, therefore better preparing students for the requirements of the national examination.12 As such, use of SP encounters throughout medical school training, including in the preclerkship years, is now common in most medical schools,13,14 and we need more empirical evidence about the relationship between school-administered SP-based exams and Level 2-PE. 
Non-performance–based exams that prompt application of clinical and decision-making skills have also been positively associated with performance-based clinical skill assessments.5,15 For example, the Comprehensive Osteopathic Medical Achievement Test (COMAT) subject examinations, designed to test learning in clerkships, has been demonstrated in previous studies to have a positive correlation with COMLEX-USA Level 2-CE performance,15,16 giving us reason to hypothesize that there could also be a positive correlation with Level 2-PE performance. Now that COMAT subject exams are more commonly used to support decision-making at the clerkship level,9,16 it is important to examine the relationship between COMAT performance and COMLEX-USA Level 2-PE success as well. 
Following the American Association of Colleges of Osteopathic Medicine's (AACOM's) 2012 introduction of 7 Osteopathic Core Competencies (OCCs) for medical students and the 2016 addition of 13 Entrustable Professional Activities (EPAs) at the undergraduate level, which were adapted by AACOM from the American Association of Medical Colleges’ core EPAs, workplace-based assessment of student competencies and clinical skills has grown in importance.17,18 Given the role of the Level 2-PE in assessing OCCs and patient care skills, the relationship between workplace-based assessments and simulation-based exam performance is important to understand. Prior empirical studies have only explored the relationship between workplace clinical evaluations and COMLEX-USA Level 2-CE performance, but found them significantly correlated.15 Again, given that both are performance-based assessments, it is reasonable to hypothesize that there may also be an association between workplace clinical evaluation and Level 2-PE performance, as student exposure to different clinical scenarios, simulated or authentic, should facilitate clinical skills that transfer between different patient care scenarios.19 
At Rowan University School of Osteopathic Medicine (RowanSOM) in Stratford, New Jersey, students in their third year are evaluated on a range of clinical skills performance measures, including COMAT subject exams, COMSAE Phase 2 exam, a range of OSCEs, and a workplace-based evaluation of core competencies. Students are required to take and meet a minimum passing score on COMAT subject exams to pass their clerkships, as well as on the COMSAE Phase 2 to advance to the COMLEX-USA Level 2. 
As for OSCEs, RowanSOM students have been required to pass a school-administered benchmark high-stakes OSCE, the Clinical Skill Comprehensive Exam (CSCE), before they take the Level 2-PE. The CSCE is an 8-station assessment designed to closely align with the format and structure of the COMLEX-USA Level 2-PE. Students are assessed on interpersonal communication skills (ICS), data gathering (DG, including history taking and physical exam), and subjective, objective, assessment, and plan (SOAP) note documentation. Three cases on the CSCE are also used to assess osteopathic principles and practice (OPP) skills. This comprehensive CSCE has been administered and validated for 4 years and has achieved good reliability of 0.9 for the whole OSCE as well as an acceptable level of reliability of 0.8 or above in 3 of the 4 skills areas20,21 across multi year cohorts. (DG has a relatively weaker reliability of 0.61.) 
During the clerkship year, 6 clerkships also administer discipline-specific OSCEs (internal medicine, geriatrics, pediatrics, obstetrics and gynecology, osteopathic manipulative medicine, and neuromuscular and pain management) in which history-taking and physical exam skills are tested, along with ICS, clinical reasoning, and documentation skills. For most clerkships, students are not required to pass the OSCE to receive a passing grade, except in OMM. 
Finally, since 2017, a workplace assessment instrument completed by clinical preceptors has been used to assess student clinical performance on each rotation. The 15-item instrument uses a 5-level behaviorally anchored scale to assess the 6 major medical school competency domains (medical knowledge, patient care, practice-based learning and improvement, interpersonal communication skills, system-based practice, and professionalism). This evaluation is a part of students’ final grade for each clerkship, and each level of the 15 items is assigned a score ranging from 62.5 (unacceptable) to 100 (advanced), with a score of 85 equating to minimum competence. 
The purpose of this study was to examine the relationship between student performance on these clinical skills and competency assessments commonly administered at RowanSOM during third-year medical school training and later performance on the COMLEX-USA Level 2-PE, in hopes of advancing our understanding of predictive associations between these assessments (COMATs, COMSAE Phase 2, OSCEs, and clinical performance evaluations) and Level 2-PE. 
Methods
This retrospective study used a nonexperimental design to assess the relationship between third-year medical school clinical skill performance and performance on the COMLEX-USA Level 2-PE.The sample in this study included students who took the COMLEX-USA Level 2-PE for the first time between June 2016 and May 2019. Given that the COMLEX-USA Level 2-PE is scored on pass/fail scale and there was a nonnormal distribution of performance scores among the students who failed the Level 2-PE, a series of nonparametric Mann-Whitney U tests were applied to assess the differences in medical school performance between students who pass and fail COMLEX-USA Level 2-PE. 
Medical school performance metrics included both objective and subjective measures of clinical skills: (1) national standardized exam performance (COMAT subject Exams and COMSAE Phase 2); (2) clinical skills performance scores (including a high-stakes OSCE and the average OSCE scores over 6 discipline-specific clerkship OSCEs); and (3) workplace competency-based evaluations by physician preceptors during rotations. Effect size (η2) was also calculated to determine the magnitude of the differences between pass/fail groups. 
Seven subject COMATs (excluding Emergency Medicine) were administered during the third year. Each subject COMAT score and a combined average COMAT score were analyzed in relation to COMLEX-USA Level 2-PE performance. The COMSAE Phase 2 was administered toward the end of the third year as a screening test for COMLEX-USA Level 2-CE and was included as a factor in our investigation. 
Because students are required to pass all 4 skills (ICS, DG, SOAP, and OPP) independently on the CSCE to advance to the COMLEX-USA Level 2-PE exam, the relationships between performance in each of the 4 skills and the corresponding 2 COMLEX-USA level 2-PE domains (humanistic and biomedical) were examined separately in this study. Most of RowanSOM's OSCEs assess at least 1 of the 4 fundamental clinical skills; because the number of stations in these OSCEs range from 1 to 6, an average score over all 6 OSCEs was obtained to get a more reliable score for our analysis. 
In the current analysis, an average workplace clinical evaluation score over 9 clerkships (family medicine, internal medicine, geriatrics, pediatrics, obstetrics and gynecology, osteopathic manipulative medicine, neuromuscular & pain management, psychiatry, and surgery) was calculated for analysis in relation to Level 2-PE performance. Then 4 individual competency item scores (history-taking, physical exam, SOAP documentation, and ICS) averaged over the 9 clerkships were also examined in relation to Level 2-PE performance. 
Results
A comparison of performance on different clinical skill assessments between students who passed (n=440) vs those who failed (n=11) Level 2-PE was conducted period. 
Results (Table 1) showed that students who passed COMLEX-USA Level 2-PE had significantly higher average COMAT scores (mean, 101.7 vs. 92.7; Mann-Whitney U=703.5; η2=0.04), as well as higher individual subject COMAT scores (η2 ranges, 0.02 to 0.03) in all subjects except Psychiatry. The comparison of pass/fail groups also showed a statistically significant difference in COMSAE Phase 2 performance (mean, 517.8 vs. 393.1; Mann-Whitney U= 602; η2=0.04). The size of the differences across all performance measures (η2) was small.22,23 
Table 1.
Associations of Objective Clinical Skills Performance and COMLEX-USA Level 2-PE Performance
COMLEX-USA Level 2-PE Overall
Pass (N=440) Mean (SD) Fail (N=11) Mean (SD) Mann-Whitney U Effect Size (η2)
Written Standardized Tests
Mean
 COMAT 101.7 (7.1) 92.7 (4.8) 703.5b 0.04
 COMAT-internal medicine 99.7 (9.7) 91.0 (5.4) 1052.5b 0.02
 COMAT-family medicine 101.4 (9.3) 92.8 (9.0) 1215.5b 0.02
 COMAT-pediatrics 102.8 (9.6) 91.1 (9.5) 950.5b 0.03
 COMAT-OBGYN 99.7 (9.8) 87.6 (8.6) 816.0b 0.03
 COMAT-surgery 101.5 (9.7) 92.5 (11.2) 1175.5b 0.02
 COMAT-OMM 103.0 (8.4) 94.9 (8.1) 1046.5b 0.02
 COMAT-psychiatry 104.1 (9.4) 99.8 (6.9) 1700.5b 0.01
 COMSAE Phase 2 517.8 (98.3) 393.1 (52.0) 602.0b 0.04
SP-Based Performance
 Six subject OSCEs average 80.8 (3.4) 78.4 (2.6) 1365.0b 0.01
COMLEX-USA Level 2-PE Humanistic
 Comprehensive clinical skills exam Pass (N=445) Mean (SD) Fail (N=6) Mean (SD) Mann-Whitney U Effect Size (η2)
 ICS 87.6 (6.0) 82.0 (2.8) 510.5a 0.02
COMLEX-USA Level 2-PE Biomedical
Pass (N=446) Mean (SD) Fail (N=5) Mean (SD) Mann-Whitney U
 Data gathering (history + PE) 73.6 (5.5) 67.4 (5.1) 435.0a 0.02
 OPP 70.7 (6.6) 70.0 (2.8) 955 0
 SOAP 80.0 (5.5) 74.6 (1.1) 376.5a 0.01

a Statistically significant at P=.05.

b Statistically significant at P=.01.

Abbreviations: COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; ICS, interpersonal communication skills; OB/GYN, obstetrics and gynecology; OMM, osteopathic manipulative medicine; OPP, Osteopathic Principles and Practice; PE, Performance Evaluation; SOAP, subjective, objective, assessment, plan.

Table 1.
Associations of Objective Clinical Skills Performance and COMLEX-USA Level 2-PE Performance
COMLEX-USA Level 2-PE Overall
Pass (N=440) Mean (SD) Fail (N=11) Mean (SD) Mann-Whitney U Effect Size (η2)
Written Standardized Tests
Mean
 COMAT 101.7 (7.1) 92.7 (4.8) 703.5b 0.04
 COMAT-internal medicine 99.7 (9.7) 91.0 (5.4) 1052.5b 0.02
 COMAT-family medicine 101.4 (9.3) 92.8 (9.0) 1215.5b 0.02
 COMAT-pediatrics 102.8 (9.6) 91.1 (9.5) 950.5b 0.03
 COMAT-OBGYN 99.7 (9.8) 87.6 (8.6) 816.0b 0.03
 COMAT-surgery 101.5 (9.7) 92.5 (11.2) 1175.5b 0.02
 COMAT-OMM 103.0 (8.4) 94.9 (8.1) 1046.5b 0.02
 COMAT-psychiatry 104.1 (9.4) 99.8 (6.9) 1700.5b 0.01
 COMSAE Phase 2 517.8 (98.3) 393.1 (52.0) 602.0b 0.04
SP-Based Performance
 Six subject OSCEs average 80.8 (3.4) 78.4 (2.6) 1365.0b 0.01
COMLEX-USA Level 2-PE Humanistic
 Comprehensive clinical skills exam Pass (N=445) Mean (SD) Fail (N=6) Mean (SD) Mann-Whitney U Effect Size (η2)
 ICS 87.6 (6.0) 82.0 (2.8) 510.5a 0.02
COMLEX-USA Level 2-PE Biomedical
Pass (N=446) Mean (SD) Fail (N=5) Mean (SD) Mann-Whitney U
 Data gathering (history + PE) 73.6 (5.5) 67.4 (5.1) 435.0a 0.02
 OPP 70.7 (6.6) 70.0 (2.8) 955 0
 SOAP 80.0 (5.5) 74.6 (1.1) 376.5a 0.01

a Statistically significant at P=.05.

b Statistically significant at P=.01.

Abbreviations: COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; ICS, interpersonal communication skills; OB/GYN, obstetrics and gynecology; OMM, osteopathic manipulative medicine; OPP, Osteopathic Principles and Practice; PE, Performance Evaluation; SOAP, subjective, objective, assessment, plan.

×
An analysis of OSCE performance between groups demonstrated that the passing group had a statistically significant higher average discipline-specific OSCE score (mean, 80.8 vs. 78.4; Mann-Whitney U=1365.0; η2=0.01). Similarly, in a comparison of the 4 individual clinical skills scores for RowanSOM's CSCE to the Level 2-PE humanistic and biomedical subdomain performance, the group that passed the humanistic domain had statistically significant higher scores in ICS on their CSCE (mean, 87.6 vs 82.0; Mann-Whitney U=510.5; η2=0.02) and those who passed the biomedical domain had significantly higher data gathering (mean, 73.6 vs. 67.4; Mann-Whitney U=435.0; η2=0.02) and SOAP note documentation scores (mean, 80.0 vs. 74.6; Mann-Whitney U=376.5; η2=0.01). The sizes of these differences in OSCEs performance (η2) were all small. 
Last, in a comparison of workplace-based assessment performance between groups (n=174, because the school started to implement this assessment in academic year 2017-2018), students who passed the Level 2-PE did not have significantly higher average overall scores or higher performance in the individual competency scores for history taking, physical exam, documentation, or ICS (Table 2). 
Table 2.
Associations of Subjective Clinical Skills Performance and COMLEX-USA Level 2-PE Performancea
COMLEX-USA Level 2-PE Overall
Pass (n=172) Mean (SD) Fail (n=4) Mean (SD) Mann-Whitney U Effect Size (η2)
Workplace-Based Competency Evaluationa
 Mean clinical evaluation score 89.3 (2.5) 87.4 (2.0) 174 0.02
COMLEX-USA Level 2-PE Humanistic
 Individual competency score Pass (N=174) Mean (SD) Fail (N=2) Mean (SD)
 ICS 89.7 (2.7) 88.4 (0.3) 116.0 0
COMLEX-USA Level 2-PE Biomedical
Pass (N=174) Mean (SD) Fail (N=2) Mean (SD)
 History taking 88.6 (2.6) 89.5 (0.6) 235.0 0
 Physical exam 87.9 (2.5) 88.6 (1.9) 216.0 0
 SOAP documentation 88.6 (2.7) 89.3 (2.6) 203.0 0

a The smaller sample size is because workplace-based competency evaluation was implemented in academic year 2017-2018.

Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; ICS, interpersonal communication skills; PE, Performance Evaluation; SOAP, subjective, objective, assessment, plan.

Table 2.
Associations of Subjective Clinical Skills Performance and COMLEX-USA Level 2-PE Performancea
COMLEX-USA Level 2-PE Overall
Pass (n=172) Mean (SD) Fail (n=4) Mean (SD) Mann-Whitney U Effect Size (η2)
Workplace-Based Competency Evaluationa
 Mean clinical evaluation score 89.3 (2.5) 87.4 (2.0) 174 0.02
COMLEX-USA Level 2-PE Humanistic
 Individual competency score Pass (N=174) Mean (SD) Fail (N=2) Mean (SD)
 ICS 89.7 (2.7) 88.4 (0.3) 116.0 0
COMLEX-USA Level 2-PE Biomedical
Pass (N=174) Mean (SD) Fail (N=2) Mean (SD)
 History taking 88.6 (2.6) 89.5 (0.6) 235.0 0
 Physical exam 87.9 (2.5) 88.6 (1.9) 216.0 0
 SOAP documentation 88.6 (2.7) 89.3 (2.6) 203.0 0

a The smaller sample size is because workplace-based competency evaluation was implemented in academic year 2017-2018.

Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; ICS, interpersonal communication skills; PE, Performance Evaluation; SOAP, subjective, objective, assessment, plan.

×
Discussion
The current study examined the relationships between different measures of school-based clinical skills performance at RowanSOM with subsequent COMLEX-USA Level 2-PE performance. Our results add to the empirical evidence about strategies to support student success on the COMLEX-USA Level 2-PE and provide further insight into the relationships between different clinical skills and competency assessments and student exam performance. 
The study findings of small but significant associations between the average OSCE score and Level 2-PE performance was consistent with previous studies and provides further evidence of the value of school-based standardized patient exams, particularly high-stakes OSCEs, as a strategy to support student success on the Level 2-PE.11 Our results also showed significant relationships between the 3 clinical skills assessed on RowanSOM's comprehensive OSCE (ICS, DG, and SOAP note documentation) and Level 2-PE subcomponent (humanistic and biomedical domains) performance. Compared with previous studies that demonstrated a limited relationship between school-based clinical skills assessments and the Level 2-PE, likely due to less reliable SP-based exams,5 our results extended the empirical evidence of the correlation between specific clinical skills in SP-based exams and subcomponents of Level 2-PE and provided support for our school policy, which requires all students to distinctly pass each of the 4 clinical skills assessments on the comprehensive OSCE before taking the Level 2-PE. The OPP score was the only component score that did not show a significant difference between students who passed and those who failed the Level 2-PE, possibly due to the limited number of cases and checklist items that assessed OPP. 
The results of our study also provide support for expanding clinical skills assessment for both formative and summative evaluations earlier in the curriculum. A new CSCE benchmark has been added for all students at the end of their preclerkship training to establish readiness for clerkships. Meanwhile, efforts are ongoing to improve the rigor and reliability of current discipline-specific OSCEs during clerkships and further enhance testing before entering residency. These strategies are designed to assess student progress in meeting competency milestones across all 4 years of the curriculum, as well as to prepare students to pass their board exams and to ensure their readiness for residency. 
The association between COMAT scores (both average COMAT scores and individual subject COMATs) and Level 2-PE performance also suggests that the COMAT exams are useful tools to help identify and remediate student weakness in clinical skills before they take the Level 2-PE. Average COMAT score results showed the largest difference between students who passed and failed Level 2-PE, suggesting that a comprehensive view and use of COMAT exams may contribute additional insight into student performance as they prepare for COMLEX-USA. Individual subject COMATs likewise support student COMLEX-USA performance. Pediatrics and Obstetrics and Gynecology COMATs seemed to have the most associations with Level 2-PE, followed by Family Medicine, Internal Medicine, Surgery, and OPP. This pattern of association is similar to results from other studies on COMATs and COMLEX-USA Level 2-CE.15 Currently, COMAT performance is being used to counsel students in their preparation for the Level 2-CE. Results of this study suggest that COMAT performance may be also useful in guiding student preparation for the Level 2-PE. Similarly, our data suggest that the COMSAE Phase 2 is associated with Level 2-PE performance and may serve as an additional tool in determining student readiness for the Level 2-PE as well. 
Our results did not show a significant association between our workplace-based clinical evaluation of OCC and the Level 2-PE. Many factors may help to explain this outcome. First, there is often a lack of differentiation in preceptor ratings across students, limiting the ability to establish any correlation.15 Others have suggested that SP-based performance is not necessarily transferrable to the workplace due to the context-specific nature of both types of assessments,19,24 which indicates they each assesses different types of clinical skills and both should be implemented into clerkship training in undergraduate medical training. However, previous research on COMLEX-USA Level 2-CE scores did find a moderate correlation with clinical performance evaluation, albeit in a much larger sample size and between continuous performance measures.15 Our analysis of the relationship between Level 2-PE outcomes and workplace-based assessment was also limited by sample size, which was too small to have the statistical power to determine a relationship between individual clinical skills items on the instrument and overall pass/fail performance on the Level 2-PE. 
Conclusion
The generalizability of any study at a single institution is limited. However, the results of this study do provide further evidence in support of previous studies and specifically offers insight for osteopathic medical schools and COMLEX-USA Level 2-PE outcomes. School-based OSCEs, especially high-stakes OSCEs aligned with the Level 2-PE blueprint, clearly support student success. Other clinically related assessment tools, including COMAT exams and the COMSAE Phase 2 exam, can also provide important information to guide student preparation and to support decisions about a student's readiness for the exam. As additional Level 2-PE performance outcomes are collected and an adequate sample is accumulated for both study groups, the initial outcomes from this study will be used to help establish a predictive model of student success and ultimately be used to inform the effectiveness of school-based assessment strategies. Additionally, as we continue to collect data on the workplace-based instrument and expand our sample size, we will be able to further explore this relationship and advance our effective use of preceptor ratings to inform the development of clinical skills and to train better physicians. 
References
Commission on Osteopathic College Accreditation. Accreditation of Colleges of Osteopathic medicine: COM Continuing Accreditation Standards. Chicago, IL: Commission on Osteopathic College Accreditation; 2019.
Sefcik DJ, Prozlaleck WC, O'Hare TH. Characteristics of the courses that best predict COMLEX-USA level 1 performance. J Am Osteopath Assoc. 2003;103(10):491-494. [PubMed]
Dixon D. Relation between variables of preadmission, medical school performance, and COMLEX–USA levels 1 and 2 performance. J Am Osteopath Assoc. 2004;104(8):332-336. [PubMed]
Maholtz DE, Erickson MJ, Cymet T. Comprehensive osteopathic medical licensing examination-USA level 1 and level 2-cognitive evaluation preparation and outcomes. J Am Osteopath Assoc. . 2015;115(4):232-235. [CrossRef] [PubMed]
Baker HH, Cope MK, Adelman MD, Schuler S, Foster RW, Gimpel JR. Relationships between scores on the COMLEX-USA level 2-performance evaluation and selected school performance measures. J Am Osteopath Assoc. 2006;106(5):290-295. [PubMed]
Agahi F, Speicher MR, Cisek G. Association between undergraduate performance predictors and academic and clinical performance of osteopathic medical students. J Am Osteopath Assoc. . 2018;118(2):106-114. [CrossRef] [PubMed]
Veale P, Busche K, Touchie C, Coderre S. Choosing your own pathway to competency-based undergraduate medical education. Acad Med. 2019;94(1):25-30. [CrossRef] [PubMed]
ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010;32(8):669-675. doi: 10.3109/0142159x.2010.500897 [CrossRef] [PubMed]
National Board of Osteopathic Medical Examiners. Evidence-Based Assessment Impacting Patient Care Annual Report. Chicago, IL: National Board of Osteopathic Medical Examiners; 2019.
Weidner AC, Gimpel JR, Boulet JR, Solomon M. Using standardized patients to assess the communication skills of graduating physicians for the comprehensive medical licensing examination (COMLEX) level-2 performance evaluation (level-2 PE). Teach Learn Med. 2010;22(1):8-15. [CrossRef] [PubMed]
Sandella JM, Peters A, Smith LL, Gimpel JR. Preparation strategies of osteopathic medical students for the COMLEX-USA level 2-PE. J Am Osteopath Assoc. 2016;116(4):234. doi: 10.7556/jaoa.2016.048 [CrossRef] [PubMed]
Lockwood MD, Tucker-Potter S, Sargentini NJ. Curricular analysis of competency-based osteopathic medical education: application of a matrix for quality enhancement to a standardized patient encounter example. J Am Osteopath Assoc. 2009;109(9):486-500. [PubMed]
Langenau EE, Dyer C, Roberts WL, Wilson C, Gimpel J. Five-year summary of COMLEX-USA level 2-PE examinee performance and survey data. J Am Osteopath Assoc. 2010;110(3):114-125. [PubMed]
Gimpel JR, Weidner AC, Boulet JR, Wilson C, Errichetti AM. Standardized patients and mechanical simulators in teaching and assessment at colleges of osteopathic medicine. J Am Osteopath Assoc. . 2007;107(12):557-561. [PubMed]
Dixon D. Prediction of osteopathic medical school performance on the basis of MCAT score, GPA, sex, undergraduate major, and undergraduate institution. . J Am Osteopath Assoc. 2012;112(4):175-181. [PubMed]
Li F, Kalinowski KE, Song H, Bates BP. Relationships between the comprehensive osteopathic medical achievement test (COMAT) subject examinations and the COMLEX-USA level 2-cognitive evaluation. J Am Osteopath Assoc. 2014;114(9):714-721. [CrossRef] [PubMed]
American Association of Colleges of Osteopathic Medicine. Osteopathic Core Competencies for Medical Students. Chevy Chase, MD: American Association of Colleges of Osteopathic Medicine; 2012.
American Association of Colleges of Osteopathic Medicine. Osteopathic Considerations for Core Entrustable Professional Activities (EPAs) for Entering Residency. Chevy Chase, MD: American Association of Colleges of Osteopathic Medicine; 2012.
Mattick K, Dennis I, Bradley P, Bligh J. Content specificity: is it the full story? statistical modelling of a clinical skills examination. Med Ed. 2008;42:589-599. doi: 10.1111/j.1365-2923.2008.03020.x [CrossRef]
Khan KZ, Gaunt K, Ramachandran S, Pushkar P. The objective structured clinical examination (OSCE): AMEE guide no. 81. Part II: organization and administration. Med Teach. 2013;35(9):1447-1463. [CrossRef]
Daniels VJ, Pugh D. Twelve tips for developing an OSCE that measures what you want. Med Teach. 2018;40(12):1208-1213. [CrossRef] [PubMed]
Fritz CO, Morris PE, Richler JJ. Effect size estimates: current use, calculations, and interpretation. J Exp Psychol Gen. 2012;141(1):2-18. doi: 10.1037/a0024338 [CrossRef] [PubMed]
Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. New York, NY: Lawrence Erlbaum Associates; 1988.
Sargeant J, Eva KW, Armson H, et al. Features of assessment learners use to make informed self-assessments of clinical performance. Med Ed. 2011;45:636-647. doi: 10.1111/j.1365-2923.2010.03888.x [CrossRef]
Table 1.
Associations of Objective Clinical Skills Performance and COMLEX-USA Level 2-PE Performance
COMLEX-USA Level 2-PE Overall
Pass (N=440) Mean (SD) Fail (N=11) Mean (SD) Mann-Whitney U Effect Size (η2)
Written Standardized Tests
Mean
 COMAT 101.7 (7.1) 92.7 (4.8) 703.5b 0.04
 COMAT-internal medicine 99.7 (9.7) 91.0 (5.4) 1052.5b 0.02
 COMAT-family medicine 101.4 (9.3) 92.8 (9.0) 1215.5b 0.02
 COMAT-pediatrics 102.8 (9.6) 91.1 (9.5) 950.5b 0.03
 COMAT-OBGYN 99.7 (9.8) 87.6 (8.6) 816.0b 0.03
 COMAT-surgery 101.5 (9.7) 92.5 (11.2) 1175.5b 0.02
 COMAT-OMM 103.0 (8.4) 94.9 (8.1) 1046.5b 0.02
 COMAT-psychiatry 104.1 (9.4) 99.8 (6.9) 1700.5b 0.01
 COMSAE Phase 2 517.8 (98.3) 393.1 (52.0) 602.0b 0.04
SP-Based Performance
 Six subject OSCEs average 80.8 (3.4) 78.4 (2.6) 1365.0b 0.01
COMLEX-USA Level 2-PE Humanistic
 Comprehensive clinical skills exam Pass (N=445) Mean (SD) Fail (N=6) Mean (SD) Mann-Whitney U Effect Size (η2)
 ICS 87.6 (6.0) 82.0 (2.8) 510.5a 0.02
COMLEX-USA Level 2-PE Biomedical
Pass (N=446) Mean (SD) Fail (N=5) Mean (SD) Mann-Whitney U
 Data gathering (history + PE) 73.6 (5.5) 67.4 (5.1) 435.0a 0.02
 OPP 70.7 (6.6) 70.0 (2.8) 955 0
 SOAP 80.0 (5.5) 74.6 (1.1) 376.5a 0.01

a Statistically significant at P=.05.

b Statistically significant at P=.01.

Abbreviations: COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; ICS, interpersonal communication skills; OB/GYN, obstetrics and gynecology; OMM, osteopathic manipulative medicine; OPP, Osteopathic Principles and Practice; PE, Performance Evaluation; SOAP, subjective, objective, assessment, plan.

Table 1.
Associations of Objective Clinical Skills Performance and COMLEX-USA Level 2-PE Performance
COMLEX-USA Level 2-PE Overall
Pass (N=440) Mean (SD) Fail (N=11) Mean (SD) Mann-Whitney U Effect Size (η2)
Written Standardized Tests
Mean
 COMAT 101.7 (7.1) 92.7 (4.8) 703.5b 0.04
 COMAT-internal medicine 99.7 (9.7) 91.0 (5.4) 1052.5b 0.02
 COMAT-family medicine 101.4 (9.3) 92.8 (9.0) 1215.5b 0.02
 COMAT-pediatrics 102.8 (9.6) 91.1 (9.5) 950.5b 0.03
 COMAT-OBGYN 99.7 (9.8) 87.6 (8.6) 816.0b 0.03
 COMAT-surgery 101.5 (9.7) 92.5 (11.2) 1175.5b 0.02
 COMAT-OMM 103.0 (8.4) 94.9 (8.1) 1046.5b 0.02
 COMAT-psychiatry 104.1 (9.4) 99.8 (6.9) 1700.5b 0.01
 COMSAE Phase 2 517.8 (98.3) 393.1 (52.0) 602.0b 0.04
SP-Based Performance
 Six subject OSCEs average 80.8 (3.4) 78.4 (2.6) 1365.0b 0.01
COMLEX-USA Level 2-PE Humanistic
 Comprehensive clinical skills exam Pass (N=445) Mean (SD) Fail (N=6) Mean (SD) Mann-Whitney U Effect Size (η2)
 ICS 87.6 (6.0) 82.0 (2.8) 510.5a 0.02
COMLEX-USA Level 2-PE Biomedical
Pass (N=446) Mean (SD) Fail (N=5) Mean (SD) Mann-Whitney U
 Data gathering (history + PE) 73.6 (5.5) 67.4 (5.1) 435.0a 0.02
 OPP 70.7 (6.6) 70.0 (2.8) 955 0
 SOAP 80.0 (5.5) 74.6 (1.1) 376.5a 0.01

a Statistically significant at P=.05.

b Statistically significant at P=.01.

Abbreviations: COMAT, Comprehensive Osteopathic Medical Achievement Test; COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; ICS, interpersonal communication skills; OB/GYN, obstetrics and gynecology; OMM, osteopathic manipulative medicine; OPP, Osteopathic Principles and Practice; PE, Performance Evaluation; SOAP, subjective, objective, assessment, plan.

×
Table 2.
Associations of Subjective Clinical Skills Performance and COMLEX-USA Level 2-PE Performancea
COMLEX-USA Level 2-PE Overall
Pass (n=172) Mean (SD) Fail (n=4) Mean (SD) Mann-Whitney U Effect Size (η2)
Workplace-Based Competency Evaluationa
 Mean clinical evaluation score 89.3 (2.5) 87.4 (2.0) 174 0.02
COMLEX-USA Level 2-PE Humanistic
 Individual competency score Pass (N=174) Mean (SD) Fail (N=2) Mean (SD)
 ICS 89.7 (2.7) 88.4 (0.3) 116.0 0
COMLEX-USA Level 2-PE Biomedical
Pass (N=174) Mean (SD) Fail (N=2) Mean (SD)
 History taking 88.6 (2.6) 89.5 (0.6) 235.0 0
 Physical exam 87.9 (2.5) 88.6 (1.9) 216.0 0
 SOAP documentation 88.6 (2.7) 89.3 (2.6) 203.0 0

a The smaller sample size is because workplace-based competency evaluation was implemented in academic year 2017-2018.

Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; ICS, interpersonal communication skills; PE, Performance Evaluation; SOAP, subjective, objective, assessment, plan.

Table 2.
Associations of Subjective Clinical Skills Performance and COMLEX-USA Level 2-PE Performancea
COMLEX-USA Level 2-PE Overall
Pass (n=172) Mean (SD) Fail (n=4) Mean (SD) Mann-Whitney U Effect Size (η2)
Workplace-Based Competency Evaluationa
 Mean clinical evaluation score 89.3 (2.5) 87.4 (2.0) 174 0.02
COMLEX-USA Level 2-PE Humanistic
 Individual competency score Pass (N=174) Mean (SD) Fail (N=2) Mean (SD)
 ICS 89.7 (2.7) 88.4 (0.3) 116.0 0
COMLEX-USA Level 2-PE Biomedical
Pass (N=174) Mean (SD) Fail (N=2) Mean (SD)
 History taking 88.6 (2.6) 89.5 (0.6) 235.0 0
 Physical exam 87.9 (2.5) 88.6 (1.9) 216.0 0
 SOAP documentation 88.6 (2.7) 89.3 (2.6) 203.0 0

a The smaller sample size is because workplace-based competency evaluation was implemented in academic year 2017-2018.

Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA; ICS, interpersonal communication skills; PE, Performance Evaluation; SOAP, subjective, objective, assessment, plan.

×