Abstract
Context:
Osteopathic medical students are required to pass the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Levels 1- and 2-Cognitive Evaluation and COMLEX-USA Level 2-Performance Evaluation (PE) to graduate. Predictors of COMLEX-USA cognitive exam performance are well established, but relatively few studies have explored factors associated with performance on the Level 2-PE.
Objective:
To evaluate the relationship between school-based clinical competency assessments (written, simulation, and workplace evaluations) and Level 2-PE performance to ensure that these assessment efforts are effective and grounded in evidence to support student readiness.
Methods:
School-based performance measures for 451 first-time takers of COMLEX-USA Level 2-PE were analyzed. A series of Mann-Whitney analyses were applied to compare 3 types of clinical performance assessments against COMLEX-USA Level 2-PE pass-fail performance: (1) internal objective structured clinical examinations (OSCE; average discipline-specific OSCE score and a comprehensive OSCE); (2) national examination performance (average clinical subject Comprehensive Osteopathic Medical Achievement Test, or COMAT, scores and Comprehensive Osteopathic Medical Self-Assessment Examination, or COMSAE, Phase 2 scores); and (3) a workplace-based clinical evaluation by preceptors.
Results:
Students who passed the Level 2-PE had a significantly higher average discipline-specific OSCE score, COMSAE Phase 2 performance score, average COMAT score, and individual subject COMAT scores in all subjects except Psychiatry. Students who passed the Level 2-PE humanistic domain also had significantly higher scores in the communication skill component of the school's comprehensive OSCE. Similarly, students who passed the Level 2-PE biomedical domain had significantly higher scores in the data gathering and subjective, objective, assessment, and plan documentation scores on the internal OSCE. The size of these differences (η2) was relatively small. Student performance on the competency-based preceptor evaluation showed no significant relationship with pass-fail performance on the Level 2-PE.
Conclusion:
High-stakes OSCEs aligned with the level 2-PE blueprint are effective predictors of performance and are an important way to support student readiness for the Level 2-PE. Other assessments, such as subject-based COMATs and COMSAE Phase 2, add value to school-based assessments over workplace-based assessments.
Osteopathic medical students are required to pass Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Levels 1 and 2-Cognitive Evaluation (CE) and COMLEX-USA Level 2-Performance Evaluation (PE) to graduate medical school in accordance with the Commission on Osteopathic College Accreditation (COCA) standards.
1 Numerous studies have shown a predictive correlation between performance on COMLEX-USA Levels 1 and 2-CE,
2-4 and preadmission variables such as Medical College Admission Test scores (total or Biology) and preclerkship year performance. Few studies, however, have explored predictive factors correlating to COMLEX-USA Level 2-PE performance, despite the importance of the exam for successful graduation and residency match.
5 What is the relationship between school-administered clinical performance assessments and COMLEX-USA Level 2-PE performance? What factors are most effective at predicting COMLEX-USA Level 2-PE success? A better understanding of both answers could help guide medical schools in preparing students for COMLEX-USA Level 2-PE and supporting students’ overall success. In addition, with increased emphasis on competency-based assessments to demonstrate resident readiness at the undergraduate level, determination of student competencies and entrustment has emerged as a new area of focus during the clerkship years.
7,8 The relationship between school-administered assessments of clinical skills and COMLEX-USA Level 2-PE performance is important to understand within this context, as it could provide insight on how best to prepare students for future success.
Since its inception in 2004, the COMLEX-USA Level 2-PE has been a required clinical skills exam that uses standard patients (SP) cases, administered in a time-restricted manner, to assess fundamental core competencies and patient care skills.
9,10 Previous studies of preparatory strategies for Level 2-PE demonstrate that a significantly larger percentage of students who passed the PE reported having prerequisite SP encounters at their schools.
11 Other studies confirmed this association between school SP-based exams (eg, objective structured clinical examinations or OSCE) and Level 2-PE with actual performance data.
5 The use of SP encounters in school curricula help students integrate and apply medical knowledge to clinical contexts and facilitate development of core competencies, therefore better preparing students for the requirements of the national examination.
12 As such, use of SP encounters throughout medical school training, including in the preclerkship years, is now common in most medical schools,
13,14 and we need more empirical evidence about the relationship between school-administered SP-based exams and Level 2-PE.
Non-performance–based exams that prompt application of clinical and decision-making skills have also been positively associated with performance-based clinical skill assessments.
5,15 For example, the Comprehensive Osteopathic Medical Achievement Test (COMAT) subject examinations, designed to test learning in clerkships, has been demonstrated in previous studies to have a positive correlation with COMLEX-USA Level 2-CE performance,
15,16 giving us reason to hypothesize that there could also be a positive correlation with Level 2-PE performance. Now that COMAT subject exams are more commonly used to support decision-making at the clerkship level,
9,16 it is important to examine the relationship between COMAT performance and COMLEX-USA Level 2-PE success as well.
Following the American Association of Colleges of Osteopathic Medicine's (AACOM's) 2012 introduction of 7 Osteopathic Core Competencies (OCCs) for medical students and the 2016 addition of 13 Entrustable Professional Activities (EPAs) at the undergraduate level, which were adapted by AACOM from the American Association of Medical Colleges’ core EPAs, workplace-based assessment of student competencies and clinical skills has grown in importance.
17,18 Given the role of the Level 2-PE in assessing OCCs and patient care skills, the relationship between workplace-based assessments and simulation-based exam performance is important to understand. Prior empirical studies have only explored the relationship between workplace clinical evaluations and COMLEX-USA Level 2-CE performance, but found them significantly correlated.
15 Again, given that both are performance-based assessments, it is reasonable to hypothesize that there may also be an association between workplace clinical evaluation and Level 2-PE performance, as student exposure to different clinical scenarios, simulated or authentic, should facilitate clinical skills that transfer between different patient care scenarios.
19
At Rowan University School of Osteopathic Medicine (RowanSOM) in Stratford, New Jersey, students in their third year are evaluated on a range of clinical skills performance measures, including COMAT subject exams, COMSAE Phase 2 exam, a range of OSCEs, and a workplace-based evaluation of core competencies. Students are required to take and meet a minimum passing score on COMAT subject exams to pass their clerkships, as well as on the COMSAE Phase 2 to advance to the COMLEX-USA Level 2.
As for OSCEs, RowanSOM students have been required to pass a school-administered benchmark high-stakes OSCE, the Clinical Skill Comprehensive Exam (CSCE), before they take the Level 2-PE. The CSCE is an 8-station assessment designed to closely align with the format and structure of the COMLEX-USA Level 2-PE. Students are assessed on interpersonal communication skills (ICS), data gathering (DG, including history taking and physical exam), and subjective, objective, assessment, and plan (SOAP) note documentation. Three cases on the CSCE are also used to assess osteopathic principles and practice (OPP) skills. This comprehensive CSCE has been administered and validated for 4 years and has achieved good reliability of 0.9 for the whole OSCE as well as an acceptable level of reliability of 0.8 or above in 3 of the 4 skills areas
20,21 across multi year cohorts. (DG has a relatively weaker reliability of 0.61.)
During the clerkship year, 6 clerkships also administer discipline-specific OSCEs (internal medicine, geriatrics, pediatrics, obstetrics and gynecology, osteopathic manipulative medicine, and neuromuscular and pain management) in which history-taking and physical exam skills are tested, along with ICS, clinical reasoning, and documentation skills. For most clerkships, students are not required to pass the OSCE to receive a passing grade, except in OMM.
Finally, since 2017, a workplace assessment instrument completed by clinical preceptors has been used to assess student clinical performance on each rotation. The 15-item instrument uses a 5-level behaviorally anchored scale to assess the 6 major medical school competency domains (medical knowledge, patient care, practice-based learning and improvement, interpersonal communication skills, system-based practice, and professionalism). This evaluation is a part of students’ final grade for each clerkship, and each level of the 15 items is assigned a score ranging from 62.5 (unacceptable) to 100 (advanced), with a score of 85 equating to minimum competence.
The purpose of this study was to examine the relationship between student performance on these clinical skills and competency assessments commonly administered at RowanSOM during third-year medical school training and later performance on the COMLEX-USA Level 2-PE, in hopes of advancing our understanding of predictive associations between these assessments (COMATs, COMSAE Phase 2, OSCEs, and clinical performance evaluations) and Level 2-PE.
This retrospective study used a nonexperimental design to assess the relationship between third-year medical school clinical skill performance and performance on the COMLEX-USA Level 2-PE.The sample in this study included students who took the COMLEX-USA Level 2-PE for the first time between June 2016 and May 2019. Given that the COMLEX-USA Level 2-PE is scored on pass/fail scale and there was a nonnormal distribution of performance scores among the students who failed the Level 2-PE, a series of nonparametric Mann-Whitney U tests were applied to assess the differences in medical school performance between students who pass and fail COMLEX-USA Level 2-PE.
Medical school performance metrics included both objective and subjective measures of clinical skills: (1) national standardized exam performance (COMAT subject Exams and COMSAE Phase 2); (2) clinical skills performance scores (including a high-stakes OSCE and the average OSCE scores over 6 discipline-specific clerkship OSCEs); and (3) workplace competency-based evaluations by physician preceptors during rotations. Effect size (η2) was also calculated to determine the magnitude of the differences between pass/fail groups.
Seven subject COMATs (excluding Emergency Medicine) were administered during the third year. Each subject COMAT score and a combined average COMAT score were analyzed in relation to COMLEX-USA Level 2-PE performance. The COMSAE Phase 2 was administered toward the end of the third year as a screening test for COMLEX-USA Level 2-CE and was included as a factor in our investigation.
Because students are required to pass all 4 skills (ICS, DG, SOAP, and OPP) independently on the CSCE to advance to the COMLEX-USA Level 2-PE exam, the relationships between performance in each of the 4 skills and the corresponding 2 COMLEX-USA level 2-PE domains (humanistic and biomedical) were examined separately in this study. Most of RowanSOM's OSCEs assess at least 1 of the 4 fundamental clinical skills; because the number of stations in these OSCEs range from 1 to 6, an average score over all 6 OSCEs was obtained to get a more reliable score for our analysis.
In the current analysis, an average workplace clinical evaluation score over 9 clerkships (family medicine, internal medicine, geriatrics, pediatrics, obstetrics and gynecology, osteopathic manipulative medicine, neuromuscular & pain management, psychiatry, and surgery) was calculated for analysis in relation to Level 2-PE performance. Then 4 individual competency item scores (history-taking, physical exam, SOAP documentation, and ICS) averaged over the 9 clerkships were also examined in relation to Level 2-PE performance.
A comparison of performance on different clinical skill assessments between students who passed (n=440) vs those who failed (n=11) Level 2-PE was conducted period.
Results (
Table 1) showed that students who passed COMLEX-USA Level 2-PE had significantly higher average COMAT scores (mean, 101.7 vs. 92.7; Mann-Whitney
U=703.5; η
2=0.04), as well as higher individual subject COMAT scores (η
2 ranges, 0.02 to 0.03) in all subjects except Psychiatry. The comparison of pass/fail groups also showed a statistically significant difference in COMSAE Phase 2 performance (mean, 517.8 vs. 393.1; Mann-Whitney
U= 602; η
2=0.04). The size of the differences across all performance measures (η
2) was small.
22,23
Table 1.
Associations of Objective Clinical Skills Performance and COMLEX-USA Level 2-PE Performance
| COMLEX-USA Level 2-PE Overall | | |
| Pass (N=440)
Mean (SD) | Fail (N=11)
Mean (SD) | Mann-Whitney U | Effect Size (η2) |
| Written Standardized Tests | | | | |
| Mean | | | | |
| COMAT | 101.7 (7.1) | 92.7 (4.8) | 703.5b | 0.04 |
| COMAT-internal medicine | 99.7 (9.7) | 91.0 (5.4) | 1052.5b | 0.02 |
| COMAT-family medicine | 101.4 (9.3) | 92.8 (9.0) | 1215.5b | 0.02 |
| COMAT-pediatrics | 102.8 (9.6) | 91.1 (9.5) | 950.5b | 0.03 |
| COMAT-OBGYN | 99.7 (9.8) | 87.6 (8.6) | 816.0b | 0.03 |
| COMAT-surgery | 101.5 (9.7) | 92.5 (11.2) | 1175.5b | 0.02 |
| COMAT-OMM | 103.0 (8.4) | 94.9 (8.1) | 1046.5b | 0.02 |
| COMAT-psychiatry | 104.1 (9.4) | 99.8 (6.9) | 1700.5b | 0.01 |
| COMSAE Phase 2 | 517.8 (98.3) | 393.1 (52.0) | 602.0b | 0.04 |
| SP-Based Performance | | | | |
| Six subject OSCEs average | 80.8 (3.4) | 78.4 (2.6) | 1365.0b | 0.01 |
| COMLEX-USA Level 2-PE Humanistic | | |
| Comprehensive clinical skills exam | Pass (N=445)
Mean (SD) | Fail (N=6)
Mean (SD) | Mann-Whitney U | Effect Size (η2) |
| ICS | 87.6 (6.0) | 82.0 (2.8) | 510.5a | 0.02 |
| COMLEX-USA Level 2-PE Biomedical | | |
| Pass (N=446)
Mean (SD) | Fail (N=5)
Mean (SD) | Mann-Whitney U | |
| Data gathering (history + PE) | 73.6 (5.5) | 67.4 (5.1) | 435.0a | 0.02 |
| OPP | 70.7 (6.6) | 70.0 (2.8) | 955 | 0 |
| SOAP | 80.0 (5.5) | 74.6 (1.1) | 376.5a | 0.01 |
Table 1.
Associations of Objective Clinical Skills Performance and COMLEX-USA Level 2-PE Performance
| COMLEX-USA Level 2-PE Overall | | |
| Pass (N=440)
Mean (SD) | Fail (N=11)
Mean (SD) | Mann-Whitney U | Effect Size (η2) |
| Written Standardized Tests | | | | |
| Mean | | | | |
| COMAT | 101.7 (7.1) | 92.7 (4.8) | 703.5b | 0.04 |
| COMAT-internal medicine | 99.7 (9.7) | 91.0 (5.4) | 1052.5b | 0.02 |
| COMAT-family medicine | 101.4 (9.3) | 92.8 (9.0) | 1215.5b | 0.02 |
| COMAT-pediatrics | 102.8 (9.6) | 91.1 (9.5) | 950.5b | 0.03 |
| COMAT-OBGYN | 99.7 (9.8) | 87.6 (8.6) | 816.0b | 0.03 |
| COMAT-surgery | 101.5 (9.7) | 92.5 (11.2) | 1175.5b | 0.02 |
| COMAT-OMM | 103.0 (8.4) | 94.9 (8.1) | 1046.5b | 0.02 |
| COMAT-psychiatry | 104.1 (9.4) | 99.8 (6.9) | 1700.5b | 0.01 |
| COMSAE Phase 2 | 517.8 (98.3) | 393.1 (52.0) | 602.0b | 0.04 |
| SP-Based Performance | | | | |
| Six subject OSCEs average | 80.8 (3.4) | 78.4 (2.6) | 1365.0b | 0.01 |
| COMLEX-USA Level 2-PE Humanistic | | |
| Comprehensive clinical skills exam | Pass (N=445)
Mean (SD) | Fail (N=6)
Mean (SD) | Mann-Whitney U | Effect Size (η2) |
| ICS | 87.6 (6.0) | 82.0 (2.8) | 510.5a | 0.02 |
| COMLEX-USA Level 2-PE Biomedical | | |
| Pass (N=446)
Mean (SD) | Fail (N=5)
Mean (SD) | Mann-Whitney U | |
| Data gathering (history + PE) | 73.6 (5.5) | 67.4 (5.1) | 435.0a | 0.02 |
| OPP | 70.7 (6.6) | 70.0 (2.8) | 955 | 0 |
| SOAP | 80.0 (5.5) | 74.6 (1.1) | 376.5a | 0.01 |
×
An analysis of OSCE performance between groups demonstrated that the passing group had a statistically significant higher average discipline-specific OSCE score (mean, 80.8 vs. 78.4; Mann-Whitney U=1365.0; η2=0.01). Similarly, in a comparison of the 4 individual clinical skills scores for RowanSOM's CSCE to the Level 2-PE humanistic and biomedical subdomain performance, the group that passed the humanistic domain had statistically significant higher scores in ICS on their CSCE (mean, 87.6 vs 82.0; Mann-Whitney U=510.5; η2=0.02) and those who passed the biomedical domain had significantly higher data gathering (mean, 73.6 vs. 67.4; Mann-Whitney U=435.0; η2=0.02) and SOAP note documentation scores (mean, 80.0 vs. 74.6; Mann-Whitney U=376.5; η2=0.01). The sizes of these differences in OSCEs performance (η2) were all small.
Last, in a comparison of workplace-based assessment performance between groups (n=174, because the school started to implement this assessment in academic year 2017-2018), students who passed the Level 2-PE did not have significantly higher average overall scores or higher performance in the individual competency scores for history taking, physical exam, documentation, or ICS (
Table 2).
Table 2.
Associations of Subjective Clinical Skills Performance and COMLEX-USA Level 2-PE Performancea
| COMLEX-USA Level 2-PE Overall | | |
| Pass (n=172)
Mean (SD) | Fail (n=4)
Mean (SD) | Mann-Whitney U | Effect Size (η2) |
| Workplace-Based Competency Evaluationa |
| Mean clinical evaluation score | 89.3 (2.5) | 87.4 (2.0) | 174 | 0.02 |
| COMLEX-USA Level 2-PE Humanistic | | |
| Individual competency score | Pass (N=174)
Mean (SD) | Fail (N=2)
Mean (SD) | | |
| ICS | 89.7 (2.7) | 88.4 (0.3) | 116.0 | 0 |
| COMLEX-USA Level 2-PE Biomedical | | |
| Pass (N=174)
Mean (SD) | Fail (N=2)
Mean (SD) | | |
| History taking | 88.6 (2.6) | 89.5 (0.6) | 235.0 | 0 |
| Physical exam | 87.9 (2.5) | 88.6 (1.9) | 216.0 | 0 |
| SOAP documentation | 88.6 (2.7) | 89.3 (2.6) | 203.0 | 0 |
Table 2.
Associations of Subjective Clinical Skills Performance and COMLEX-USA Level 2-PE Performancea
| COMLEX-USA Level 2-PE Overall | | |
| Pass (n=172)
Mean (SD) | Fail (n=4)
Mean (SD) | Mann-Whitney U | Effect Size (η2) |
| Workplace-Based Competency Evaluationa |
| Mean clinical evaluation score | 89.3 (2.5) | 87.4 (2.0) | 174 | 0.02 |
| COMLEX-USA Level 2-PE Humanistic | | |
| Individual competency score | Pass (N=174)
Mean (SD) | Fail (N=2)
Mean (SD) | | |
| ICS | 89.7 (2.7) | 88.4 (0.3) | 116.0 | 0 |
| COMLEX-USA Level 2-PE Biomedical | | |
| Pass (N=174)
Mean (SD) | Fail (N=2)
Mean (SD) | | |
| History taking | 88.6 (2.6) | 89.5 (0.6) | 235.0 | 0 |
| Physical exam | 87.9 (2.5) | 88.6 (1.9) | 216.0 | 0 |
| SOAP documentation | 88.6 (2.7) | 89.3 (2.6) | 203.0 | 0 |
×
The current study examined the relationships between different measures of school-based clinical skills performance at RowanSOM with subsequent COMLEX-USA Level 2-PE performance. Our results add to the empirical evidence about strategies to support student success on the COMLEX-USA Level 2-PE and provide further insight into the relationships between different clinical skills and competency assessments and student exam performance.
The study findings of small but significant associations between the average OSCE score and Level 2-PE performance was consistent with previous studies and provides further evidence of the value of school-based standardized patient exams, particularly high-stakes OSCEs, as a strategy to support student success on the Level 2-PE.
11 Our results also showed significant relationships between the 3 clinical skills assessed on RowanSOM's comprehensive OSCE (ICS, DG, and SOAP note documentation) and Level 2-PE subcomponent (humanistic and biomedical domains) performance. Compared with previous studies that demonstrated a limited relationship between school-based clinical skills assessments and the Level 2-PE, likely due to less reliable SP-based exams,
5 our results extended the empirical evidence of the correlation between specific clinical skills in SP-based exams and subcomponents of Level 2-PE and provided support for our school policy, which requires all students to distinctly pass each of the 4 clinical skills assessments on the comprehensive OSCE before taking the Level 2-PE. The OPP score was the only component score that did not show a significant difference between students who passed and those who failed the Level 2-PE, possibly due to the limited number of cases and checklist items that assessed OPP.
The results of our study also provide support for expanding clinical skills assessment for both formative and summative evaluations earlier in the curriculum. A new CSCE benchmark has been added for all students at the end of their preclerkship training to establish readiness for clerkships. Meanwhile, efforts are ongoing to improve the rigor and reliability of current discipline-specific OSCEs during clerkships and further enhance testing before entering residency. These strategies are designed to assess student progress in meeting competency milestones across all 4 years of the curriculum, as well as to prepare students to pass their board exams and to ensure their readiness for residency.
The association between COMAT scores (both average COMAT scores and individual subject COMATs) and Level 2-PE performance also suggests that the COMAT exams are useful tools to help identify and remediate student weakness in clinical skills before they take the Level 2-PE. Average COMAT score results showed the largest difference between students who passed and failed Level 2-PE, suggesting that a comprehensive view and use of COMAT exams may contribute additional insight into student performance as they prepare for COMLEX-USA. Individual subject COMATs likewise support student COMLEX-USA performance. Pediatrics and Obstetrics and Gynecology COMATs seemed to have the most associations with Level 2-PE, followed by Family Medicine, Internal Medicine, Surgery, and OPP. This pattern of association is similar to results from other studies on COMATs and COMLEX-USA Level 2-CE.
15 Currently, COMAT performance is being used to counsel students in their preparation for the Level 2-CE. Results of this study suggest that COMAT performance may be also useful in guiding student preparation for the Level 2-PE. Similarly, our data suggest that the COMSAE Phase 2 is associated with Level 2-PE performance and may serve as an additional tool in determining student readiness for the Level 2-PE as well.
Our results did not show a significant association between our workplace-based clinical evaluation of OCC and the Level 2-PE. Many factors may help to explain this outcome. First, there is often a lack of differentiation in preceptor ratings across students, limiting the ability to establish any correlation.
15 Others have suggested that SP-based performance is not necessarily transferrable to the workplace due to the context-specific nature of both types of assessments,
19,24 which indicates they each assesses different types of clinical skills and both should be implemented into clerkship training in undergraduate medical training. However, previous research on COMLEX-USA Level 2-CE scores did find a moderate correlation with clinical performance evaluation, albeit in a much larger sample size and between continuous performance measures.
15 Our analysis of the relationship between Level 2-PE outcomes and workplace-based assessment was also limited by sample size, which was too small to have the statistical power to determine a relationship between individual clinical skills items on the instrument and overall pass/fail performance on the Level 2-PE.
The generalizability of any study at a single institution is limited. However, the results of this study do provide further evidence in support of previous studies and specifically offers insight for osteopathic medical schools and COMLEX-USA Level 2-PE outcomes. School-based OSCEs, especially high-stakes OSCEs aligned with the Level 2-PE blueprint, clearly support student success. Other clinically related assessment tools, including COMAT exams and the COMSAE Phase 2 exam, can also provide important information to guide student preparation and to support decisions about a student's readiness for the exam. As additional Level 2-PE performance outcomes are collected and an adequate sample is accumulated for both study groups, the initial outcomes from this study will be used to help establish a predictive model of student success and ultimately be used to inform the effectiveness of school-based assessment strategies. Additionally, as we continue to collect data on the workplace-based instrument and expand our sample size, we will be able to further explore this relationship and advance our effective use of preceptor ratings to inform the development of clinical skills and to train better physicians.