Free
Medical Education  |   April 2016
Preparation Strategies of Osteopathic Medical Students for the COMLEX-USA Level 2-PE
Author Notes
  • From the National Board of Osteopathic Medical Examiners in Conshohocken, Pennsylvania. 
  •  *Address correspondence to Jeanne M. Sandella, DO, Clinical Skills Testing, National Board of Osteopathic Medical Examiners, 101 W Elm St, Suite 150, Conshohocken, PA 19428-2004. E-mail: jsandella@nbome.org
     
Article Information
Medical Education / COMLEX-USA
Medical Education   |   April 2016
Preparation Strategies of Osteopathic Medical Students for the COMLEX-USA Level 2-PE
The Journal of the American Osteopathic Association, April 2016, Vol. 116, 234-242. doi:10.7556/jaoa.2016.048
The Journal of the American Osteopathic Association, April 2016, Vol. 116, 234-242. doi:10.7556/jaoa.2016.048
Web of Science® Times Cited: 1
Abstract

Context: Since 2002, osteopathic medical schools have made curricular changes to further enhance the clinical skills of their students, to prepare them for residency training, and to pass the Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation (COMLEX-USA Level 2-PE).

Objective: To report how students at osteopathic medical schools prepare for the COMLEX-USA Level 2-PE, and to investigate the effect of these techniques on examination performance.

Methods: A standardized survey was given to students before the beginning of their examination to assess the preparation of osteopathic medical students for the COMLEX-USA Level 2-PE, such as coursework, orientation materials, and standardized patient (SP) encounters. Surveys that were completed by first-time test takers during the 2013-2014 and 2014-2015 test cycles were included in this study.

Results: Of 9120 surveys administered, 8733 were completed, achieving a response rate of 95.8%. Of those 8733 respondents, 8706 students (99.7%) reported having SP encounters during the first and second year of medical school, and 7379 (84%) reported having at least 1 SP encounter in years 3 and 4. Of 8733 students, 6079 (70%) reported receiving feedback from an osteopathic physician on their SP encounters, and 6049 (69%) and 6253 (72%) reported having viewed the COMLEX-USA Level 2-PE orientation video online and having read the examination’s orientation guide, respectively. The largest difference in preparation between students who passed the COMLEX-USA Level 2-PE and students who did not was a prerequisite SP examination at their school, with 5574 students (68.9%) who passed reporting having participated compared with 364 students (56.5%) who failed. None of the differences in clinical skills training and test preparation was associated with statistically significant differences in pass or fail status.

Conclusion: Osteopathic medical students use a variety of methods to enhance their clinical skills in preparation for the COMLEX-USA Level 2-PE, with universal use of SP programs since the COMLEX-USA Level 2-PE was implemented in 2004. Educators should continue to foster ways to develop students’ clinical skills that reflect new advances in education and assessment to ensure that future osteopathic physicians can demonstrate competency in fundamental clinical skills before beginning postgraduate training.

This Medical Education theme issue introduces a new collaboration between the JAOA and the American Association of Colleges of Osteopathic Medicine (AACOM) to recruit, peer review, edit, and distribute articles through the JAOA on osteopathic medical education research and other scholarly issues related to medical education.

 
The National Board of Osteopathic Medical Examiners (NBOME) has been administering the Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation (COMLEX-USA Level 2-PE) for more than 10 years as part of the COMLEX-USA series of licensure examinations. This examination serves to augment the multiple choice examinations to fulfill the public mandate for clinical skills assessments of osteo-pathic physicians. Since 2004, national licensure in the United States has included clinical skills assessment for all osteopathic and allopathic physicians.1,2 
Like allopathic medical schools, osteopathic medical schools are experiencing evolving curricular changes, such as standardized patient (SP) programs, problem-based learning approaches, flipped classroom approaches, small group discussions, and other forms of unique educational experiences3-6 to help educate students in an unconventional way for the COMLEX-USA series and other newer types of assessments. Changes in the assessments may play a role in how well a new educational model is received, particularly with performance assessments and even newer multiple choice-based formats focusing on decision making and critical thinking rather than basic fact recall.7,8 Students want to be confident that they are going to perform well enough on the assessments to be good osteopathic physicians, and they are changing the way they learn to better prepare themselves for these assessments. For example, they are involved in the teaching process as well as in the learning process.9 Some students also supplement their own learning beyond their medical school curriculum and may think traditional education methods are not as effective for them.10 
Some educational researchers have attempted to define how students should prepare for performance assessments. Students in New Zealand who were asked about their preparations used “practical preparation” (eg, preparatory objective structured clinical evaluations [OSCEs], practicing on each other and in groups).11 The authors identified relevant topics likely to be covered in the examination through “theoretical preparation” (eg, review of the faculty-developed OSCE handbook, past OSCE papers, and a booklet developed by students for test preparation).11 Researchers in the United Kingdom reported that students responded positively to clinical skills teaching delivered by near-peer tutors to prepare for their final OSCE.12 The New Zealand study11 showed how a sum-mative OSCE affected the learning behaviors of medical students by encouraging them to practice clinical skills on each other, to rehearse routines for how they perform clinical examination, and to work in groups. Although students did not list clinical practice or clinical exposure as a way to prepare for an OSCE, they did mention clinical experience as advice to students for preparing for an OCSE.11 A study done at Michigan State University in East Lansing13 concluded that previous academic performance rather than preparatory study time was a better predictor of OSCE outcomes. The best and worst performers on the 8-station formative OSCE reported the least amount of time preparing for it. 13 
In a 2002 study,14 a survey was conducted in preparation for the launch of the COMLEX-USA Level 2-PE about the use of SPs at osteopathic medical schools. At the time of the study, all 19 osteopathic medical schools responded to the survey. Twelve of the 19 schools reported having active SP programs, and 5 (42%) reported extending the SP programs into years 3 and 4. In a follow-up survey in 2005, 19 of 23 COMs (87%) reported having SP programs.15 
The purpose of the current study was to report how students at osteopathic medical schools prepare for the COMLEX-USA Level 2-PE and to investigate the effect of preparation techniques, if any, on examination performance. 
Methods
This survey-based investigation was conducted using a nonexperimental research design with a convenience sample of osteopathic medical students who completed the COMLEX-USA Level 2-PE between July 7, 2013, and February 13, 2015 (ie, the 2013-2014 and 2014-2015 test cycles). Participation in the survey was voluntary, and students were informed that their participation would not affect their examination score in any way. Students’ names and identification numbers were collected so that the data could be linked to pass or fail status, and data were reported in aggregate. Students signed an agreement acknowledging the use of their survey and examination data for research purposes before completing the survey. The NBOME used a standardized pretest survey to assess the preparation of osteopathic medical students for the COMLEX-USA Level 2-PE. A modified pretest survey was offered to students to complete on-site before beginning the COMLEX-USA Level 2-PE. The psychometrics team and physician staff at the NBOME with advisement from the Clinical Skills Testing Advisory Committee, which consists of osteopathic physicians and experts in the field of performance assessment, created the survey. 
The 12-question survey was administered to each student either electronically via a personal digital assistant that recorded the answers in the database, or on paper, with answers manually entered by staff into the database. Of the 12 survey questions, 2 were excluded from analysis because they were deemed irrelevant to this study. 
Data analysis involved tabulating descriptive statistics for frequency and comparing responses between students who passed or failed the examination. Data from first-time test-takers and from surveys that were completed were included in the tabulations. A χ2 test was used to investigate survey responses with COMLEX-USA Level 2-PE pass rates. A P value of .01 was used to identify statistical significance, and SAS statistical software version 9.4 (SAS Institute, Inc) was used for all analyses. 
Results
The survey was sent to 9120 eligible students at 30 osteopathic medical schools, and 8733 students (95.8%) completed the surveys. Of the 8733 students, 8089 (92.6%) passed the examination (group 1), and 644 (7.4%) failed (group 2). A total of 4716 students (54%) were men and 4017 (46%) were women, with a mean age of 28 years (range, 22-64 years). Students indicated their race or ethnicity as follows: 5614 (68%), white; 1381 (17%), Asian; 196 (2%), black or African American; 125 (1.5%), other; and 901 (11%) chose not to answer. In addition, 373 students (4.5%) reported that their ethnicity was of Hispanic, Latino, or Spanish origin. 
Table 1 shows the reported frequency of SP encounters during medical school, as well as scoring of and feedback for those encounters. Both groups engaged in a comparable amount of SP encounters in year 1, though group 2 tended toward fewer encounters overall. In years 3 and 4, 134 students in group 2 (20.8%) had no SP encounters compared with the 1220 (15.1%) in group 1. In group 2, 98 students (15.2%) reported having more than 20 SP encounters in years 3 and 4, compared with 856 (10.6%) in group 1. Table 1 shows that 571 students from group 2 (88.7%) also reported fewer scored encounters than those in group 1 (7564 [93.5%]) and similar levels of written feedback in years 1 and 2, whereas in years 3 and 4, both scores and written feedback were lower for group 2. Table 2 shows that little difference was reported in the amount or origin of face-to-face feedback after these encounters. 
Table 1.
Standardized Patient Encounters for Students Preparing to Take the COMLEX-USA Level 2-PEa
Survey Item Years 1 and 2, No. (%) Years 3 and 4, No. (%)
Group 1 (n=8089)b Group 2 (n=644)c Group 1 (n=8089)b Group 2 (n=644)c
  How many standardized patient encounters did you have at your medical school in years 3 and 4?
   0 23 (0.3) 4 (0.6) 1220 (15.1) 134 (20.8)
   1-5 479 (5.9) 63 (9.8) 3031 (37.5) 217 (33.7)
   6-10 1661 (20.5) 145 (22.5) 1680 (20.8) 107 (16.6)
   11-20 2999 (37.1) 232 (36) 1302 (16.1) 88 (13.7)
   >20 2927 (36.2) 200 (31.1) 856 (10.6) 98 (15.2)
  Did you receive scores for these encounters?
   Yes 7564 (93.5) 571 (88.7) 5304 (65.6) 371 (57.6)
   No 525 (6.5) 73 (11.3) 2785 (34.4) 273 (42.4)
  Did you receive written feedback beyond a grade or score?
   Yes 7136 (88.2) 572 (88.8) 5425 (67.1) 402 (62.4)
   No 532 (10.8) 72 (11.2) 2664 (32.9) 242 (37.6)

a The standardized patient encounters occurred during cycles 10 and 11, which were from July 7, 2013, to February 13, 2015.

b Group 1 consisted of students who passed.

c Group 2 consisted of students who failed.

Abbreviation: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation.

Table 1.
Standardized Patient Encounters for Students Preparing to Take the COMLEX-USA Level 2-PEa
Survey Item Years 1 and 2, No. (%) Years 3 and 4, No. (%)
Group 1 (n=8089)b Group 2 (n=644)c Group 1 (n=8089)b Group 2 (n=644)c
  How many standardized patient encounters did you have at your medical school in years 3 and 4?
   0 23 (0.3) 4 (0.6) 1220 (15.1) 134 (20.8)
   1-5 479 (5.9) 63 (9.8) 3031 (37.5) 217 (33.7)
   6-10 1661 (20.5) 145 (22.5) 1680 (20.8) 107 (16.6)
   11-20 2999 (37.1) 232 (36) 1302 (16.1) 88 (13.7)
   >20 2927 (36.2) 200 (31.1) 856 (10.6) 98 (15.2)
  Did you receive scores for these encounters?
   Yes 7564 (93.5) 571 (88.7) 5304 (65.6) 371 (57.6)
   No 525 (6.5) 73 (11.3) 2785 (34.4) 273 (42.4)
  Did you receive written feedback beyond a grade or score?
   Yes 7136 (88.2) 572 (88.8) 5425 (67.1) 402 (62.4)
   No 532 (10.8) 72 (11.2) 2664 (32.9) 242 (37.6)

a The standardized patient encounters occurred during cycles 10 and 11, which were from July 7, 2013, to February 13, 2015.

b Group 1 consisted of students who passed.

c Group 2 consisted of students who failed.

Abbreviation: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation.

×
Students were asked to report on how their osteo-pathic medical school prepared them and how they prepared themselves (Table 2). Of the 8089 students in group 1, 5956 (73.6%) studied textbooks on physical diagnosis compared with 453 (70.3%) in group 2. Group 1 engaged in more coursework in physical diagnosis (4983 [61.6%]) than group 2 (363 [56.4%]). In group 1, 3341 students (41.3%) reported coursework in humanistic domain skills (ie, patient-physician communication, interpersonal skills, and professionalism) vs 235 (36.5%) in group 2. The largest difference in preparation between the 2 groups was a prerequisite SP examination delivered at their school, with 5574 in group 1 (68.9%) reporting having participated compared with 364 in group 2 (56.5%). Both groups reported a large percentage of SP encounters at clinical rotations, with 6270 (77.5%) in group 1 and 461 (71.6%) in group 2. 
Table 2.
Summary of Survey Questions Assessing Students’ Preparation for the COMLEX-USA Level 2-PE
Survey Item No. (%)
Group 1 (n=8089)a Group 2 (n=644)b
  Did you receive face-to-face feedback during or after your school standardized patient encounters from any of the following?c
   Osteopathic physician faculty 5619 (69.5) 460 (71.4)
   PhD/MA/MS or communications expert 856 (10.6) 75 (11.6)
   Standardized patients 4846 (59.9) 355 (55.1)
   Peer-to-peer 2592 (32) 211 (32.8)
   No feedback 839 (10.4) 72 (11.2)
   I have not had any standardized patient encounters 68 (0.8) 6 (0.9)
  How did your osteopathic medical school prepare you to take the COMLEX-USA Level 2-PE?c
   Textbooks on physical diagnosis 5956 (73.6) 453 (70.3)
   Coursework in physical diagnosis 4983 (61.6) 363 (56.4)
   Coursework in humanistic qualities of a physician 3341 (41.3) 235 (36.5)
   High-stakes standardized patient performance examination at school (prerequisite) 5574 (68.9) 364 (56.5)
   COMLEX-USA Level 2-PE preparation course (administered by school) 1597 (19.7) 148 (23)
   Practice at rotations 6270 (77.5) 461 (71.6)
   Other 184 (2.3) 15 (2.3)
  How did you prepare yourself to take the COMLEX-USA Level 2-PE?c
   NBOME instructional video on website 5634 (69.7) 415 (64.4)
   COMLEX-USA Level 2-PE orientation guide on website 5899 (72.9) 354 (55)
   COMLEX-USA Level 2-PE specific study guide 4773 (59) 327 (50.8)
   USMLE specific study guide 2389 (29.5) 168 (26.1)
   Generic commercial performance evaluation preparation course 297 (3.7) 26 (4)
   COMLEX-USA Level 2-PE specific commercial preparation course 491 (6.1) 33 (5.1)
   Study or role-play with friends 3048 (37.6) 207 (32.1)
   Other 224 (2.8) 16 (2.5)

a Group 1 consisted of students who passed.

b Group 2 consisted of students who faild.

c Respondents were asked to check all that applied.

Abbreviations: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation; NBOME, National Board of Osteopathic Medical Examiners; USMLE, United States Medical Licensing Examination.

Table 2.
Summary of Survey Questions Assessing Students’ Preparation for the COMLEX-USA Level 2-PE
Survey Item No. (%)
Group 1 (n=8089)a Group 2 (n=644)b
  Did you receive face-to-face feedback during or after your school standardized patient encounters from any of the following?c
   Osteopathic physician faculty 5619 (69.5) 460 (71.4)
   PhD/MA/MS or communications expert 856 (10.6) 75 (11.6)
   Standardized patients 4846 (59.9) 355 (55.1)
   Peer-to-peer 2592 (32) 211 (32.8)
   No feedback 839 (10.4) 72 (11.2)
   I have not had any standardized patient encounters 68 (0.8) 6 (0.9)
  How did your osteopathic medical school prepare you to take the COMLEX-USA Level 2-PE?c
   Textbooks on physical diagnosis 5956 (73.6) 453 (70.3)
   Coursework in physical diagnosis 4983 (61.6) 363 (56.4)
   Coursework in humanistic qualities of a physician 3341 (41.3) 235 (36.5)
   High-stakes standardized patient performance examination at school (prerequisite) 5574 (68.9) 364 (56.5)
   COMLEX-USA Level 2-PE preparation course (administered by school) 1597 (19.7) 148 (23)
   Practice at rotations 6270 (77.5) 461 (71.6)
   Other 184 (2.3) 15 (2.3)
  How did you prepare yourself to take the COMLEX-USA Level 2-PE?c
   NBOME instructional video on website 5634 (69.7) 415 (64.4)
   COMLEX-USA Level 2-PE orientation guide on website 5899 (72.9) 354 (55)
   COMLEX-USA Level 2-PE specific study guide 4773 (59) 327 (50.8)
   USMLE specific study guide 2389 (29.5) 168 (26.1)
   Generic commercial performance evaluation preparation course 297 (3.7) 26 (4)
   COMLEX-USA Level 2-PE specific commercial preparation course 491 (6.1) 33 (5.1)
   Study or role-play with friends 3048 (37.6) 207 (32.1)
   Other 224 (2.8) 16 (2.5)

a Group 1 consisted of students who passed.

b Group 2 consisted of students who faild.

c Respondents were asked to check all that applied.

Abbreviations: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation; NBOME, National Board of Osteopathic Medical Examiners; USMLE, United States Medical Licensing Examination.

×
In self-preparation, the largest differences can be seen in using the COMLEX-USA Level 2-PE instructional video (5634 [69.7%] in group 1 vs 415 [64.4%] in group 2), reading the orientation guide on the website (5899 [72.9%] in group 1 vs 354 [55%] in group 2), using a COMLEX-USA Level 2-PE study guide (4773 [59%] in group 1 vs 327 [50.8%] in group 2), and studying or role-playing with friends (3058 [37.6%] in group 1 vs 207 [32.1%] in group 2). 
The last question asked how well prepared students believed they were for the biomedical/biomechanical domain and the humanistic domain. The former includes skills in data-gathering (ie, history taking, physical examination, osteopathic structural examination, osteopathic manipulative treatment, and clinical decision making and documentation in the form of a postencounter note [subjective, objective, assessment, and plan format]). Of 8733 students, 8680 (99.3%) indicated that they were either somewhat prepared or very prepared in the biomedical/biomechanical domain, and 8685 (99.9%) were either somewhat prepared or very prepared in the humanistic domain. In group 1, 54 (0.7%) indicated that they were not well prepared in the biomedical/biomechanical domain and 40 (0.5%) in the humanistic domain, compared with 9 (1.4%) and 8 (1.2%) in group 2, respectively. Overall, students were more likely to be very prepared for the humanistic domain than for the biomedical/biomechanical domain. 
A χ2 analysis was performed to examine the relationship between student preparation as identified by survey questions on test preparation and COMLEX-USA Level 2-PE pass rates. A second χ2 analysis was used to examine the relationship between the presence of SP encounters in years 3 and 4 and COMLEX-USA Level 2-PE pass rates (Table 3). Many SP relationship preparation items were statistically significant (P<.01), which can be attributed to the large sample size. With a 92.6% pass rate among surveyed students, the amount of variance in pass rate that can be attributed to each preparation item is limited, and all items had cp coefficients of less than 0.1, the largest being 0.069 for previous experience with a high-stakes SP-based examination administered by a school. 
Table 3.
Statistical Relationship Between the Presence of Standardized Patient Encounters and COMLEX-USA Level 2-PE Pass Rates
Survey Item χ12 φ
  How many standardized patient encounters did you have at your medical school in years 3 and 4?
   0 vs 1 or more 14.93 0.041
  School Preparation
   Textbooks on physical diagnosis 3.30 0.020
   Coursework in physical diagnosis 6.89a 0.028
   Coursework in humanistic qualities of a physician 5.71b −0.026
   High-stakes standardized patient performance examination at school (prerequisite) 42.06a 0.069
   COMLEX-USA Level 2-PE preparation course (administered by school) 3.91b 0.021
   Practice at rotations 11.87a 0.037
   Other 0.01 0.001
  Self-preparation
   NBOME instructional video on website 7.60a 0.030
   COMLEX-USA Level 2-PE orientation guide on website 28.13a 0.057
   COMLEX-USA Level 2-PE specific study guide 27.83a 0.056
   USMLE specific study guide 1.72 −0.014
   Generic commercial performance evaluation preparation course 2.01 −0.015
   COMLEX-USA Level 2-PE specific commercial preparation course 0.22 0.005
   Study or role-play with friends 7.82a −0.030
   Other 10.71a −0.035

a P<.01.

b P<.05.

Abbreviations: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation; NBOME, National Board of Osteopathic Medical Examiners; USMLE, United States Medical Licensing Examination

Table 3.
Statistical Relationship Between the Presence of Standardized Patient Encounters and COMLEX-USA Level 2-PE Pass Rates
Survey Item χ12 φ
  How many standardized patient encounters did you have at your medical school in years 3 and 4?
   0 vs 1 or more 14.93 0.041
  School Preparation
   Textbooks on physical diagnosis 3.30 0.020
   Coursework in physical diagnosis 6.89a 0.028
   Coursework in humanistic qualities of a physician 5.71b −0.026
   High-stakes standardized patient performance examination at school (prerequisite) 42.06a 0.069
   COMLEX-USA Level 2-PE preparation course (administered by school) 3.91b 0.021
   Practice at rotations 11.87a 0.037
   Other 0.01 0.001
  Self-preparation
   NBOME instructional video on website 7.60a 0.030
   COMLEX-USA Level 2-PE orientation guide on website 28.13a 0.057
   COMLEX-USA Level 2-PE specific study guide 27.83a 0.056
   USMLE specific study guide 1.72 −0.014
   Generic commercial performance evaluation preparation course 2.01 −0.015
   COMLEX-USA Level 2-PE specific commercial preparation course 0.22 0.005
   Study or role-play with friends 7.82a −0.030
   Other 10.71a −0.035

a P<.01.

b P<.05.

Abbreviations: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation; NBOME, National Board of Osteopathic Medical Examiners; USMLE, United States Medical Licensing Examination

×
Discussion
Osteopathic medical school curricula are designed to prepare students for graduate medical education and clinical practice. American Osteopathic Association Commission on Osteopathic College Accreditation requires successful performance on COMLEX-USA Level 1, COMLEX-USA Level 2-Cognitive Evaluation, and COMLEX-USA Level 2-PE for graduation from an os-teopathic medical school. The teaching and assessment of clinical skills have become a priority since the implementation of the COMLEX-USA Level 2-PE in 2004, adding evidence for the consequential validity of performance examinations with respect to medical education. Students have to be prepared to not only demonstrate their cognitive knowledge and clinical reasoning skills, but also how they would apply these skills in the context of clinical encounters with patients. 
Over the past 10 years, these requirements have resulted in the increased focus of osteopathic medical schools on the education of clinical skills, including history taking, physical examination and diagnosis, critical thinking, and osteopathic manipulative treatment. These skills are identified as competencies that are extremely important in the education of an osteopathic physician but were previously not well assessed (or not assessed at all) at the licensure level.15,16 Because of these requirements, all osteopathic medical schools now include SP encounters in the first and second years, and the majority extend these encounters into the clinical years 3 and 4. This inclusion has further increased the number of SP encounters from that reported in previous studies.14,16 
Fewer than half of respondents in both groups (3341 students [41.3%] in group 1 and 235 students [36.5%] in group 2) reported having specific coursework in humanistic qualities of a physician, which is perplexing because in the past, osteopathic medical school graduates have noted the importance of patient-physician communication, interpersonal skills, professionalism, and empathy. 17 It is possible that formal instruction in this area is lacking or that the present study’s students misinterpreted the question. 
In the current study, students had many preparation methods specifically for the COMLEX-USA Level 2-PE, such as purchasing study guides and attending preparation courses. Anecdotally, although all students who take the COMLEX-USA Level 2-PE may have been aware that they had to participate in a 50-minute on-site orientation that included part of the online orientation video before the examination, they invested considerably in external methods of preparation when these and free resources were not universally used. We found little difference in how prepared students in group 1 and group 2 reported they were for the examination. Adult learners including physicians are generally inaccurate at self-assessment.18,19 
We tested these results on the relationship between SP encounters and pass rates for statistical significance (Table 3). However, the interpretation of null hypothesis significance tests can be complicated in large samples, simply because with a large enough sample size, even effects that are trivial from a practical standpoint can be statistically significant. For example, participation in high-stakes SP encounters (χ12 = 42.06; P<.01) and review of the COMLEX-USA Level 2-PE orientation guide (χ12=28.13; P<.01) showed correlations with pass status (φ=0.069 and φ=0.057, respectively). Although the χ2 test showed statistical significance, the correlations show that a small amount of the variance in pass status was accounted for by either preparation technique. 
In these data comparing SP encounters with pass rates (Table 3), the interpretation is complicated by the fact that failing the examination is a low base rate behavior, which can be difficult to predict. Correlations tended to show a small amount of pass or fail variability being accounted for by the other variables because there was not much variability in pass status. Because of these issues, the results presented are better viewed overall as suggestive trends than as definitive answers to the question of how students who pass prepare differently from students who fail. 
The findings of the current study indicate that students were less likely to receive feedback for an SP encounter score in the third and fourth year. Some schools require that students take a prerequisite clinical skills examination before taking the COMLEX-USA Level 2-PE. Having a prerequisite examination was reported more commonly by students who went on to pass the COMLEX-USA Level 2-PE, but this was not a statistically significant correlation. Whether these types of prerequisite examinations increase student confidence when taking the licensure examination should be studied. Overall, 6079 osteopathic physicians (69.6%) and 5201 SPs (59.6%) gave feedback to students (Table 2). We did not investigate whether there was any relationship between the kind of feedback given to the students and their performance, but it is a topic that should be explored. 
Although group 1 reported that they were more likely to see SPs in their third and fourth years, this trend did not continue as the number of SP encounters increased; 98 students in group 2 (15.5%) reported having more than 20 SP encounters in years 3 and 4. A possible reason is that schools may have already identified these students as needing more remediation and therefore provided more experiences for them. Regardless, this trend implies that the quality of the experience is more helpful than the number of experiences. 
The present study is limited by the fact that the preparation strategies were self-reported; students may have over- or underrepresented their curricula or preparation activities knowing that this survey was administered by the NBOME, which may have accounted for the low number of students who reported specific coursework in humanistic domain skills of a physician. Also, students may not recall some of the experiences from their first and second year, which may affect the results. Finally, this survey did not include use of the online electronic COMLEX-USA Level 2-PE subjective, objective, assessment, and plan note tutorial, as it was not available for the first half of the study period. Future studies should look at how students and schools use this tool. 
Another limitation of the current study is that it focused on preparation strategies of first-time test-takers. Future studies should focus on the preparation strategies of students who are repeating the examination, with the goal of informing remediation strategies. 
Conclusion
Osteopathic medical students use a variety of educational methods to enhance their clinical skills in preparation for the COMLEX-USA Level 2-PE. New assessments will continue to be developed with the goal of improved validity of their benchmarks. Preparing students to demonstrate the competencies measured in those assessments continues to be refined. Despite differences in clinical skills training and test preparation, including a positive benefit of participating in a prerequisite SP-based clinical skills examination, none of these methods yielded statistical significance with regard to COMLEX-USA Level 2-PE pass or fail status. We must continue to foster ways to develop students’ clinical skills to ensure that not only are they current with advances in education and assessment but also that they can demonstrate competency in fundamental clinical skills before beginning their postgraduate training. 
Author Contributions
Drs Sandella and Smith and Mr Peters provided substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; Drs Sandella and Gimpel drafted the article or revised it critically for important intellectual content; Drs Sandella and Smith and Mr Peters gave final approval of the version of the article to be published; and all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. 
References
Step 2 CS. United States Medical Licensing Examination website. http://www.usmle.org/step-2-cs/. Accessed February 2, 2016.
Level 2 Performance Evaluation. National Board of Osteopathic Medical Examiners website. http://www.nbome.org/comlex-pe.asp?m=can. Accessed February 15, 2016.
Hoff G, Hirsch NJ, Means JJ, Streyffeler L. A call to include medical humanities in the curriculum of colleges of osteopathic medicine and in applicant selection. J Am Osteopath Assoc. 2014;114(10):798-804. doi:10.7556/jaoa.2014.154. [CrossRef] [PubMed]
Ferretti SM, Mesina JE, Gabel LL. Lake Erie College of Osteopathic Medicine's independent study pathway program: an alternative medical school curriculum design. J Am Osteopath Assoc. 2000;100(11):713-716. [PubMed]
Wilson MA, Blondefield PJ. Teaching "doctoring": a model curriculum for family medicine. J Am Osteopath Assoc. 2007;107(1):30-34. [PubMed]
McNabb JE, Kangas JC, Higbee D, Dawson B. Postdoctoral core competencies in the predoctoral curriculum. J Am Osteopath Assoc. 2006;106(9):527-528. [PubMed]
Mattick K, Crocker G, Bligh J. Medical student attendance at non-compulsory lectures [published online October 14, 2006]. Adv Health Sci Educ Theory Pract. 2007;12(2):201-210. [CrossRef] [PubMed]
Shen L, Li F, Wattleworth R, Filipetto F. The promise and challenge of including multimedia items in medical licensure examinations: some insights from an empirical trial. Acad Med. 2010;85(10 suppl):S56-S59. doi:10.1097/ACM.0b013e3181ed3c80. [CrossRef] [PubMed]
May W, Chung EK, Elliott D, Fisher D. The relationship between medical students' learning approaches and performance on a summative high-stakes clinical performance examination. Med Teach. 2012;34(4):e236-e241. doi:10.3109/0142159X.2012.652995. [CrossRef] [PubMed]
Holtzman KZ, Swanson DB, Ouyang W, Hussie K, Allbee K. Use of multimedia on the Step 1 and Step 2 Clinical Knowledge components of USMLE: a controlled trial of the impact on item characteristics. Acad Med. 2009;84(10 suppl):S90-S93. doi:10.1097/ACM.0b013e3181b37b0b. [CrossRef] [PubMed]
Rudland J, Wilkinson T, Smith-Han K, Thompson-Fawcett M. "You can do it late at night or in the morning. You can do it at home, I did it with my flatmate." The educational impact of an OSCE. Med Teach. 2008;30(2):206-211. doi:10.1080/01421590701851312. [CrossRef] [PubMed]
Rashid MS, Sobowale O, Gore D. A near-peer teaching program designed, developed and delivered exclusively by recent medical graduates for final year medical students sitting the final objective structured clinical examination (OSCE). BMC Med Educ. 2011;11:11. doi:10.1186/1472-6920-11-11. [CrossRef] [PubMed]
Mavis BE. Does studying for an objective structured clinical examination make a difference? Med Educ. 2000;34(10):808-812. [CrossRef] [PubMed]
Errichetti AM, Gimpel JR, Boulet JR. State of the art in standardized patient programs: a survey of osteopathic medical schools. J Am Osteopath Assoc. 2002;102(11):627-631. [PubMed]
Gimpel JR, Weidner AC, Boulet JR, Wilson C, Errichetti AM. Standardized patients and mechanical simulators in teaching and assessment at colleges of osteopathic medicine. J Am Osteopath Assoc. 2007;107(12):557-561. [PubMed]
Boulet JR, Smee SM, Dillon GF, Gimpel JR. The use of standardized patient assessments for certification and licensure decisions. Simul Healthc. 2010;4(1):35-42. doi:10.1097/SIH.0b013e318182fc6c. [CrossRef]
Gimpel JR, Boulet DO, Errichetti AM. Evaluating the clinical skills of osteopathic medical students. J Am Osteopath Assoc. 2003;103(6):267-279. [PubMed]
Calabrese LH, Bianco JA, Mann D, Massello D, Hojat M. Correlates and changes in empathy and attitudes toward interprofessional collaboration in osteopathic medical students. J Am Osteopath Assoc. 2013;113(12):898-907. doi:10.7556/jaoa.2013.068. [CrossRef] [PubMed]
Regehr G, Eva K. Self-assessment, self-direction, and the self-regulating professional. Clin Orthop Relat Res. 2006;449:34-38. [PubMed]
Table 1.
Standardized Patient Encounters for Students Preparing to Take the COMLEX-USA Level 2-PEa
Survey Item Years 1 and 2, No. (%) Years 3 and 4, No. (%)
Group 1 (n=8089)b Group 2 (n=644)c Group 1 (n=8089)b Group 2 (n=644)c
  How many standardized patient encounters did you have at your medical school in years 3 and 4?
   0 23 (0.3) 4 (0.6) 1220 (15.1) 134 (20.8)
   1-5 479 (5.9) 63 (9.8) 3031 (37.5) 217 (33.7)
   6-10 1661 (20.5) 145 (22.5) 1680 (20.8) 107 (16.6)
   11-20 2999 (37.1) 232 (36) 1302 (16.1) 88 (13.7)
   >20 2927 (36.2) 200 (31.1) 856 (10.6) 98 (15.2)
  Did you receive scores for these encounters?
   Yes 7564 (93.5) 571 (88.7) 5304 (65.6) 371 (57.6)
   No 525 (6.5) 73 (11.3) 2785 (34.4) 273 (42.4)
  Did you receive written feedback beyond a grade or score?
   Yes 7136 (88.2) 572 (88.8) 5425 (67.1) 402 (62.4)
   No 532 (10.8) 72 (11.2) 2664 (32.9) 242 (37.6)

a The standardized patient encounters occurred during cycles 10 and 11, which were from July 7, 2013, to February 13, 2015.

b Group 1 consisted of students who passed.

c Group 2 consisted of students who failed.

Abbreviation: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation.

Table 1.
Standardized Patient Encounters for Students Preparing to Take the COMLEX-USA Level 2-PEa
Survey Item Years 1 and 2, No. (%) Years 3 and 4, No. (%)
Group 1 (n=8089)b Group 2 (n=644)c Group 1 (n=8089)b Group 2 (n=644)c
  How many standardized patient encounters did you have at your medical school in years 3 and 4?
   0 23 (0.3) 4 (0.6) 1220 (15.1) 134 (20.8)
   1-5 479 (5.9) 63 (9.8) 3031 (37.5) 217 (33.7)
   6-10 1661 (20.5) 145 (22.5) 1680 (20.8) 107 (16.6)
   11-20 2999 (37.1) 232 (36) 1302 (16.1) 88 (13.7)
   >20 2927 (36.2) 200 (31.1) 856 (10.6) 98 (15.2)
  Did you receive scores for these encounters?
   Yes 7564 (93.5) 571 (88.7) 5304 (65.6) 371 (57.6)
   No 525 (6.5) 73 (11.3) 2785 (34.4) 273 (42.4)
  Did you receive written feedback beyond a grade or score?
   Yes 7136 (88.2) 572 (88.8) 5425 (67.1) 402 (62.4)
   No 532 (10.8) 72 (11.2) 2664 (32.9) 242 (37.6)

a The standardized patient encounters occurred during cycles 10 and 11, which were from July 7, 2013, to February 13, 2015.

b Group 1 consisted of students who passed.

c Group 2 consisted of students who failed.

Abbreviation: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation.

×
Table 2.
Summary of Survey Questions Assessing Students’ Preparation for the COMLEX-USA Level 2-PE
Survey Item No. (%)
Group 1 (n=8089)a Group 2 (n=644)b
  Did you receive face-to-face feedback during or after your school standardized patient encounters from any of the following?c
   Osteopathic physician faculty 5619 (69.5) 460 (71.4)
   PhD/MA/MS or communications expert 856 (10.6) 75 (11.6)
   Standardized patients 4846 (59.9) 355 (55.1)
   Peer-to-peer 2592 (32) 211 (32.8)
   No feedback 839 (10.4) 72 (11.2)
   I have not had any standardized patient encounters 68 (0.8) 6 (0.9)
  How did your osteopathic medical school prepare you to take the COMLEX-USA Level 2-PE?c
   Textbooks on physical diagnosis 5956 (73.6) 453 (70.3)
   Coursework in physical diagnosis 4983 (61.6) 363 (56.4)
   Coursework in humanistic qualities of a physician 3341 (41.3) 235 (36.5)
   High-stakes standardized patient performance examination at school (prerequisite) 5574 (68.9) 364 (56.5)
   COMLEX-USA Level 2-PE preparation course (administered by school) 1597 (19.7) 148 (23)
   Practice at rotations 6270 (77.5) 461 (71.6)
   Other 184 (2.3) 15 (2.3)
  How did you prepare yourself to take the COMLEX-USA Level 2-PE?c
   NBOME instructional video on website 5634 (69.7) 415 (64.4)
   COMLEX-USA Level 2-PE orientation guide on website 5899 (72.9) 354 (55)
   COMLEX-USA Level 2-PE specific study guide 4773 (59) 327 (50.8)
   USMLE specific study guide 2389 (29.5) 168 (26.1)
   Generic commercial performance evaluation preparation course 297 (3.7) 26 (4)
   COMLEX-USA Level 2-PE specific commercial preparation course 491 (6.1) 33 (5.1)
   Study or role-play with friends 3048 (37.6) 207 (32.1)
   Other 224 (2.8) 16 (2.5)

a Group 1 consisted of students who passed.

b Group 2 consisted of students who faild.

c Respondents were asked to check all that applied.

Abbreviations: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation; NBOME, National Board of Osteopathic Medical Examiners; USMLE, United States Medical Licensing Examination.

Table 2.
Summary of Survey Questions Assessing Students’ Preparation for the COMLEX-USA Level 2-PE
Survey Item No. (%)
Group 1 (n=8089)a Group 2 (n=644)b
  Did you receive face-to-face feedback during or after your school standardized patient encounters from any of the following?c
   Osteopathic physician faculty 5619 (69.5) 460 (71.4)
   PhD/MA/MS or communications expert 856 (10.6) 75 (11.6)
   Standardized patients 4846 (59.9) 355 (55.1)
   Peer-to-peer 2592 (32) 211 (32.8)
   No feedback 839 (10.4) 72 (11.2)
   I have not had any standardized patient encounters 68 (0.8) 6 (0.9)
  How did your osteopathic medical school prepare you to take the COMLEX-USA Level 2-PE?c
   Textbooks on physical diagnosis 5956 (73.6) 453 (70.3)
   Coursework in physical diagnosis 4983 (61.6) 363 (56.4)
   Coursework in humanistic qualities of a physician 3341 (41.3) 235 (36.5)
   High-stakes standardized patient performance examination at school (prerequisite) 5574 (68.9) 364 (56.5)
   COMLEX-USA Level 2-PE preparation course (administered by school) 1597 (19.7) 148 (23)
   Practice at rotations 6270 (77.5) 461 (71.6)
   Other 184 (2.3) 15 (2.3)
  How did you prepare yourself to take the COMLEX-USA Level 2-PE?c
   NBOME instructional video on website 5634 (69.7) 415 (64.4)
   COMLEX-USA Level 2-PE orientation guide on website 5899 (72.9) 354 (55)
   COMLEX-USA Level 2-PE specific study guide 4773 (59) 327 (50.8)
   USMLE specific study guide 2389 (29.5) 168 (26.1)
   Generic commercial performance evaluation preparation course 297 (3.7) 26 (4)
   COMLEX-USA Level 2-PE specific commercial preparation course 491 (6.1) 33 (5.1)
   Study or role-play with friends 3048 (37.6) 207 (32.1)
   Other 224 (2.8) 16 (2.5)

a Group 1 consisted of students who passed.

b Group 2 consisted of students who faild.

c Respondents were asked to check all that applied.

Abbreviations: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation; NBOME, National Board of Osteopathic Medical Examiners; USMLE, United States Medical Licensing Examination.

×
Table 3.
Statistical Relationship Between the Presence of Standardized Patient Encounters and COMLEX-USA Level 2-PE Pass Rates
Survey Item χ12 φ
  How many standardized patient encounters did you have at your medical school in years 3 and 4?
   0 vs 1 or more 14.93 0.041
  School Preparation
   Textbooks on physical diagnosis 3.30 0.020
   Coursework in physical diagnosis 6.89a 0.028
   Coursework in humanistic qualities of a physician 5.71b −0.026
   High-stakes standardized patient performance examination at school (prerequisite) 42.06a 0.069
   COMLEX-USA Level 2-PE preparation course (administered by school) 3.91b 0.021
   Practice at rotations 11.87a 0.037
   Other 0.01 0.001
  Self-preparation
   NBOME instructional video on website 7.60a 0.030
   COMLEX-USA Level 2-PE orientation guide on website 28.13a 0.057
   COMLEX-USA Level 2-PE specific study guide 27.83a 0.056
   USMLE specific study guide 1.72 −0.014
   Generic commercial performance evaluation preparation course 2.01 −0.015
   COMLEX-USA Level 2-PE specific commercial preparation course 0.22 0.005
   Study or role-play with friends 7.82a −0.030
   Other 10.71a −0.035

a P<.01.

b P<.05.

Abbreviations: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation; NBOME, National Board of Osteopathic Medical Examiners; USMLE, United States Medical Licensing Examination

Table 3.
Statistical Relationship Between the Presence of Standardized Patient Encounters and COMLEX-USA Level 2-PE Pass Rates
Survey Item χ12 φ
  How many standardized patient encounters did you have at your medical school in years 3 and 4?
   0 vs 1 or more 14.93 0.041
  School Preparation
   Textbooks on physical diagnosis 3.30 0.020
   Coursework in physical diagnosis 6.89a 0.028
   Coursework in humanistic qualities of a physician 5.71b −0.026
   High-stakes standardized patient performance examination at school (prerequisite) 42.06a 0.069
   COMLEX-USA Level 2-PE preparation course (administered by school) 3.91b 0.021
   Practice at rotations 11.87a 0.037
   Other 0.01 0.001
  Self-preparation
   NBOME instructional video on website 7.60a 0.030
   COMLEX-USA Level 2-PE orientation guide on website 28.13a 0.057
   COMLEX-USA Level 2-PE specific study guide 27.83a 0.056
   USMLE specific study guide 1.72 −0.014
   Generic commercial performance evaluation preparation course 2.01 −0.015
   COMLEX-USA Level 2-PE specific commercial preparation course 0.22 0.005
   Study or role-play with friends 7.82a −0.030
   Other 10.71a −0.035

a P<.01.

b P<.05.

Abbreviations: COMLEX-USA Level 2-PE, Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation; NBOME, National Board of Osteopathic Medical Examiners; USMLE, United States Medical Licensing Examination

×