Abstract
Context: Osteopathic medical students have a choice to take the United States Medical Licensing Examination (USMLE) in addition to the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA). However, taking the USMLE requires additional commitments of time, effort, and expense, often for uncertain return. No data are available about the attitudes of graduating osteopathic medical students toward their options regarding the USMLE and how they decide whether to take this examination.
Objective: To uncover attitudes among graduating osteopathic medical students on taking the USMLE.
Method: Using an Internet-based questionnaire, the authors surveyed graduating osteopathic medical students about their experiences with deciding whether to take the USMLE and whether they would advise other students to take the examination.
Results: Nineteen osteopathic medical colleges agreed to participate in the survey. Of the 2744 graduating students at those schools, 978 (35.7%) completed the survey. Students in higher quintiles (ie, top 40%) of class rank were more likely to take the USMLE than those in lower quintiles (ie, bottom 40%) (P<.001). The most common reason cited by students for taking the USMLE was to “keep [their] options open” for residencies (233 of 507 respondents [46%]). Of the 474 students who did not take the USMLE, 171 respondents (36.1%) reported experiencing discrimination for not taking the examination. Four hundred seventy of 981 respondents (47.9%) reported finding residency programs that specifically require the USMLE, and 689 of 981 respondents (70.2%) recommended that future students take at least 1 step of the USMLE.
Conclusion: The majority of graduating osteopathic medical students polled in the present study believed that osteopathic medical students should take the USMLE.
Osteopathic medical students may seek postdoctoral training in residency programs accredited by either the American Osteopathic Association (AOA) or the Accreditation Council for Graduate Medical Education (ACGME). On the basis of our calculations of data from the National Residency Match Program, 1444 of the total 3631 graduating osteopathic medical students (39.8%) in 2011 matched into ACGME-accredited residency programs.
1,2 Osteopathic medical students choose ACGME-accredited residency programs for many reasons, including desirable geographic locations, perceptions regarding quality of programming and educational opportunities, and prospects of training in larger institutions.
3
The requirements for AOA-accredited and ACGME-accredited residencies can differ. The Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) is required by AOA-accredited residency programs. Some residencies accredited by the ACGME require completion of the United States Medical Licensing Examination (USMLE), while others accept scores from either COMLEX-USA or the USMLE. This issue is further complicated by ACGME fellowship programs that sometimes require a USMLE score for admission. The ACGME currently has a pending proposal to require an ACGME-accredited residency for acceptance into an ACGME-accredited fellowship.
4 This proposal, if implemented, may have an effect on other aspects of prerequisites, including the USMLE.
We have observed that osteopathic medical students often ask their instructors and advisors about pros and cons of COMLEX-USA and the USMLE; specifically, they ask whether or not they should take the USMLE. Students who take the USMLE in addition to COMLEX-USA must make additional commitments of time, effort, and expense—often for uncertain return. Few data exist to compare these 2 examinations, including the ways residency programs use examination scores in the application and selection process and the ways students decide whether to take these examinations. Both COMLEX-USA and the USMLE examine student and graduate physicians in basic and generic medical background and skills. In addition, COMLEX-USA integrates the assessment of osteopathic principles and practice throughout the examination.
Although both COMLEX-USA and the USMLE consist of 3 parts, the third level of COMLEX-USA and the third step of the USMLE are administered after graduation and therefore are not typically available for the residency application process. Part 2 of each examination has 2 components; COMLEX-USA Level 2 comprises the Cognitive Evaluation (COMLEX USA Level 2-CE) and the Performance Evaluation (COMLEX-USA Level 2-PE), and the USMLE Step 2 comprises the Clinical Knowledge assessment (USMLE Step 2 CK) and the Clinical Skills assessment (USMLE Step 2 CS). For the present study, we focused our research on COMLEX-USA Levels 1 and 2-CE and USMLE Steps 1 and 2 CK because residency programs frequently use scores from the first part of the examinations and the cognitive portion of the second parts of the examinations for their selection process.
5
We developed a simple questionnaire to assess the attitudes of osteopathic medical students toward COMLEX-USA and the USMLE (
1). This information may assist advisors, professors, and other mentors in counseling students more personally and realistically about the advisability of taking the USMLE in addition to COMLEX-USA.
The Nova Southeastern University Institutional Review Board reviewed the study protocol and gave it an “exempt” status.
From March 29, 2010, to June 1, 2010, we sent surveys to graduating osteopathic medical students at 29 osteopathic medical college locations. We obtained the list of the college locations as well as a list of representatives for each location from the American Association of Colleges of Osteopathic Medicine. We sent a survey link to the representatives and asked them to forward the survey to the graduating osteopathic medical students at their schools. During the study period, we reviewed the data monthly to assess which locations were responding to the survey. We sent e-mail reminders to the representatives of the locations from which we had not received any responses and reminded and encouraged them to forward the link to their students. Locations that were not expected to graduate any students were not sent any reminders.
The survey was built on the SurveyMonkey Web site. The instrument consisted of questions in a “Yes/No” format (dichotomous scale), with room for respondents to make personal comments (
1). Because the instrument was not administered in a complex survey design, the data were not weighted. The survey was analyzed as a cross-sectional study.
6,7 For analysis purposes, the students self-categorized their class rank by quintiles (eg, top 20%).
Our survey questions met the content, cognitive, and usability standards to ensure that the questions asked the right information, that the respondents could understand and answer the questions, and that the entire survey could be used in practice.
6,8 The Spearman rank-order correlation coefficient (ρ) was measured to assess correlations among the multiple-choice questions. Because the item variances were large, the standardized Cronbach α correlation coefficient was used. Contingency tables for the categorical data with the corresponding χ
2 tests of independence were used to assess associations when appropriate. All tests were 2-sided and the level of α error was .05.
The total number of osteopathic medical students who graduated in 2010 was 3631.
9 One school declined to participate in the survey and 9 others either did not respond to the request or had graduating students who chose not to respond; thus only 19 of the 29 locations (65.5%) responded to our request. The 19 participating locations had a total of 2744 graduating students. Of those, 981 students completed the survey, for a response rate of 35.7%. Colleges with the highest number of respondents included Des Moines University College of Osteopathic Medicine (136 [13.9%]), Philadelphia College of Osteopathic Medicine (101 [10.3%]), New York College of Osteopathic Medicine of New York Institute of Technology (92 [9.4%]), and Western University of Health Sciences College of Osteopathic Medicine of the Pacific (89 [9.1%]).
Eight hundred fifty-two respondents (87%) reported being in the top 60% of their medical school class. Students in higher quintiles (ie, top 40%) were more likely than those in lower quintiles to take the USMLE (
P<.001) (
Table 1). Of the respondents, 460 (46.9%) reported taking the USMLE Step 1, while 264 (26.9%) reported taking the USMLE Step 2 CK. Among the 507 respondents who took at least 1 step of the USMLE, primary reasons for taking the USMLE included “to keep my options open” (233 [46.0%]) and “to enhance my chances of getting into an allopathic residency” (179 [35.3%]) (
Table 2). Other reasons cited included location (29 [5.7%]) and lack of osteopathic residency programs in a chosen specialty (8 [1.6%]). Thirty-three respondents (6.5%) reported that their primary reason for taking the USMLE was a specific aversion to osteopathic residency programs. Of the 474 students who took neither Step 1 nor Step 2 CK of the USMLE, 171 (36.1%) reported experiencing discrimination for not taking the examination (
P<.001).
Four hundred seventy respondents (47.9%) reported finding residency programs that specifically required the USMLE. When asked what advice they would give to other osteopathic medical students regarding the USMLE, 689 respondents (70.2%) recommended that students take at least 1 step of the USMLE, while 292 respondents (29.8%) recommended that students not take the examination (
Figure 1). Three hundred eight respondents (31.4%) advised that students take USMLE Step 1 only, and 330 (33.6%) recommend that students take Steps 1 and 2 CK. Fifty-one respondents (5. 2%) advised that students take Step 2 CK only.
Some questions were correlated to a statistically significant degree using the Spearman rank-order correlation coefficient (ρ). For example, the question regarding class rank was significantly correlated with the questions on taking USMLE Step 1 and Step 2 CK, finding programs that specifically required USMLE, experiencing discrimination for not taking the USMLE, giving advice on whether to take the USMLE (
Table 3), and giving reasons for why the USMLE was taken (
P<.001). The question on taking USMLE Step 1 was significantly correlated with the question on osteopathic medical school (
P=.006). Whether respondents indicated they would recommend that other students take USMLE Step 1 and Step 2 CK was statistically significantly correlated with whether they reported taking USMLE Step 1 (
P<.001) and Step 2 CK (
P<.001) (
Table 4).
The 8 analyzed questions had a moderate reliability
8,10 in measuring the overall response (ie, standardized Cronbach α=0.35).
Our Web survey had a response rate of 35.7%. One of the major limitations of the present study was the dissemination of the survey using electronic mailing lists. Respondents may have not seen the e-mail, or the e-mail might have been deleted (either manually or by software), which might have decreased our response rate.
Another limitation was lack of response from the colleges. Not all college representatives responded to our request. Of those that did not participate, only 1 provided a reason for not participating. The dean of 1 osteopathic medical school opted not to allow the survey to go out to the students because of concerns about how the data may be used. The lack of participation of other schools might be a result of administrative oversight or other factors. These factors likely decreased our response rate. A future study could be designed to directly e-mail graduating osteopathic medical students for a greater response rate.
A high percentage of survey respondents reported being in the higher quintiles of class rank. It is possible that students with a higher class rank are more interested in the topic of the USMLE and perhaps more likely to apply to ACGME residencies than students with a lower class rank. This finding may be indicative of a recall bias and also may create a selection bias. These quintiles are self-reported and not confirmed, which may be a potential source of error.
Step 1 of the USMLE emphasizes the understanding of scientific mechanisms, while Step 2 CK emphasizes diagnosis. It would be reasonable to think that residency program directors would place more importance on USMLE Step 2 CK scores when evaluating candidates because the content of Step 2 CK could be viewed as more clinically relevant. However, our survey respondents were more likely to report taking USMLE Step 1 and recommending that future students take USMLE Step 1 than USMLE Step 2 CK. One potential reason for this finding could be that students believe that Step 1 is more important to allopathic residency programs. In addition, students may believe that if they do well on Step 1, Step 2 CK is unnecessary. Or, students who do poorly on Step 1 may not wish to continue additional USMLE steps. Further investigation is needed into why students are more likely to take Step 1 than Step 2 CK.
Interestingly, a survey of ACGME program directors
5 found that the USMLE Step 1 score is the second most important factor from a list of 14 criteria in their decision making, after grades in required clerkships. Grades in senior electives in candidates' chosen specialties and number of honors grades both ranked above the USMLE Step 2 CK score. It would be valuable to poll osteopathic residency program directors about the importance of COMLEX-USA Levels 1 and 2-CE scores and—more generally—how they rank numerous criteria in candidate selection.
The most common response cited for why students chose to take the USMLE was that they wanted to keep their options open. This result is likely because students complete the first part of both COMLEX-USA and the USMLE halfway through their medical education. At this point, students have not yet completed any clinical rotations, and many are unsure of their residency plans. Nearly one-third of the respondents who did not take any part of the USMLE reported experiencing discrimination. The term “discrimination” was not defined by our survey but rather was left open to interpretation by student participants. Further investigation is needed to understand the perception of discrimination among osteopathic medical students who apply to ACGME programs.
Students who reported being in the upper quintiles of their classes are likely to advise other students to take USMLE, as well as to take it themselves. However, this advice might not be appropriate for all students. In a study by Chick et al,
11 scores of USMLE Step 1 and COMLEX-USA Level 1 were compared among a large cohort of students applying for allopathic internal medicine residencies. Mean COMLEX-USA scores for this cohort were higher than average COMLEX-USA Level 1 scores, although USMLE scores for the same group were average. These findings suggest that students who perform at or below the mean on COMLEX-USA may underperform on USMLE, which could negatively affect their application prospects. However, this study did not take into account test content or scaling. Thus, these students might not have been representative of all osteopathic medical students, and the fact that their USMLE and COMLEX-USA scores were comparable cannot be extrapolated to mean that the same would hold true for all osteopathic medical students. In a smaller study, Sarko et al
12 found that COMLEX-USA Level 1 and USMLE Step 1 scores among a group of osteopathic medical students who applied to the same emergency medicine residency program were not interchangeable, with a fairly large correlation (
r2=0.62). Again, this study did not consider test content or scaling.
It should be noted that osteopathic medical students tend to score lower on USMLE Step 1 and 2 CK than medical students of schools accredited by the Liaison Committee on Medical Education but score higher than graduates of foreign medical schools.
13 An extensive survey conducted by Graneto
14 demonstrated that, even among leaders in osteopathic medical education, considerable uncertainty exists about whether USMLE and COMLEX-USA are comparable ways to evaluate students.
No data exist on how osteopathic faculty currently counsel osteopathic medical students on whether or not to take the USMLE. None of the respondents cited faculty advice as their reason for deciding to take the USMLE. This finding could be because of the following possibilities: a dearth of data exist for faculty to draw upon to counsel their students on this topic, faculty may be advising their students not to take the USMLE, or students are not using faculty advice when deciding to take the USMLE. We believe that the present study represents the first reasonably objective findings specific to this issue that might aid faculty in their ability to counsel osteopathic medical students on this topic. Further studies on faculty and ACGME program director's opinions regarding these examinations might be beneficial.
The present study did not address the relationship between graduating osteopathic medical student opinion on the USMLE and specialty choice. This area warrants further investigation. Future studies may also examine the relationship between graduating osteopathic medical student opinion on the USMLE and the osteopathic institution that they attend.
We could find no data on the number of ACGME-accredited residency programs that require students to take the USMLE. This topic may warrant future investigation.
The medium reliability coefficient can be explained by the large variances of the analyzed items and by the binary level of the questions, instead of using a Likert-scale format.
The majority of graduating osteopathic medical students polled in the present study believed that osteopathic medical students should take the USMLE. Faculty advisors might consider the perceptions reported by the respondents in this survey when advising students. Overall, this study may be used as a pilot study for future opinion survey studies.