In 1987, Tonesk and Buchanan¹ conducted a pilot study of 10 medical schools that described common problems with clerkship evaluation systems. Some problems cited were the evaluators' unwillingness to record negative evaluations, inadequately defined evaluation criteria, and lack of training for evaluators. In 1990, Magarian and Mazur² conducted a survey of 101 medical schools on the type of clerkship evaluation system used to assess student performance. The survey found that 68% of schools used a pass-fail grading system, 28% used letter grades, and 4% reported numeric scores. In 1992, Hunt³ described a model evaluation system containing four stages and provided a list of “signs and symptoms” one could consult to detect an evaluation system with problems. Other articles^4-7 express a general desire for a valid and reliable clerkship evaluation system that provides adequate and appropriate evaluation and feedback concerning the developing clinical skills of medical students. 

Many articles have also addressed the validity, reliability, and generalizability of data gathered from existing clerkship evaluation systems.^7-12 While some studies^7,8 have supported the validity of rating forms used in these evaluation systems, many researchers argue that the topic of form reliability has not been adequately addressed in the literature. For example, Levine and colleagues⁹ determined that one way to increase form reliability would be to have multiple preceptors rate each student. Reznick and coauthors¹⁰ used interrater agreement coefficients to look at relative and absolute reliabilities and determined that only five raters per student were required to obtain the commonly recommended reliability of 0.80. Carline and coinvestigators¹¹ determined that a minimum of seven raters per student were required to obtain a reliability of 0.80. One study¹² noted that “raters usually have had limited opportunities to observe a student's skills before completing the evaluation, affecting the reliability of the assessment.” Researchers¹² further posited that three raters were required to obtain rating reliability. 

In the process of demonstrating the validity of clerkship evaluation forms, many authors^13-17 have shown correlations between these evaluation scores. Correlations with other academic measures or correlations with measures from outside agencies such as the National Board of Osteopathic Medical Examiners (NBOME) or the National Board of Medical Examiners (NBME) have also been demonstrated.^13-17 Lazaro and colleagues¹³ reported the relative lack of correlation between clinical performance and written examinations whether they were generated internally or by national agencies. Metheny¹⁴ also found a weak relationship between clinical evaluations and NBME Subject Examination scores. Campos-Outcalt and colleagues¹⁵ compared ratings of clinical performance and problem-solving skills with preadmission measures (ie, undergraduate GPA and Medical College Admission Test [MCAT] score) and also with NBME Parts I and II subtests and total score values. Their study found no correlation between clinical evaluations and problem-solving skills and any preadmission measures. In addition, their study found mild correlations between clinical evaluations and NBME Parts I and II scores. They found no correlation between problem-solving skills and NBME Part I, and only a moderate correlation between problem-solving skills and NBME Part II. Although Silver and Hodgson¹⁶ reported no correlation between admissions measures (ie, undergraduate GPA and MCAT score) and clinical performance, they found a good correlation between admissions measures and NBME Part I performance. Callahan and coauthors¹⁷ reported a high (0.01) correlation between family medicine, internal medicine, obstetrics and gynecology, pediatrics, and surgery clerkship evaluations and United States Medical Licensing Examination (USMLE) Step 2 and a high (0.01) correlation between family medicine, internal medicine, and pediatrics clerkship evaluations and USMLE Step 3. Ferguson and Kreiter¹⁸ reported a high correlation between preclinical and clinical training measures such as objective structured clinical evaluations (OSCEs), case-based reports, multiple-choice clinical skills examinations, and clinical preceptor evaluation forms. They also noted that basic science grades, undergraduate GPA, and MCAT scores did not correlate with clinical preceptor evaluation forms. 

To determine form validity, we looked at clerkship evaluation systems and forms and their correlation to institutional and national measures of both preclinical and clinical performance. The Clinical Education Grade Form was created in Microsoft Word 98 for Windows (Microsoft Corporation, Redmond, Wash). It was studied previously¹⁹ and researchers found that student grades were distributed more accurately (ie, more even distribution and lower mean) using the Clinical Education Grade Form than WVSOM's previous paper-based form. 

Prior to 2002, WVSOM used a paper-based clinical education grading form that had several descriptive items that required individual numeric ratings from clinical preceptors. However, the final letter grade was intended as a global assessment and was assigned without reference to the numeric ratings. Although students were satisfied with the results achieved using the old form, WVSOM administration was concerned about its validity and the potential for grade inflation. For the WVSOM graduating class of 1998, 81% of the grades assigned by preceptors were A's (which was assigned a numeric grade of 99); 17%, B (89); and 2%, C (79) or F (65), resulting in a mean GPA of 96.8 (almost perfect) for phase 3, the clinical years of osteopathic medical school. The same students received an average of 85.7 for phase 1 (courses taught through April of year 1) and 89.5 for phase 2 (organ systems courses and related clinical courses taught in the last part of year 1 and throughout year 2), indicating significant grade inflation during phase 3. Further, for this graduating class, the correlation between phase 3 GPA and performance on the Comprehensive Osteopathic Medical Licensing Examination USA Level 2 Cognitive Evaluation (COMLEX-USA Level 2-CE) written examination was only 0.16, leading to even more questions about the form's validity. 

Data analysis was used to evaluate the extent to whether the Clinical Education Grade Form decreased grade inflation and correlated with other measures of academic success, including OSCE performance. Other than those found in the OSCE and COMLEX-USA examinations, the current investigation does not include analysis of written tests or case studies required during phase 3 for the class of 2005. This omission is due in part to changes in process at WVSOM with regard to how those components are assigned. 

Approval was obtained from WVSOM's institutional review board to conduct this retrospective review using existing educational data. Subjects for the study were 70 students who were in the class of 2005 as of April 2004 and who took WVSOM's year 3 OSCE as well as the COMLEX-USA Level 2-CE (written examination) and COMLEX-USA Level 2 Performance Evaluation (COMLEX-USA Level 2-PE), a clinical skills examination. 

Preceptors are asked to evaluate student performance on clinical rotations. Five performance factors are addressed (Figure 1): 

Possible ratings were “failure,” “needs improvement” (61-68), “adequate” (73-77), “good” (81-89), “excellent” (92-96), “truly exceptional” (100), or “not observed.” A brief description of the behaviors associated with each rating is included in the Clinical Education Grade Form. All ratings were converted to numeric scores using the values listed. The subscore values for all core clerkships, selective clerkships, and elective clerkships were averaged for use in analysis. Figure 2 lists the various clinical rotations available at WVSOM and describes their respective durations at the time of the study. 

At WVSOM, clinical rotation grades are determined predominately by ratings from clinical preceptors who have not previously taught the student in a more formal educational environment. These preceptors spend extensive time in one-on-one interaction with the students and thus have enough direct contact with students to provide a thorough evaluation. 

For a few rotations (predominately electives), students had the option of spending 50% of the rotation with one preceptor and the other 50% with a different preceptor (9.1% of total rotations, 2.5% of required and selective rotations, and 36% of elective rotations). In these instances, the subscore values assigned by each preceptor were averaged to determine the rotation grade. 

The National Board of Osteopathic Medical Examiners administers a series of licensing examinations designed to evaluate student competency at different stages in the educational process. At the end of year 2, students take COMLEX-USA Level 1; during year 4, students take COMLEX-USA Level 2-CE. In the spring of 2005, COMLEX-USA Level 2-PE was introduced. All members of WVSOM's class of 2005 were required to pass COMLEX-USA Level 1 and COMLEX-USA Level 2-CE in order to graduate. All members of WVSOM's class of 2005 were required to participate in, but not necessarily pass, COMLEX-USA Level 2-PE. Numeric scores were available for both the Level 1 and Level 2-CE examinations, and pass-fail status was established for the Level 2-PE examination. The scores for the COMLEX-USA Level 2-PE examination were coded as “1” for pass and “0” for fail. 

As noted, the scores for COMLEX-USA Level 2-PE were reported to colleges only as a “pass” or “fail” for each student. Therefore, these data are dichotomous. The other variables under study were continuous. Accordingly, appropriate correlations were calculated using SPSS for Windows (version 10.1; SPSS Inc, Chicago, Ill). 

Correlations with selected academic performance measures are shown in the Table. 

The reliability of the subscores from the Clinical Education Grade Form was varied. Subscore 1 had a reliability of 0.80; subscore 2, 0.59; subscore 3, 0.65; and subscore 5, 0.78. 

For subscore 4, there were not enough data to determine reliability because all WVSOM students in this graduating class took at least one rotation with a preceptor (sometimes an allopathic physician [MD]) who did not evaluate students for OPP. The clerkships that had the largest proportion of osteopathic physicians (DOs) serving as preceptors, namely the three family medicine clerkships, were isolated from the rest of the dataset and used to determine the reliability of the OPP subscore. All other subscores for these clerkships were also reevaluated for reliability based on this reduced dataset. 

The reliability ratings for the family medicine clerkship subscores were as follows: subscore 1, 0.52; subscore 2, 0.53; subscore 3, 0.51; subscore 4, 0.44; and subscore 5, 0.34. 

All subscores were correlated, ranging from a low correlation of 0.46 to a high correlation of 0.94. The highest correlation was between subscore 1, clinical knowledge, and subscore 3, application of knowledge. 

The correlations of subscores 2 through 5 with COMLEX-USA Level 1 were not statistically significant. Subscore 1 was significantly correlated with student performance on COMLEX-USA Level 1 (P=.05). The correlation of subscores 1, 2, 3, and 5 with student performance on COMLEX-USA Level 2-CE was statistically significant (P=.01). Subscore 4 was not correlated with student performance on COMLEX-USA Level 2-CE. Subscores 1 through 4 were not correlated with student performance on COMLEX-USA Level 2-PE. However, subscore 5 did have a statistically significant correlation with student performance on COMLEX-USA Level 2-PE (P=.05). 

Subscore 1, subscore 3, and subscore 5 were significantly correlated with all academic measures (GPAs for both preclinical phases and years, as well as for physical diagnosis and OPP courses) (P=.01). Subscore 2 was significantly correlated both with phase 2 and year 2 GPAs (P=.01) as well as with GPAs from physical diagnosis and OPP courses (P=.05). Subscore 2 had no significant correlation with either phase 1 GPA or year 1 GPA. Subscore 4 was correlated with GPAs for both preclinical phases and years as well as OPP courses (P=.01). Subscore 4 was significantly correlated with GPA for physical diagnosis courses (P=.05). 

The correlations for WVSOM's OSCE were as follows: subscore 1 was significantly correlated with total score, history, SOAP Note Form, organization, and professionalism (P=.01), as well as with physical examination (P=.05). Subscore 1 was not correlated with communication. Subscore 2 was significantly correlated with organization (P=.01), total score, and SOAP Note Form (P=.05). Subscore 2 was not correlated with history, physical examination, communication, or professionalism. Subscore 3 was significantly correlated with total score, history, SOAP Note Form, organization, and professionalism (P=.01), as well as with physical examination (P=.05). Subscore 3 was not correlated with communication. Subscore 4 was significantly correlated with total score and physical examination (P=.01). Subscore 4 was not correlated with physical examination, SOAP Note Form, communication, organization, or professionalism. Subscore 5 was significantly correlated with total score, organization, and professionalism (P=.01), as well as with physical examination (P=.05). Subscore 5 was not correlated with communication.1 

As determined by the weighted averages of all clinical clerkship grades in the last 2 years of training, the average WVSOM student GPA was 92.6 as compared to the average GPA for the first 2 years of training, 87.3. The distribution of grades for the final 2 years of training was A, 75.9%; B, 21.2%; C, 2.9%, and F, 2.9%; as compared to the distribution of grades for the first 2 years of training A, 27.1%; B, 70%; C, 2.9%, and F, 2.9%. 

The electronic Clinical Education Grade Form was developed to help preceptors conduct better evaluations of student performance in clinical rotations. The Clinical Education Grade Form had an average preceptor rating of 92.6, somewhat lower than the average rating of 96.8 for the class of 1998 (P=.05), which used the old paper-based form that was discontinued in 2002. While we would like to believe that students excelled on their clinical rotations, we believe it is more likely the preceptor ratings remain inflated with the Clinical Education Grade Form. We will continue to expand faculty development activities to better define WVSOM's necessary expectations in order for students to receive an “A” grade. 

With the previous paper-based grading form, the correlation between preceptor ratings and COMLEX-USA Level 2-CE was low (0.16). With the new electronic Clinical Education Grade Form, the correlation between subscore 1 and COMLEX-USA Level 2-CE was 0.50, which seems more appropriate. Correlations with other written and performance examination results seem to suggest the validity of the new instrument. For example, the correlation between subscore 1 and performance on WVSOM's OSCE is 0.43, which supports the validity of both measures. 

All subscores were correlated, with the highest correlation (0.94) between subscore 1 and subscore 3. Because these two variables (clinical knowledge and application of knowledge) are logically related, we believe this high correlation is appropriate. We intend to leave both items on the scale because students' final grades are determined by calculating the simple average of each subscore. We think it is appropriate to have both items on the Clinical Education Grade Form. 

Preceptors did not always fully complete the new form. Consequently, students did not receive a rating for all subscores. The subscore most frequently left incomplete by preceptors was subscore 4, OPP. We must grant that some rotations are not always appropriate for the demonstration of the clinical skills included in the OPP subscore. In addition, preceptors were either DOs or MDs. Most MDs did not provide a rating for subscore 4. We assume the omission of this rating was often intentional and that most MDs decided not to rate students on this aspect of their performance due to their relative lack of training in OPP. However, some DOs also did not rate students on this item. Because of incomplete results for subscore 4, we were unable to determine the reliability of the OPP subscore. When the dataset was reduced to rotations with a high proportion of preceptors who were DOs (eg, family medicine clerkships), subscore 4 reliability was 0.44. This reliability is lower than expected—and low enough to reduce the possibility that these ratings would highly correlate with academic variables documented earlier in the medical school curriculum (ie, phase 1 or phase 2). Some faculty development in the application of the OPP subscore has already occurred. We intend to work more closely with preceptors to standardize and build confidence in their abilities to evaluate students for OPP. 

The current study has several limitations. First, the data were collected at only one rural, osteopathic medical school, with a graduating class composed of 70 students. The ability of schools with other missions and structures to adopt the new electronic Clinical Education Grade Form may be limited; adaptation may be preferable. Also, the current analysis does not consider the impact of written examinations taken at or near the end of rotations. Similarly, it does not include the impact of case studies on rotation grades. Neither does this investigation address the extent to which preceptor ratings may correlate with such measures. Beginning with WVSOM's class of 2005, changes in end-of-rotation examinations have been implemented, so analysis of their impact would have no value for curriculum planning. However, for the continued improvement of the program, eventually it will be necessary to analyze how much end-of-rotation examinations correlate with preceptor ratings. 

As noted, changes to the electronic Clinical Education Grade Form have been ongoing and other changes are underway. First, as initially implemented, the numeric values associated with the grade scale were intentionally removed in the hope that preceptors would focus on the characteristics described (eg, “consistently demonstrates expected, good knowledge”) rather than the associated grade value (eg, “85,” a mid-B grade). However, discussions with students at graduation exit interviews revealed that some students wrote these numbers on the forms in the hope of receiving a better grade. Student-altered forms are a possible source of error variation (ie, variability not related to the skills and abilities we wish to measure). Therefore, to prevent some students from having an unfair advantage, we placed numeric grade values on the electronic evaluation form. 

Second, the Clinical Education Grade Form was developed before an emphasis on “core competencies”^19,20 was implemented at the predoctoral level for colleges of osteopathic medicine, and before WVSOM's faculty expanded the application of contemporary models for teaching and evaluating communication skills.^21,22 The new electronic evaluation form incorporates taking a patient's medical history as one of the examples for subscore 2, and the item “worked well with other healthcare team members” as an example for subscore 5. The Clinical Education Grade Form is also being revised to make interpersonal communication a separate subscore: interpersonal communication and professionalism. Accordingly, some descriptors will be modified to emphasize communication skills. Other ways of measuring the core competencies (eg, basic knowledge of OPP and osteopathic manipulative treatment) will also be pursued.²⁰ 

Finally, it is clear that WVSOM must continue to expand faculty-development activities for clinical preceptors, specifically with regard to faculty expectations for WVSOM students. At its midwinter continuing medical education conferences, WVSOM has offered special sessions regarding contemporary practices in physical examination, communication skills, writing physician progress notes, and osteopathic manipulative medicine. In February 2005, session participants were asked to take part in a three-station OSCE. In February 2006, they were asked to critique three aspects of student performance: a video of osteopathic manipulative treatment, a role play of communication skills, and sample written progress notes. A major objective of these programs was to help preceptors become more confident with WVSOM's current expectations for students' clinical performance. Similar programs are being planned, as well as additional training for preceptors that specifically focus on the new electronic evaluation form. 

The new electronic Clinical Education Grade Form decreased grade inflation for clinical rotations at WVSOM but did not eliminate it completely. Some subscores demonstrated acceptable reliability and were moderately correlated with other measures of academic ability. These moderate correlations provide some support for the validity of the new electronic form. Continued faculty development will be undertaken at WVSOM regarding the form. A special emphasis on standardization and improving preceptor confidence in using the OPP subscale is planned. Furthermore, the Clinical Education Grade Form will be further revised to focus more on the core competencies now required at colleges of osteopathic medicine.