Free
Original Contribution  |   January 2011
Effects of Repeated Use of the American Osteopathic Association's Clinical Assessment Program on Measures of Care for Patients With Diabetes Mellitus
Author Notes
  • From the Ohio University College of Osteopathic Medicine in Athens (Dr Shubrook), Applied Health Services in Worthington, Ohio (Dr Snow), and the Department of Quality and Research at the American Osteopathic Association in Chicago, Illinois (Ms McGill). 
  • Address correspondence to Jay H. Shubrook, Jr, DO, Associate Professor of Family Medicine, Ohio University College of Osteopathic Medicine, Grosvenor Hall, Athens, OH 45701-2979. E-mail: shubrook@ohio.edu. 
Article Information
Endocrinology / Medical Education / Hypertension/Kidney Disease / Professional Issues / Diabetes / Graduate Medical Education
Original Contribution   |   January 2011
Effects of Repeated Use of the American Osteopathic Association's Clinical Assessment Program on Measures of Care for Patients With Diabetes Mellitus
The Journal of the American Osteopathic Association, January 2011, Vol. 111, 13-20. doi:10.7556/jaoa.2011.111.1.13
The Journal of the American Osteopathic Association, January 2011, Vol. 111, 13-20. doi:10.7556/jaoa.2011.111.1.13
Abstract

Context: The American Osteopathic Association developed its Clinical Assessment Program (AOA-CAP) for Residencies to provide a mechanism for osteopathic residency programs to measure and improve their quality of patient care.

Objective: To compare program performance in processes of care and intermediate outcomes for patients with diabetes mellitus in residency programs that contributed data to the AOA-CAP for the first time vs residency programs that contributed data repeatedly.

Methods: Osteopathic family medicine residency programs that entered data into the AOA-CAP diabetes registry between July 1, 2005, and December 31, 2007, were included in the present study. Residency programs were separated into those that entered data into the registry for the first time during the 2005-2007 cycle (ie, first-time programs) and those that also entered data into the registry during the previous cycle (2003-2005) (ie, repeat programs). Measures of processes of care were annual foot examination, annual referral for ophthalmologic examination, annual microalbuminuria screening, use of angiotensin-converting enzyme (ACE) inhibitors or angiotensin receptor blockers (ARBs) if albuminuria is present, use of ACE inhibitors or ARBs if hypertension is present, glycosylated hemoglobin (HbA1c) test in the previous year, and low-density lipoprotein cholesterol (LDL-C) test in the previous year. Measures of intermediate outcomes were control of blood pressure, HbA1c, and LDL-C. Processes of care and outcome composite scores were also computed.

Results: Data from 52 osteopathic family medicine residency programs consisting of 2568 patient cases were analyzed. Twenty-three first-time programs with 992 cases and 29 repeat programs with 1576 cases entered data into the registry in the 2005-2007 cycle. Repeat programs had statistically significant better performance than first-time programs in the composite measure of processes of care (P=.0023)—largely the result of increased use of ACE inhibitors and ARBs in patients with albuminuria (P=.0087). The difference in the composite measure of intermediate outcomes was not statistically significant between the 2 groups.

Conclusion: Repeated participation in the AOA-CAP registry was associated with improved residency program performance on the composite process of care measure but not on intermediate outcome measures for patients with diabetes mellitus. This finding suggests that osteopathic residency programs need to provide better training on adjusting patient care according to performance results.

In 2001, the Institute of Medicine's Crossing the Quality Chasm: A New Health System for the 21st Century1 summarized concerns about large gaps between the healthcare that patients should receive and the healthcare that they actually receive. That report provided a vision for reducing gaps in healthcare by providing care that is safe, effective, patient centered, timely, efficient, and equitable. The need for improving the healthcare system arose from a rapid expansion in the knowledge base of medical care and from the increasing percentage of patients with chronic disease. For the healthcare system to provide the ideal model of care, sweeping changes in graduate medical education are required. 
The incidence of cardiovascular disease, diabetes mellitus, and chronic lung conditions have increased dramatically as our population has aged. Approximately 45% of people in the United States have a chronic disease, accounting for about 75% of US healthcare expenditures.2,3 Collectively, chronic diseases are the leading cause of disability and death in the United States.4 
Managing chronic disease requires a healthcare system that is focused on continually assessing event risk and that uses staged approaches for primary and secondary prevention. Osteopathic physicians need to be aware of resources in their communities that can help improve treatment and outcomes for patients with chronic disease. Equally important, osteopathic physicians need to understand the complexity of transitions of patients between healthcare providers and the risks that poor communication or poor coordination have for clinical, financial, and functional outcomes. 
Healthcare remains focused on episodic care. This narrow focus may contribute to the healthcare fragmentation that many patients perceive today. To respond to the changing needs of the population, medical educators need to teach the tenets of delivering evidence-based care throughout the healthcare continuum. As healthcare moves away from the management of complications, the prevention of chronic disease complications becomes the new paradigm. 
Diabetes mellitus is a disease that is closely monitored by insurers. Payers monitor processes of care and intermediate outcomes. Registries exist for healthcare professionals who are willing to submit data on clinical care of patients with diabetes mellitus. In turn, these registries provide healthcare professionals with information on healthcare processes and outcomes, as well as comparative data from other contributors. Ample evidence from such registries has shown that the processes of care and outcomes related to diabetes mellitus have not come close to meeting well-defined and widely supported goals for care.5-9 This lack of progress is often referred to as “clinical inertia.” 
Graduate medical education developed around an apprenticeship model, and it has continued to reflect that framework. Interns and residents learn in the context of 1 patient at a time, and consequently, they often struggle to understand the perspective of an overall medical practice and the concept of patient care in the context of public health. Interns and residents are accustomed to receiving performance reviews on at least a daily basis, if not on a patient-by-patient basis. These reviews often come as direct feedback from a supervising physician. However, often lacking is a broader review of how interns and residents handle processes of care and how they perform within the context of the overall care provided in the office, region, and nation. 
Clinical Assessment Program for Residencies
The American Osteopathic Association (AOA) developed its Clinical Assessment Program (AOA-CAP) for Residencies in 2003 to provide a mechanism for osteopathic residency programs in family medicine and general internal medicine to measure and improve their quality of patient care.10,11 The program is a Web-based registry that uses a standard method of sampling patients and collecting information from their medical records on key processes and outcomes of care. The AOA-CAP process and outcome measures were developed by a steering committee, and they are consistent with national evolving measures. In the AOA-CAP, current clinical practices are measured and then compared to evidence-based practice guidelines representing such state-of-the-art professional standards as the National Committee for Quality Assurance's Health Plan Employer Data and Information Set (NCQA HEDIS),12 the US Public Health Services' Healthy People 2010 targets,13 and the American Diabetes Association's standards of medical care.14 
The AOA-CAP is based on several guiding principles, including the following10,11: 
  • Information technology drives the program. This guideline resulted in the development of an AOA-supported, Web-based architecture providing key components to participating residency programs and osteopathic physicians.
  • The AOA's measures are standardized to national measures, including those of the NCQA's HEDIS, the federal Centers for Medicare and Medicaid Services, the Doctors' Office Quality-Information Technology Initiative, and The Joint Commission (formerly the Joint Commission on Accreditation of Healthcare Organizations).
  • Tools using evidence-based standards were developed to provide osteopathic residency programs with methods of measuring the quality and safety of care that they provide to their patients. These tools are organized as sets of measures pertinent to clinical conditions, such as diabetes mellitus and hypertension.
  • Continuous quality improvement is incorporated into graduate medical education and the practice of osteopathic medicine.
In the AOA-CAP for Residencies, data abstracted from patients' medical records are analyzed to determine residency program performance. These data can be used as a quality-improvement tool to measure the impact and effectiveness of residency program treatment protocols in meeting standards of practice for specific clinical categories of patients. The 8 clinical categories currently available in the AOA-CAP for Residencies are asthma, childhood immunization, chronic obstructive pulmonary disease, coronary artery disease, diabetes mellitus, hypertension, low back pain, and women's health.10,11 
Using reports based on AOA-CAP data, residency program directors can follow the professional development of their residents longitudinally, over the full course of training, and they can compare their programs' progress to a national sample. Thus, the reports promote personal and group improvement, and they introduce residents to physician-level and practice-level issues, such as opportunity gaps in care and ways to systematically improve care. 
The AOA-CAP for Residencies retrospectively evaluates healthcare within 8 clinical measure sets. Sample size that residency programs need for each clinical entity has been defined by using either population estimates or pre-intervention and postintervention sample frames. Each participating residency program is provided with standardized materials to submit to its institutional review board under the assumption that the residency program will submit AOA-CAP data for an expedited review as a quality improvement project. Reports are generated for each program using indicator constructs similar to those used by external bodies such as the NCQA and The Joint Commission.10,11 The AOA-CAP Steering Committee has the flexibility to generate additional indicators. 
Figure 1.
Diabetes Measurement Abstraction Tool used to record measures of processes of care and intermediate outcomes for osteopathic family medicine residency programs using the diabetes registry of the American Osteopathic Association's Clinical Assessment Program (AOA-CAP). This tool was altered for graphic enhancement only. Abbreviations: ACE, angiotensin-converting enzyme; ARB, angiotensin receptor blocker; BMI, body mass index; BP Dia, diastolic blood pressure; BP Sys, systolic blood pressure; DM, diabetes mellitus; Edu, education; HbA1c, glycosylated hemoglobin; HTN, hypertension; ID, resident identification; Intol, intolerance; LDL, low-density lipoprotein; Rec, recovery; Wt, weight.
Figure 1.
Diabetes Measurement Abstraction Tool used to record measures of processes of care and intermediate outcomes for osteopathic family medicine residency programs using the diabetes registry of the American Osteopathic Association's Clinical Assessment Program (AOA-CAP). This tool was altered for graphic enhancement only. Abbreviations: ACE, angiotensin-converting enzyme; ARB, angiotensin receptor blocker; BMI, body mass index; BP Dia, diastolic blood pressure; BP Sys, systolic blood pressure; DM, diabetes mellitus; Edu, education; HbA1c, glycosylated hemoglobin; HTN, hypertension; ID, resident identification; Intol, intolerance; LDL, low-density lipoprotein; Rec, recovery; Wt, weight.
The AOA-CAP for the diabetes measure set focuses on the following processes of care: annual foot examination, annual referral for ophthalmologic examination, annual microalbuminuria screening, use of angiotensin-converting enzyme (ACE) inhibitors or angiotensin receptor blockers (ARBs) if albuminuria is present, use of ACE inhibitors or ARBs if hypertension is present, and completion of a glycosylated hemoglobin (HbA1c) test and a low-density lipoprotein cholesterol (LDL-C) test in the previous year. The intermediate outcomes tracked in the AOA-CAP diabetes database are control of blood pressure (normal level <80/130 mm Hg), HgA1c (normal level <7%), and LDL-C (normal level <100 mg/dL).10,11 The Diabetes Measurement Abstraction Tool used to record these measures is shown in Figure 1. 
In the current study, we evaluated absolute performance of osteopathic family medicine residency programs on process and outcome measures of care of patients with diabetes mellitus. We then compared residency programs that contributed data to the AOA-CAP registry for the first time to residency programs that participated a second time. The latter group had an opportunity to review their performance based on previously submitted data. By comparing the performance of these 2 groups, we tested the hypothesis that repeated use of a clinical registry facilitates improved quality of care for patients with diabetes mellitus. 
Methods
Residency program directors were informed about the AOA-CAP at the annual program directors meeting. No training was provided for data entry. Residents contributing data to the AOA-CAP diabetes registry were provided a data dictionary defining consistent methods of data abstraction. Each resident was instructed by his or her residency program director to enter data on a random sample of 35 patients with diabetes mellitus from among the patients attending his or her residency clinic. The inclusion criteria for patients of the AOA-CAP diabetes database were a diagnosis of diabetes mellitus for at least 1 year and at least 2 diabetes-related visits to the residency clinic in the previous 12 months. Patients who were younger than 18 years and who were not taking any medication to control their diabetes mellitus were excluded from the database. In the present study, we assumed that program directors oversaw the data entry by residents. 
The following process of care data (ie, number of patients who received each process of care) were collected for patients: an annual foot examination, an annual referral for an ophthalmologic examination, an annual microalbuminuria screening, the use of ACE inhibitors or ARBs if albuminuria is present, the use of ACE inhibitors or ARBs if hypertension is present, and the completion of an HbA1c test and LDL-C test in the previous year. A process of care composite score was computed by dividing the number of patients who received all indicated processes of care by the total number of patients with diabetes mellitus. 
Figure 2.
The process of care and intermediate outcome variables recorded by osteopathic family medicine residency programs using the diabetes registry of the American Osteopathic Association's Clinical Assessment Program (AOA-CAP) and the numerators and denominators used to calculate composite scores. Abbreviations: ACE, angiotensin-converting enzyme inhibitor; ARB, angiotensin receptor blocker; DM, diabetes mellitus; HbA1c, glycosylated hemoglobin; LDL-C, low-density lipoprotein cholesterol.
Figure 2.
The process of care and intermediate outcome variables recorded by osteopathic family medicine residency programs using the diabetes registry of the American Osteopathic Association's Clinical Assessment Program (AOA-CAP) and the numerators and denominators used to calculate composite scores. Abbreviations: ACE, angiotensin-converting enzyme inhibitor; ARB, angiotensin receptor blocker; DM, diabetes mellitus; HbA1c, glycosylated hemoglobin; LDL-C, low-density lipoprotein cholesterol.
Three outcome variables (blood pressure, HbA1c, and LDL-C) were measured for each patient. An outcomes composite score was computed by dividing the number of patients with normal levels of blood pressure, HbA1c, and LDL-C by the total number of patients with diabetes mellitus. The measured process of care and outcome variables, with numerators and denominators used to calculate composite scores, are listed in Figure 2. 
The data dictionary provided the residents with specific procedures and definitions to ensure the uniformity and reliability of abstracted data. Each data dictionary included a general description of clinical conditions in the measurement set; information necessary to precisely select the medical records to be abstracted; and detailed background, definitions, and instructions regarding each data element to be abstracted. Data were abstracted from the medical records retrospectively. Some data were abstracted onto paper forms and later transferred to an online data entry tool for transmission to the AOA-CAP for analysis. Other data were abstracted directly from medical records to the data entry tool. 
To determine the effect of data contributions in multiple cycles to the AOA-CAP diabetes registry, we compared the performance of all residency programs that contributed to the database for the first time between July 1, 2005, and December 31, 2007 (ie, first-time programs), to the performance of those residency programs that contributed to the database during that cycle and the previous cycle of July 1, 2003, to June 30, 2005 (ie, repeat programs). 
Statistical analyses included the χ2 test for dichotomous variables and the t test for continuous, multiple variable comparisons, which were carried out using logistic regression. All analyses were completed using SAS statistical software (version 9.1; SAS Institute Inc, Cary, North Carolina). The Ohio University's Institutional Review Board for Human Subjects Research approved all aspects of this project. The level of statistical significance was defined as α=.05. 
Results
Fifty-two osteopathic family medicine residency programs, comprising 23 first-time programs [44.2%] and 29 repeat programs [55.8%], entered data into the AOA-CAP diabetes registry during the 2005-2007 cycle for a total of 2568 patient cases. First-time programs had 992 patient cases (38.6%), and repeat programs had 1576 patient cases (61.4%). 
The mean age of patients was 55.2 years. Residency programs that contributed data in 1 cycle had patients with a statistically significant higher mean age than did programs that contributed data in both cycles (56.2 years vs 54.5 years, respectively; P=.002). The 2 program groups were statistically similar with respect to percent of patients who were men, percent of patients who were African American, and percent of patients who had commercial insurance. Statistically significant differences were detected between the program groups in other ethnicities and in other insurance types. Data for these comparisons are presented in Table 1. 
Table 1.
Characteristics of Patients With Diabetes Mellitus in Osteopathic Family Practice Residency Programs Contributing Data to the AOA-CAP (N=2568) *

Characteristic

All Programs

First-Time Programs

Repeat Programs

P Value
Mean Age, y55.1656.2154.51.002
Sex, Men 1019 (39.69) 394 (39.74) 625 (39.66) .9679
Ethnicity
□ African American 476 (18.52) 181 (18.20) 295 (18.72) .7423
□ Hispanic307 (11.97)102 (10.25)203 (12.88).0725
□ White 1304 (50.76) 543 (54.70) 761 (48.29) .0016
□ Other or unavailable482 (18.75)164 (16.58)317 (20.11).0257
Using Insulin 737 (28.69) 331 (33.37) 406 (25.76) <.001
Insurance Type
□ Commercial 576 (22.42) 205 (20.63) 371 (23.54) .085
□ Medicaid617 (24.02)163 (16.48)453 (28.74)<.001
□ Medicare 650 (25.30) 312 (31.45) 338 (21.45) <.001
□ Self-pay225 (8.77)29 (2.93)196 (12.44)<.001
□ Other or unavailable 331 (12.90) 206 (20.73) 126 (7.99) <.001
 Abbreviation: AOA-CAP, American Osteopathic Association's Clinical Assessment Program.
 *All data presented as No. (%) unless otherwise indicated. First-time programs were those contributing data in the 2005-2007 cycle only. Repeat programs were those contributing data in both the 2003-2005 and 2005-2007 cycles.
Table 1.
Characteristics of Patients With Diabetes Mellitus in Osteopathic Family Practice Residency Programs Contributing Data to the AOA-CAP (N=2568) *

Characteristic

All Programs

First-Time Programs

Repeat Programs

P Value
Mean Age, y55.1656.2154.51.002
Sex, Men 1019 (39.69) 394 (39.74) 625 (39.66) .9679
Ethnicity
□ African American 476 (18.52) 181 (18.20) 295 (18.72) .7423
□ Hispanic307 (11.97)102 (10.25)203 (12.88).0725
□ White 1304 (50.76) 543 (54.70) 761 (48.29) .0016
□ Other or unavailable482 (18.75)164 (16.58)317 (20.11).0257
Using Insulin 737 (28.69) 331 (33.37) 406 (25.76) <.001
Insurance Type
□ Commercial 576 (22.42) 205 (20.63) 371 (23.54) .085
□ Medicaid617 (24.02)163 (16.48)453 (28.74)<.001
□ Medicare 650 (25.30) 312 (31.45) 338 (21.45) <.001
□ Self-pay225 (8.77)29 (2.93)196 (12.44)<.001
□ Other or unavailable 331 (12.90) 206 (20.73) 126 (7.99) <.001
 Abbreviation: AOA-CAP, American Osteopathic Association's Clinical Assessment Program.
 *All data presented as No. (%) unless otherwise indicated. First-time programs were those contributing data in the 2005-2007 cycle only. Repeat programs were those contributing data in both the 2003-2005 and 2005-2007 cycles.
×
The results of the analysis of processes and outcomes of care are displayed in Table 2. For process of care measures, repeat programs achieved statistically significant higher rates of prescribing an ACE inhibitor and ARB to patients with albuminuria compared to first-time programs (P=.0087). Repeat programs also had a statistically significant better result on the composite measure of patients receiving all indicated care compared to first-time programs (P=.0023). There was no statistically significant difference between program groups in any of the other individual process of care measures. Also, there was no statistically significant difference between program groups in any of the intermediate outcome measures (Table 2). 
Table 2.
Osteopathic Family Practice Residency Programs Achieving Processes of Care and Intermediate Outcomes, According to Data Contributed to the AOA-CAP (N=2568) *

Variable

First-Time Programs

Repeat Programs

P Value
Processes of Care
□ Individual
– Foot examination557 (56.12)930 (59.01).1486
– Ophthalmologic examination referral 577 (58.14) 929 (58.95) .6862
– Microalbuminuria screen671 (67.64)1063 (67.45).9183
– ACE or ARB for albuminuria 192 (81.87) 336 (86.30) .0087
– ACE or ARB for hypertension619 (83.47)1019 (84.74).6697
– HbA1c test 925 (93.23) 1470 (93.27) .9619
– LDL-C test858 (86.45)1389 (88.13).2098
□ Composite 262 (26.39) 505 (32.04) .0023
Intermediate Outcomes
□ Individual
– Blood pressure,
    <80/130 mm Hg 366 (36.91) 595 (37.75) .6658
– HbA1c, <7%410 (41.35)680 (43.15).3714
– LDL-C, <100 mg/dL 387 (39.03) 657 (41.69) .1822
□ Composite99 (10.01)151 (9.58).7215
 Abbreviations: ACE, angiotensin-converting enzyme inhibitor; AOA-CAP, American Osteopathic Association's Clinical Assessment Program; ARB, angiotensin receptor blocker; HbA1c, glycosylated hemoglobin; LDL-C, low-density lipoprotein cholesterol.
 *All data presented as No. (%) of patient cases. First-time programs (992 patient cases) were those contributing data in the 2005-2007 cycle only. Repeat programs (1576 patient cases) were those contributing data in both the 2003-2005 and 2005-2007 cycles.
Table 2.
Osteopathic Family Practice Residency Programs Achieving Processes of Care and Intermediate Outcomes, According to Data Contributed to the AOA-CAP (N=2568) *

Variable

First-Time Programs

Repeat Programs

P Value
Processes of Care
□ Individual
– Foot examination557 (56.12)930 (59.01).1486
– Ophthalmologic examination referral 577 (58.14) 929 (58.95) .6862
– Microalbuminuria screen671 (67.64)1063 (67.45).9183
– ACE or ARB for albuminuria 192 (81.87) 336 (86.30) .0087
– ACE or ARB for hypertension619 (83.47)1019 (84.74).6697
– HbA1c test 925 (93.23) 1470 (93.27) .9619
– LDL-C test858 (86.45)1389 (88.13).2098
□ Composite 262 (26.39) 505 (32.04) .0023
Intermediate Outcomes
□ Individual
– Blood pressure,
    <80/130 mm Hg 366 (36.91) 595 (37.75) .6658
– HbA1c, <7%410 (41.35)680 (43.15).3714
– LDL-C, <100 mg/dL 387 (39.03) 657 (41.69) .1822
□ Composite99 (10.01)151 (9.58).7215
 Abbreviations: ACE, angiotensin-converting enzyme inhibitor; AOA-CAP, American Osteopathic Association's Clinical Assessment Program; ARB, angiotensin receptor blocker; HbA1c, glycosylated hemoglobin; LDL-C, low-density lipoprotein cholesterol.
 *All data presented as No. (%) of patient cases. First-time programs (992 patient cases) were those contributing data in the 2005-2007 cycle only. Repeat programs (1576 patient cases) were those contributing data in both the 2003-2005 and 2005-2007 cycles.
×
Because of potential bias introduced by differences in the demographic characteristics of the 2 program groups (Table 1), logistic regression analysis was performed using the process composite as the dependant variable. In this analysis, only other or unknown ethnicity and other or unknown insurance type predicted a statistically significant difference between the two groups (Table 3). No other factor explained other differences. 
Table 3.
Logistic Regression Analysis of Characteristics of Patients With Diabetes Mellitus in Osteopathic Family Medicine Residency Programs Contributing Data to the AOA-CAP for Residencies (N=2568)

Characteristic

Estimated Odds Ratio

95% Confidence Interval
Age1.0050.998-1.013
Sex, Men 1.096 0.918-1.307
Ethnicity
□ African American 1.331 0.999-1.773
□ Hispanic2.1641.586-2.952
□ White 1.025 0.802-1.309
Using Insulin1.2941.072-1.563
Insurance Type
□ Commercial1.0310.785-1.353
□ Medicaid 0.966 0.735-1.269
□ Medicare0.8860.665-1.182
□ Self-pay 0.647 0.443-0.944
Contributing Data in Both Cycles*1.3721.139-1.653
 Abbreviation: AOA-CAP, American Osteopathic Association's Clinical Assessment Program.
 *Residency programs contributing data in both the 2003-2005 and 2005-2007 cycles.
Table 3.
Logistic Regression Analysis of Characteristics of Patients With Diabetes Mellitus in Osteopathic Family Medicine Residency Programs Contributing Data to the AOA-CAP for Residencies (N=2568)

Characteristic

Estimated Odds Ratio

95% Confidence Interval
Age1.0050.998-1.013
Sex, Men 1.096 0.918-1.307
Ethnicity
□ African American 1.331 0.999-1.773
□ Hispanic2.1641.586-2.952
□ White 1.025 0.802-1.309
Using Insulin1.2941.072-1.563
Insurance Type
□ Commercial1.0310.785-1.353
□ Medicaid 0.966 0.735-1.269
□ Medicare0.8860.665-1.182
□ Self-pay 0.647 0.443-0.944
Contributing Data in Both Cycles*1.3721.139-1.653
 Abbreviation: AOA-CAP, American Osteopathic Association's Clinical Assessment Program.
 *Residency programs contributing data in both the 2003-2005 and 2005-2007 cycles.
×
Even after adjusting for all listed demographic factors, residency programs that contributed data in both cycles had a statistically significant higher rate of performance on the process composite measure than did residency programs that contributed data in only 1 cycle (37% adjusted [P<.001] vs 21% unadjusted). The process and outcome composites were developed as a summary measure of performance at the program level on the 2 classes of indicators, as defined in Figure 2. 
Comment
The present study demonstrated a statistically significant improvement in the composite process of care measure in those residency programs that had participated in the AOA-CAP for 2 cycles. This improvement was largely the result of prescribing ACE inhibitors and ARBs to patients with albuminuria. Although these results indicate limited benefit to residency programs from repeat participation in the AOA-CAP, we suspect that with continued use, program performance could improve on additional process and outcome measures. Models of rapid improvement in program quality from 1 cycle to another allow for sequential systematic change while measuring effects. 
As performance review is becoming the standard for evaluating and paying physicians, we must find ways to systematically evaluate our practices and continue to strive to improve our practices. According to a 2008 study,15 osteopathic physicians have been reluctant to embrace practice-based evaluation. The core competencies of quality improvement that have been built into the osteopathic primary care curriculum are necessary to provide future osteopathic primary care physicians with the tools needed for success and for administering the best possible care to patients. 
It is now mandatory that all residents in osteopathic family medicine and general internal medicine residencies complete an AOA-CAP project annually. This requirement was suggested at the time of data entry for the present study, but it was not enforced, and only a minority of residency programs initially entered data. As more programs participate in the AOA-CAP for Residencies and as they become more familiar with using the data-entry tool, we hope that program participation and quality of data will increase. 
By completing a medical chart review, entering the data into the AOA-CAP registry, reviewing the report the AOA generates, and then comparing the results to national data, osteopathic residents are learning how to compare their performance to national performance standards. If used repeatedly, the AOA-CAP could positively impact practice. By participating in the AOA-CAP, residents can learn the standards of care, analytically review their processes of care, identify ways to improve their care of patients, and ultimately relate their processes of care to patient outcomes. 
The AOA-CAP is unique in being Web based and extremely simple to use compared to other assessments, such as those of the NCQA. Furthermore, the AOA-CAP for Residencies and AOA-CAP for Physicians are free to osteopathic residents and to practicing osteopathic physicians who are AOA members. With this AOA program, participants can obtain personal reports describing how their performance compares to national reporting standards. These reports allow osteopathic residents and physicians to measure their performances without the pressure of a payer's review. 
Using the AOA-CAP sets the stage for practice-based interventions that can improve processes of care and documentation. The study investigators believe that this program is an invaluable tool in training osteopathic physicians to meet the competencies of systems-based practice and practice-based learning while introducing osteopathic physicians to incoming performance-measurement programs, which will likely be ubiquitous in the near future. 
Osteopathic residents must become accustomed to being “graded,” and they will need to learn how to analyze their practices to determine what patient-level changes are needed to provide the best quality of care. The AOA-CAP reports help residents and residency program directors to improve professional development and to compare program results to a national sample—introducing some of the elements that are now part of the AOA's required core competencies, including systems-based practice and practice-based learning.16 
The AOA-CAP has been analyzed in other studies,11,17,18 including research11 conducted by the present authors. However, these studies have been limited in scope, and given the richness of the AOA-CAP dataset, additional data exploration is warranted. 
Limitations
A weakness of the present study is that the residents self reported and self abstracted data. Residents abstracted data either for themselves or for their programs, so it is possible that the data were not abstracted as rigorously as they could have been by outsiders. We believe that this possibility is unlikely because the “scores” we calculated were similar to those reported in reviews that were computed by outside reviewers.19 In addition, the fact that there was no consequence for not achieving certain goal levels likely limited any incentive residents may have had to hedge the information they attained. 
Resident outcome achievement improved over time, even in those residency programs that inputted data only in the second (2005-2007) cycle. This outcome likely reflects the acceptance by residency programs of tighter standards of care for patients with diabetes mellitus. 
Another possible weakness of this study is the lack of supervision of residents in how they randomly selected their patient cases. The cases may have been selected as a convenience sample, or they may have been selected by some other method. The lack of supervision may have created a selection bias by residents. 
We cannot confirm that the residency programs that entered data into the AOA-CAP registry were similar to those that did not participate. This too is a possible source of bias that may limit the generalizability of the study results. 
Conclusion
In the present study, repeated use of the AOA-CAP modestly improved residency programs' processes of care for patients with diabetes mellitus. However, Web-based registries such as the AOA-CAP are currently underutilized. More widespread use of the AOA-CAP has the potential to not only better prepare osteopathic physicians for practice but also better demonstrate the AOA Core Competencies of systems-based practice and practice-based learning. 
 Financial Disclosures: Funding for this project came from the Osteopathic Heritage Foundation in Columbus, Ohio. Analytical support was provided by the American Osteopathic Association.
 
 Conflicts of Interest: Dr Snow and Ms McGill serve on the Steering Committee of the American Osteopathic Association's Clinical Assessment Program. Dr Shubrook reports no conflicts of interest.
 
 Editor's Note: The American Osteopathic Association's Clinical Assessment Program (AOA-CAP) for Residencies is 1 of 3 existing clinical assessment programs. The AOA-CAP for Residencies provides a mechanism for osteopathic residency programs to measure and improve the quality of patient care they provide to their patients. Similarly, the AOA-CAP for Physicians, another program, measures phsycians' current clinical practices and compares their outcomes measures to those of their peers and national measures.
 
 The AOA-CAP for Physician Quality Reporting Initiative (PQRI) is qualified by the Centers for Medicare & Medicaid Services (CMS) as an incentive payment program. It provide physicians with an opportunity to receive enhanced payment from the CMS and to understand patient care using an evidence-based process and outcome measures.
 
Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; March 2001. http://iom.edu/CMS/8089/5432/27184.aspx
About the crisis. Chronic diseases are creating a national health care crisis. Partnership to Fight Chronic Disease Web site. http://www.fightchronicdisease.org/issues/about.cfm. Accessed December 28, 2010.
Chronic diseases and health promotion. Centers for Disease Control and Prevention Web site. http://www.cdc.gov/nccdphp/overview.htm. Accessed December 28, 2010.
Lohr K, Schroeder S. A strategy for quality assurance in Medicare. N Engl J Med. 1990;332(10):707-712.
Pay-for-performance incentives for physicians are common (evidence for practice). AORN Journal. April1 , 2007.
Rosenthal MB, Landon BE, Normand ST, Frank RG, Epstein AM. Pay for performance in commercial HMOs. N Engl J Med. 2006;355(18):1895-1902.
McFarlane SI, Jacober SJ, Winer N, et al. Control of cardiovascular risk factors in patients with diabetes and hypertension at urban academic medical centers. Diabetes Care. 2002;25(4):718-723.
Grant RW, Buse JB, Meigs JB; University HealthSystem Consortium (UHC) Diabetes Benchmarking Project Team. Quality of diabetes care in U.S. academic medical centers: low rates of medical regimen change. Diabetes Care. 2005;28(2):337-342.
Rodondi N, Peng T, Karter AJ, et al. Therapy modifications in response to poorly controlled hypertension, dylipidemia, and diabetes mellitus. Ann Intern Med. 2006;144(7):475-484.
Burke, WJ, Nelson KM, Clausin-Glaser T, Snow RJ. Use of a hypertension registry to identify patients at high risk for cardiovascular events caused by metabolic syndrome. Osteopath Fam Physician. 2010;2(5):124-130.
Shubrook JH Jr, Snow RJ, McGill SL, Brannan GD. “All-or-none” (bundled) process and outcomes indicators of diabetes care. Am J Manag Care. 2010;16(1):25-32.
HEDIS & quality measurement. National Committee for Quality Assurance Web site. http://www.ncqa.org/tabid/59/Default.aspx. Accessed December 28, 2010.
HealthyPeople.gov. US Department of Health and Human Services, Healthy People Web site. http://www.healthypeople.gov. Accessed December 28, 2010.
American Diabetes Association. Standards of medical care in diabetes—2007. Diabetes Care. 2007;30(suppl 1):S1-S41. http://care.diabetesjournals.org/content/30/suppl_1/S4.full.pdf. Accessed December 28, 2010.
Locke RG, Srinivasan M. Attitudes toward pay-for-performance initiative among primary care osteopathic physicians in small group practices. J Am Osteopath Assoc. 2008;108(1):21-24. http://www.jaoa.org/cgi/reprint/108/1/21. Accessed December 6, 2010.
American Osteopathic Association. The Basic Documents for Postdoctoral Training. Rev BOT 2/2010. Chicago, IL: American Osteopathic Association; 2010. http://www.osteopathic.org/inside-aoa/accreditation/postdoctoral-training-approval/postdoctoral-training-standards/Documents/aoa-basic-document-for-postdoctoral-training.pdf. Accessed December 6, 2010.
Kalb JL, Marazon DJ, Snow RJ. Potential barriers to control of blood pressure. J Am Osteopath Assoc.. (2002). ;102(4):209-213.
Agency for Healthcare Research and Quality. Advances in Patient Safety: From Research to Implementation. Vols 1-4. Rockville, MD: Agency for Healthcare Research and Quality; February 2005. AHRQ Publication No. 050021 (1-4). http://www.ahrq.gov/qual/advances/. Accessed December 6, 2010.
Saaddine JB, Cadwell B, Gregg EW, et al. Improvements in diabetes processes of care and intermediate outcomes: United States, 1988-2002. Ann Intern Med.. (2006). ;144(7):465-474.
Figure 1.
Diabetes Measurement Abstraction Tool used to record measures of processes of care and intermediate outcomes for osteopathic family medicine residency programs using the diabetes registry of the American Osteopathic Association's Clinical Assessment Program (AOA-CAP). This tool was altered for graphic enhancement only. Abbreviations: ACE, angiotensin-converting enzyme; ARB, angiotensin receptor blocker; BMI, body mass index; BP Dia, diastolic blood pressure; BP Sys, systolic blood pressure; DM, diabetes mellitus; Edu, education; HbA1c, glycosylated hemoglobin; HTN, hypertension; ID, resident identification; Intol, intolerance; LDL, low-density lipoprotein; Rec, recovery; Wt, weight.
Figure 1.
Diabetes Measurement Abstraction Tool used to record measures of processes of care and intermediate outcomes for osteopathic family medicine residency programs using the diabetes registry of the American Osteopathic Association's Clinical Assessment Program (AOA-CAP). This tool was altered for graphic enhancement only. Abbreviations: ACE, angiotensin-converting enzyme; ARB, angiotensin receptor blocker; BMI, body mass index; BP Dia, diastolic blood pressure; BP Sys, systolic blood pressure; DM, diabetes mellitus; Edu, education; HbA1c, glycosylated hemoglobin; HTN, hypertension; ID, resident identification; Intol, intolerance; LDL, low-density lipoprotein; Rec, recovery; Wt, weight.
Figure 2.
The process of care and intermediate outcome variables recorded by osteopathic family medicine residency programs using the diabetes registry of the American Osteopathic Association's Clinical Assessment Program (AOA-CAP) and the numerators and denominators used to calculate composite scores. Abbreviations: ACE, angiotensin-converting enzyme inhibitor; ARB, angiotensin receptor blocker; DM, diabetes mellitus; HbA1c, glycosylated hemoglobin; LDL-C, low-density lipoprotein cholesterol.
Figure 2.
The process of care and intermediate outcome variables recorded by osteopathic family medicine residency programs using the diabetes registry of the American Osteopathic Association's Clinical Assessment Program (AOA-CAP) and the numerators and denominators used to calculate composite scores. Abbreviations: ACE, angiotensin-converting enzyme inhibitor; ARB, angiotensin receptor blocker; DM, diabetes mellitus; HbA1c, glycosylated hemoglobin; LDL-C, low-density lipoprotein cholesterol.
Table 1.
Characteristics of Patients With Diabetes Mellitus in Osteopathic Family Practice Residency Programs Contributing Data to the AOA-CAP (N=2568) *

Characteristic

All Programs

First-Time Programs

Repeat Programs

P Value
Mean Age, y55.1656.2154.51.002
Sex, Men 1019 (39.69) 394 (39.74) 625 (39.66) .9679
Ethnicity
□ African American 476 (18.52) 181 (18.20) 295 (18.72) .7423
□ Hispanic307 (11.97)102 (10.25)203 (12.88).0725
□ White 1304 (50.76) 543 (54.70) 761 (48.29) .0016
□ Other or unavailable482 (18.75)164 (16.58)317 (20.11).0257
Using Insulin 737 (28.69) 331 (33.37) 406 (25.76) <.001
Insurance Type
□ Commercial 576 (22.42) 205 (20.63) 371 (23.54) .085
□ Medicaid617 (24.02)163 (16.48)453 (28.74)<.001
□ Medicare 650 (25.30) 312 (31.45) 338 (21.45) <.001
□ Self-pay225 (8.77)29 (2.93)196 (12.44)<.001
□ Other or unavailable 331 (12.90) 206 (20.73) 126 (7.99) <.001
 Abbreviation: AOA-CAP, American Osteopathic Association's Clinical Assessment Program.
 *All data presented as No. (%) unless otherwise indicated. First-time programs were those contributing data in the 2005-2007 cycle only. Repeat programs were those contributing data in both the 2003-2005 and 2005-2007 cycles.
Table 1.
Characteristics of Patients With Diabetes Mellitus in Osteopathic Family Practice Residency Programs Contributing Data to the AOA-CAP (N=2568) *

Characteristic

All Programs

First-Time Programs

Repeat Programs

P Value
Mean Age, y55.1656.2154.51.002
Sex, Men 1019 (39.69) 394 (39.74) 625 (39.66) .9679
Ethnicity
□ African American 476 (18.52) 181 (18.20) 295 (18.72) .7423
□ Hispanic307 (11.97)102 (10.25)203 (12.88).0725
□ White 1304 (50.76) 543 (54.70) 761 (48.29) .0016
□ Other or unavailable482 (18.75)164 (16.58)317 (20.11).0257
Using Insulin 737 (28.69) 331 (33.37) 406 (25.76) <.001
Insurance Type
□ Commercial 576 (22.42) 205 (20.63) 371 (23.54) .085
□ Medicaid617 (24.02)163 (16.48)453 (28.74)<.001
□ Medicare 650 (25.30) 312 (31.45) 338 (21.45) <.001
□ Self-pay225 (8.77)29 (2.93)196 (12.44)<.001
□ Other or unavailable 331 (12.90) 206 (20.73) 126 (7.99) <.001
 Abbreviation: AOA-CAP, American Osteopathic Association's Clinical Assessment Program.
 *All data presented as No. (%) unless otherwise indicated. First-time programs were those contributing data in the 2005-2007 cycle only. Repeat programs were those contributing data in both the 2003-2005 and 2005-2007 cycles.
×
Table 2.
Osteopathic Family Practice Residency Programs Achieving Processes of Care and Intermediate Outcomes, According to Data Contributed to the AOA-CAP (N=2568) *

Variable

First-Time Programs

Repeat Programs

P Value
Processes of Care
□ Individual
– Foot examination557 (56.12)930 (59.01).1486
– Ophthalmologic examination referral 577 (58.14) 929 (58.95) .6862
– Microalbuminuria screen671 (67.64)1063 (67.45).9183
– ACE or ARB for albuminuria 192 (81.87) 336 (86.30) .0087
– ACE or ARB for hypertension619 (83.47)1019 (84.74).6697
– HbA1c test 925 (93.23) 1470 (93.27) .9619
– LDL-C test858 (86.45)1389 (88.13).2098
□ Composite 262 (26.39) 505 (32.04) .0023
Intermediate Outcomes
□ Individual
– Blood pressure,
    <80/130 mm Hg 366 (36.91) 595 (37.75) .6658
– HbA1c, <7%410 (41.35)680 (43.15).3714
– LDL-C, <100 mg/dL 387 (39.03) 657 (41.69) .1822
□ Composite99 (10.01)151 (9.58).7215
 Abbreviations: ACE, angiotensin-converting enzyme inhibitor; AOA-CAP, American Osteopathic Association's Clinical Assessment Program; ARB, angiotensin receptor blocker; HbA1c, glycosylated hemoglobin; LDL-C, low-density lipoprotein cholesterol.
 *All data presented as No. (%) of patient cases. First-time programs (992 patient cases) were those contributing data in the 2005-2007 cycle only. Repeat programs (1576 patient cases) were those contributing data in both the 2003-2005 and 2005-2007 cycles.
Table 2.
Osteopathic Family Practice Residency Programs Achieving Processes of Care and Intermediate Outcomes, According to Data Contributed to the AOA-CAP (N=2568) *

Variable

First-Time Programs

Repeat Programs

P Value
Processes of Care
□ Individual
– Foot examination557 (56.12)930 (59.01).1486
– Ophthalmologic examination referral 577 (58.14) 929 (58.95) .6862
– Microalbuminuria screen671 (67.64)1063 (67.45).9183
– ACE or ARB for albuminuria 192 (81.87) 336 (86.30) .0087
– ACE or ARB for hypertension619 (83.47)1019 (84.74).6697
– HbA1c test 925 (93.23) 1470 (93.27) .9619
– LDL-C test858 (86.45)1389 (88.13).2098
□ Composite 262 (26.39) 505 (32.04) .0023
Intermediate Outcomes
□ Individual
– Blood pressure,
    <80/130 mm Hg 366 (36.91) 595 (37.75) .6658
– HbA1c, <7%410 (41.35)680 (43.15).3714
– LDL-C, <100 mg/dL 387 (39.03) 657 (41.69) .1822
□ Composite99 (10.01)151 (9.58).7215
 Abbreviations: ACE, angiotensin-converting enzyme inhibitor; AOA-CAP, American Osteopathic Association's Clinical Assessment Program; ARB, angiotensin receptor blocker; HbA1c, glycosylated hemoglobin; LDL-C, low-density lipoprotein cholesterol.
 *All data presented as No. (%) of patient cases. First-time programs (992 patient cases) were those contributing data in the 2005-2007 cycle only. Repeat programs (1576 patient cases) were those contributing data in both the 2003-2005 and 2005-2007 cycles.
×
Table 3.
Logistic Regression Analysis of Characteristics of Patients With Diabetes Mellitus in Osteopathic Family Medicine Residency Programs Contributing Data to the AOA-CAP for Residencies (N=2568)

Characteristic

Estimated Odds Ratio

95% Confidence Interval
Age1.0050.998-1.013
Sex, Men 1.096 0.918-1.307
Ethnicity
□ African American 1.331 0.999-1.773
□ Hispanic2.1641.586-2.952
□ White 1.025 0.802-1.309
Using Insulin1.2941.072-1.563
Insurance Type
□ Commercial1.0310.785-1.353
□ Medicaid 0.966 0.735-1.269
□ Medicare0.8860.665-1.182
□ Self-pay 0.647 0.443-0.944
Contributing Data in Both Cycles*1.3721.139-1.653
 Abbreviation: AOA-CAP, American Osteopathic Association's Clinical Assessment Program.
 *Residency programs contributing data in both the 2003-2005 and 2005-2007 cycles.
Table 3.
Logistic Regression Analysis of Characteristics of Patients With Diabetes Mellitus in Osteopathic Family Medicine Residency Programs Contributing Data to the AOA-CAP for Residencies (N=2568)

Characteristic

Estimated Odds Ratio

95% Confidence Interval
Age1.0050.998-1.013
Sex, Men 1.096 0.918-1.307
Ethnicity
□ African American 1.331 0.999-1.773
□ Hispanic2.1641.586-2.952
□ White 1.025 0.802-1.309
Using Insulin1.2941.072-1.563
Insurance Type
□ Commercial1.0310.785-1.353
□ Medicaid 0.966 0.735-1.269
□ Medicare0.8860.665-1.182
□ Self-pay 0.647 0.443-0.944
Contributing Data in Both Cycles*1.3721.139-1.653
 Abbreviation: AOA-CAP, American Osteopathic Association's Clinical Assessment Program.
 *Residency programs contributing data in both the 2003-2005 and 2005-2007 cycles.
×