Free
Medical Education  |   April 2014
Citation and Correction of Deficiencies in Osteopathic Graduate Medical Education Programs: Opportunities for Improvement
Author Notes
  • Address correspondence to Alisa Pierson, MS, American Osteopathic Association, Department of Education, 142 E Ontario St, Chicago, IL 60611-2864. E-mail: apierson@osteopathic.org  
Article Information
Medical Education / Graduate Medical Education
Medical Education   |   April 2014
Citation and Correction of Deficiencies in Osteopathic Graduate Medical Education Programs: Opportunities for Improvement
The Journal of the American Osteopathic Association, April 2014, Vol. 114, 290-294. doi:10.7556/jaoa.2014.055
The Journal of the American Osteopathic Association, April 2014, Vol. 114, 290-294. doi:10.7556/jaoa.2014.055
Abstract

Since 2008, the American Osteopathic Association has gathered data on osteopathic graduate medical education program compliance with the association's correction of deficiencies processes. The purpose of the current article is to look at those data to discover trends and identify patterns and areas for improvement for osteopathic graduate medical education. In addition to providing quantitative data, the author also uses qualitative data to explain why corrective action plans are recommended for denial.

In 1947, the American Osteopathic Association (AOA) began granting approval to osteopathic graduate medical education (OGME) programs.1 As a means to ensure quality education in all OGME programs, the AOA sends site reviewers (previously called inspectors) to conduct a site review that is designed to uncover any existing deficiencies (ie, standards that have not been met). The resulting report is used to determine a period of continuing approval for an OGME program. 
Since 2008, the AOA has focused on the following information: (1) programs cited with at least 1 deficiency, (2) programs that subsequently submitted a corrective action plan to the AOA, (3) if such plans were approved by the specialty college, and (4) which plans were implemented. In the present article, I provide a summary of the corrective action process and the AOA's OGME data, identify trends and patterns, and suggest areas for improvement. This information may provide OGME leaders with a way to improve the quality of their programs, which in turn should result in better-prepared osteopathic physicians in the future. 
Methods
Any discussion of the corrective action process must be grounded in an understanding of the AOA's continuing approval process for internships, residencies, and fellowships, which is very similar to the accreditation process used by the Accreditation Council of Graduate Medical Education. It is important for OGME programs to have AOA approval to ensure quality and uniformity of training among a large number of programs. For example, 2 osteopathic physicians graduating from the same college of osteopathic medicine can enter 2 different family medicine residencies at 2 different hospitals and be certain that they will both receive 4 weeks of training in women's health during the first OGME year (assuming the programs are meeting the standards outlined in the AOA's Basic Standards for Residency Training in Osteopathic Family Medicine and Manipulative Treatment).2 This document outlines sufficient training procedures for specialty and subspecialty, whereas AOA approval exists to provide oversight and verify that standards are met. 
The AOA approval process for an existing program begins with a program site review (previously called an inspection) conducted by a site reviewer using a crosswalk. A crosswalk is a document with checkboxes that lists (1) all the standards to be reviewed and (2) the amount of points of each deficiency, enabling a final score to be tabulated for the program. When the site reviewer notices a standard that has not been met, he or she describes the deficiency and references the appropriate standard listed in the crosswalk. The site reviewer then issues a final report that outlines any areas in which the program does not meet AOA OGME standards.3 Each crosswalk is customized for a given specialty or subspecialty and is designed to include all relevant standards that ensure trainees receive sufficient immersion and education in their field. The Council on Postdoctoral Training oversees approval of standards and crosswalks before the final approval by the AOA Board of Trustees. Current AOA-approved standards and crosswalks can be found online at http://www.osteopathic.org/inside-aoa/accreditation/postdoctoral-training-approval/postdoctoral-training-Standards/. 
The specialty college education committee reviews the final report, and then the committee submits a resolution recommending the program's next term of continuing approval to the AOA's Program and Trainee Review Council (PTRC). The resolution details the program's basic information, any deficiencies, the site reviewer's verbatim description of each deficiency, and a recommended length of approval for the program (the amount of time until the next site review). Three weeks before the PTRC meeting, the program's Osteopathic Postdoctoral Training Institution (OPTI)—which is the academic sponsor for the program—receives a copy of the continuing approval resolution for their program. If the OPTI notices that any deficiency was cited in error, the OPTI and the program are allowed to submit evidence (ie, an “error in fact” request) to the specialty college and PTRC for review. The PTRC will determine if the deficiency was indeed cited in error and should be removed from the final recommendation. At each PTRC meeting, the AOA receives 3 to 5 “error in fact” requests from OPTIs or OGME programs. The PTRC makes the final determination on any deficiencies and the final continuing approval length. 
Within 2 weeks of the PTRC meeting, a letter is sent to the OGME program detailing the final determination, any applicable deficiencies, and the site reviewer's comments on exactly how the standard was not met. Any program cited with deficiencies is required to submit a corrective action plan explaining how it will meet the standards.4 This plan must first be submitted to the program's OPTI within 45 days of receiving the PTRC letter. The OPTI then has 30 days to review the plan. If the plan is approved by the OPTI, the plan is then forwarded to the AOA. The AOA performs an administrative review of the plan and then forwards it to the specialty college for review and final approval. The specialty college then notifies the AOA of its decision, and the AOA writes the final letter to the OGME program informing it of the specialty college's approval or denial. When the program receives the approval letter, it has 9 months to submit evidence of implementation of the corrective action plan to their OPTI. The OPTI then approves that evidence and informs the AOA. The process for a given program is then considered closed as soon as the AOA receives the evidence of approval from the OPTI. The Figure summarizes the corrective action process described above. 
Figure.
The osteopathic graduate medical education (OGME) program evaluation and corrective action process. The American Osteopathic Association (AOA) tracks all steps of the process. Abbreviations: OPTI, osteopathic postdoctoral training institution; PTRC, Program Training and Review Council; SC, specialty college.
Figure.
The osteopathic graduate medical education (OGME) program evaluation and corrective action process. The American Osteopathic Association (AOA) tracks all steps of the process. Abbreviations: OPTI, osteopathic postdoctoral training institution; PTRC, Program Training and Review Council; SC, specialty college.
The AOA tracks all steps involved in the approval or denial process. It does not, however, track programs for which deficiencies have not been cited. 
If a program was cited with deficiencies during PTRC review, an AOA staff member enters the program's name into an Excel spreadsheet (Microsoft Corporation), along with the following data: specialty area, specialty college, OPTI name, dates (reviewed by PTRC, plan received by AOA, forwarded, approved, verified by OPTI), and additional comments. The spreadsheet documented 772 program records at the time of this writing. The data were divided by year to determine if there were any patterns or inferences that could be made from year-to-year changes in the data and changes in postdoctoral training policies. The AOA tracks all programs that have received continuing approval at the PTRC level in a given year. Therefore, it was simple to find the population size (N) for each year. This tracking process made it possible to compare data on program approval, citation of deficiencies, compliance, and a specialty college's denial of an OGME program. 
A z score was used to compare different years and determine if the difference between 2 years was statistically significant. Differences of P<.05 were considered statistically significant. 
Results
The Table shows 2009 through 2013 data related to OGME program approval, citation of deficiencies, and compliance. The years 2009 and 2010 show the lowest percentage of programs cited with 1 or more deficiencies. Programs cited with deficiencies then increase steadily from 2011 to 2013. In 2011, uniform standardization of site reviews was implemented and crosswalks were launched. Before 2011, site reviews used the Basic Standards2 as a reference and were composed of a narrative report and a variety of different specialty-specific workbooks and worksheets. The goals of creating uniform standardization of crosswalks were to increase the level of objectivity in site review and to make deficiencies clearer to the site reviewer and any subsequent readers of the report. The data suggest that uniform standardization and consistent use of crosswalks has improved site reviewers' abilities to catch deficiencies. In 2010, 59 of 157 programs (38%) needed a corrective action plan; the following year, the number was 75 of 126 (60%) (z score=−3.6745; P<.001). 
Table.
Rates of Approval, Deficiency Citations, Corrective Action Plans, and Denial for Osteopathic Graduate Medical Education Programs, 2009-2013
Year
Description 2009 2010 2011 2012 2013 Total Average
Programs receiving continuing approval, No. 210 157 126 132 147 772 154
Programs with ⩾1 deficiency, No. (%) 96 (46) 59 (38) 75 (60) 68 (52) 107 (73) 405 (52) 81 (54)
Corrective action plans received, No. (%) 48 (50) 57 (97) 71 (95) 58 (85) 54 (NA)a 234 (NA)a 58 (82)
Plans recommended for denial by specialty college, No. 3 3 5 5 1 17 3
  a Percentage not applicable. Programs cited by the October 2013 Program and Trainee Review Council did not have to submit a corrective action plan until January 2014. However, not all programs had submitted by the time of publication.
Table.
Rates of Approval, Deficiency Citations, Corrective Action Plans, and Denial for Osteopathic Graduate Medical Education Programs, 2009-2013
Year
Description 2009 2010 2011 2012 2013 Total Average
Programs receiving continuing approval, No. 210 157 126 132 147 772 154
Programs with ⩾1 deficiency, No. (%) 96 (46) 59 (38) 75 (60) 68 (52) 107 (73) 405 (52) 81 (54)
Corrective action plans received, No. (%) 48 (50) 57 (97) 71 (95) 58 (85) 54 (NA)a 234 (NA)a 58 (82)
Plans recommended for denial by specialty college, No. 3 3 5 5 1 17 3
  a Percentage not applicable. Programs cited by the October 2013 Program and Trainee Review Council did not have to submit a corrective action plan until January 2014. However, not all programs had submitted by the time of publication.
×
The AOA implemented another aspect of the uniform standardization process in 2013: professional site reviewers were enlisted to conduct the majority of reviews. (In past years, the task was entrusted to volunteers.) As a result of this change, 2013 showed the highest percentage of programs cited with deficiencies. In 2012, there were 68 of 132 programs (52%) that required a corrective action plan. That proportion increased in 2013 to 107 of 147 programs (73%), a difference that was statistically significant (z score=−3.6691; P<.001). 
The percentage of programs complying with the corrective action process has also improved since 2009. In 2008, the AOA started to track submission of corrective action plans, and in 2009 the AOA informed specialty colleges, OGME programs, and OPTIs that it was tracking compliance with the corrective action process. Also in 2009, the PTRC began receiving reports of compliance with the corrective action process. (Before 2008, compliance was neither tracked nor reported.) In 2009, the AOA also began regularly contacting OPTIs if 1 of its OGME programs failed to submit a corrective action plan in accordance with established AOA timelines and guidelines. Forty-eight of 96 programs (50%) in 2009 submitted a plan for deficiencies cited, a number that grew to 57 of 59 (97%) in 2010 (z score=−6.0274, P=0). This statistically significant difference indicates that tracking and sending notification to OPTIs about OGME program delinquency with the corrective action process improves compliance with the process. 
Reasons for Denial of Corrective Action Plan
Each year a few OGME programs submit corrective action plans that are recommended for denial by the specialty college. A denial recommendation is sent to the PTRC and in most cases is approved as a denial; at that point, the OGME program is expected to resubmit a more suitable corrective action plan. Since 2009, there have been 17 corrective action plans that were denied for the following reasons:
  •  
    The deficiency was not addressed sufficiently (7 plans).
  •  
    The program suggested the deficiencies were cited in error but did not submit evidence or follow procedure for a reconsideration request (4 plans).
  •  
    The plan submitted did not meet the standards (3 plans).
  •  
    The program stated that the standard will be met but failed to detail how it will be met going forward (2 plans).
  •  
    The corrective action plan failed to answer all the questions concerning the stated deficiencies (1 plan).
A strong corrective action plan should detail how the program will meet the standard, as well as highlight steps that the program has already taken to meet the standard. It is also important for a program submitting a reconsideration request to attach evidence that the standard was met during the time of the site review. 
Discussion
Compliance tracking has revealed some key insights. First, a more uniform standardization of site reviews and site review documents has optimized the AOA's ability to determine if a standard has been met. This 5-year trend can be observed in the increase in the percentage of programs in which at least 1 deficiency has been cited. Standardizing processes for evaluation of education programs is beneficial and improves the chance of finding educational deficiencies. Once deficiencies are identified, educational programs can work to correct those areas and improve the program. 
Second, compliance with the corrective action process has increased, an effect of the openness of the AOA's tracking process, which involves both OGME programs and OPTIs. Currently, the AOA sends each OPTI a quarterly report that shows (1) all of its programs that have been cited with a deficiency and (2) if and when a corrective action plan has been received by the AOA. Each OPTI and OGME program is expected to submit any missing plans in a timely fashion. This process helps ensure that programs are in compliance with education standards and are providing quality education to residents. 
I suggest that institutions with many OGME programs and OPTIs also do their own tracking of program compliance with the corrective action process, referring to the items cited in the Reasons for Denial of Corrective Action Plan section. Armed with these reasons, programs may avoid such pitfalls and work to create a solid corrective action plan from the beginning. Currently, no AOA rules or standards exist for OGME programs that have yet to submit a corrective action plan. Such standards, if created, may help programs to increase compliance with the corrective action process. By highlighting these areas for improvement, OGME leaders can increase quality and adherence to AOA standards so that osteopathic trainees have the best possible experience and education. 
A limitation of the present study is that the data that pertain to program details are confidential and thus cannot be released to the public; however, the AOA provides these data to each OPTI and specialty college for use in their own programs. Also, the present study was not intended to prove or disprove a hypothesis but rather to find patterns and make inferences from the data. 
Conclusion
The AOA process for approval and review of OGME programs has allowed a useful set of data to be obtained on citation and correction of educational deficiencies. Compliance tracking will continue to keep the AOA, OPTIs, and programs informed of individual program compliance and of the process for correction of deficiencies. 
   Financial Disclosures: None reported.
 
References
Gevitz N. The DOs: Osteopathic Medicine in America. 2nd ed. Baltimore, MD: The Johns Hopkins University Press; 2004.
American Osteopathic Association, American College of Osteopathic Family Physicians. Basic Standards for Residency Training in Osteopathic Family Medicine and Manipulative Treatment. Chicago, IL: American Osteopathic Association; 2011. http://www.osteopathic.org/inside-aoa/accreditation/postdoctoral-training-approval/postdoctoral-training-standards/Documents/basic-standards-for-residency-training-in-osteopathic-family-practice.pdf. Accessed December 18, 2013.
The Basic Documents for Postdoctoral Training. Chicago, IL: American Osteopathic Association; 2013. http://www.osteopathic.org/inside-aoa/accreditation/postdoctoral-training-approval/postdoctoral-training-standards/Documents/aoa-basic-document-for-postdoctoral-training.pdf. Accessed February 1, 2014.
Guidelines for Correction of Deficiencies. Chicago, IL: American Osteopathic Association; 2013. http://www.osteopathic.org/inside-aoa/accreditation/postdoctoral-training-approval/Documents/guidelines-for-correction-of-deficiencies.pdf. Accessed December 18, 2013.
Figure.
The osteopathic graduate medical education (OGME) program evaluation and corrective action process. The American Osteopathic Association (AOA) tracks all steps of the process. Abbreviations: OPTI, osteopathic postdoctoral training institution; PTRC, Program Training and Review Council; SC, specialty college.
Figure.
The osteopathic graduate medical education (OGME) program evaluation and corrective action process. The American Osteopathic Association (AOA) tracks all steps of the process. Abbreviations: OPTI, osteopathic postdoctoral training institution; PTRC, Program Training and Review Council; SC, specialty college.
Table.
Rates of Approval, Deficiency Citations, Corrective Action Plans, and Denial for Osteopathic Graduate Medical Education Programs, 2009-2013
Year
Description 2009 2010 2011 2012 2013 Total Average
Programs receiving continuing approval, No. 210 157 126 132 147 772 154
Programs with ⩾1 deficiency, No. (%) 96 (46) 59 (38) 75 (60) 68 (52) 107 (73) 405 (52) 81 (54)
Corrective action plans received, No. (%) 48 (50) 57 (97) 71 (95) 58 (85) 54 (NA)a 234 (NA)a 58 (82)
Plans recommended for denial by specialty college, No. 3 3 5 5 1 17 3
  a Percentage not applicable. Programs cited by the October 2013 Program and Trainee Review Council did not have to submit a corrective action plan until January 2014. However, not all programs had submitted by the time of publication.
Table.
Rates of Approval, Deficiency Citations, Corrective Action Plans, and Denial for Osteopathic Graduate Medical Education Programs, 2009-2013
Year
Description 2009 2010 2011 2012 2013 Total Average
Programs receiving continuing approval, No. 210 157 126 132 147 772 154
Programs with ⩾1 deficiency, No. (%) 96 (46) 59 (38) 75 (60) 68 (52) 107 (73) 405 (52) 81 (54)
Corrective action plans received, No. (%) 48 (50) 57 (97) 71 (95) 58 (85) 54 (NA)a 234 (NA)a 58 (82)
Plans recommended for denial by specialty college, No. 3 3 5 5 1 17 3
  a Percentage not applicable. Programs cited by the October 2013 Program and Trainee Review Council did not have to submit a corrective action plan until January 2014. However, not all programs had submitted by the time of publication.
×