In 1947, the American Osteopathic Association (AOA) began granting approval to osteopathic graduate medical education (OGME) programs.¹ As a means to ensure quality education in all OGME programs, the AOA sends site reviewers (previously called inspectors) to conduct a site review that is designed to uncover any existing deficiencies (ie, standards that have not been met). The resulting report is used to determine a period of continuing approval for an OGME program. 

Since 2008, the AOA has focused on the following information: (1) programs cited with at least 1 deficiency, (2) programs that subsequently submitted a corrective action plan to the AOA, (3) if such plans were approved by the specialty college, and (4) which plans were implemented. In the present article, I provide a summary of the corrective action process and the AOA's OGME data, identify trends and patterns, and suggest areas for improvement. This information may provide OGME leaders with a way to improve the quality of their programs, which in turn should result in better-prepared osteopathic physicians in the future. 

Any discussion of the corrective action process must be grounded in an understanding of the AOA's continuing approval process for internships, residencies, and fellowships, which is very similar to the accreditation process used by the Accreditation Council of Graduate Medical Education. It is important for OGME programs to have AOA approval to ensure quality and uniformity of training among a large number of programs. For example, 2 osteopathic physicians graduating from the same college of osteopathic medicine can enter 2 different family medicine residencies at 2 different hospitals and be certain that they will both receive 4 weeks of training in women's health during the first OGME year (assuming the programs are meeting the standards outlined in the AOA's Basic Standards for Residency Training in Osteopathic Family Medicine and Manipulative Treatment).² This document outlines sufficient training procedures for specialty and subspecialty, whereas AOA approval exists to provide oversight and verify that standards are met. 

The AOA approval process for an existing program begins with a program site review (previously called an inspection) conducted by a site reviewer using a crosswalk. A crosswalk is a document with checkboxes that lists (1) all the standards to be reviewed and (2) the amount of points of each deficiency, enabling a final score to be tabulated for the program. When the site reviewer notices a standard that has not been met, he or she describes the deficiency and references the appropriate standard listed in the crosswalk. The site reviewer then issues a final report that outlines any areas in which the program does not meet AOA OGME standards.³ Each crosswalk is customized for a given specialty or subspecialty and is designed to include all relevant standards that ensure trainees receive sufficient immersion and education in their field. The Council on Postdoctoral Training oversees approval of standards and crosswalks before the final approval by the AOA Board of Trustees. Current AOA-approved standards and crosswalks can be found online at http://www.osteopathic.org/inside-aoa/accreditation/postdoctoral-training-approval/postdoctoral-training-Standards/. 

The specialty college education committee reviews the final report, and then the committee submits a resolution recommending the program's next term of continuing approval to the AOA's Program and Trainee Review Council (PTRC). The resolution details the program's basic information, any deficiencies, the site reviewer's verbatim description of each deficiency, and a recommended length of approval for the program (the amount of time until the next site review). Three weeks before the PTRC meeting, the program's Osteopathic Postdoctoral Training Institution (OPTI)—which is the academic sponsor for the program—receives a copy of the continuing approval resolution for their program. If the OPTI notices that any deficiency was cited in error, the OPTI and the program are allowed to submit evidence (ie, an “error in fact” request) to the specialty college and PTRC for review. The PTRC will determine if the deficiency was indeed cited in error and should be removed from the final recommendation. At each PTRC meeting, the AOA receives 3 to 5 “error in fact” requests from OPTIs or OGME programs. The PTRC makes the final determination on any deficiencies and the final continuing approval length. 

Within 2 weeks of the PTRC meeting, a letter is sent to the OGME program detailing the final determination, any applicable deficiencies, and the site reviewer's comments on exactly how the standard was not met. Any program cited with deficiencies is required to submit a corrective action plan explaining how it will meet the standards.⁴ This plan must first be submitted to the program's OPTI within 45 days of receiving the PTRC letter. The OPTI then has 30 days to review the plan. If the plan is approved by the OPTI, the plan is then forwarded to the AOA. The AOA performs an administrative review of the plan and then forwards it to the specialty college for review and final approval. The specialty college then notifies the AOA of its decision, and the AOA writes the final letter to the OGME program informing it of the specialty college's approval or denial. When the program receives the approval letter, it has 9 months to submit evidence of implementation of the corrective action plan to their OPTI. The OPTI then approves that evidence and informs the AOA. The process for a given program is then considered closed as soon as the AOA receives the evidence of approval from the OPTI. The Figure summarizes the corrective action process described above. 

The AOA tracks all steps involved in the approval or denial process. It does not, however, track programs for which deficiencies have not been cited. 

If a program was cited with deficiencies during PTRC review, an AOA staff member enters the program's name into an Excel spreadsheet (Microsoft Corporation), along with the following data: specialty area, specialty college, OPTI name, dates (reviewed by PTRC, plan received by AOA, forwarded, approved, verified by OPTI), and additional comments. The spreadsheet documented 772 program records at the time of this writing. The data were divided by year to determine if there were any patterns or inferences that could be made from year-to-year changes in the data and changes in postdoctoral training policies. The AOA tracks all programs that have received continuing approval at the PTRC level in a given year. Therefore, it was simple to find the population size (N) for each year. This tracking process made it possible to compare data on program approval, citation of deficiencies, compliance, and a specialty college's denial of an OGME program. 

A z score was used to compare different years and determine if the difference between 2 years was statistically significant. Differences of P<.05 were considered statistically significant. 

The Table shows 2009 through 2013 data related to OGME program approval, citation of deficiencies, and compliance. The years 2009 and 2010 show the lowest percentage of programs cited with 1 or more deficiencies. Programs cited with deficiencies then increase steadily from 2011 to 2013. In 2011, uniform standardization of site reviews was implemented and crosswalks were launched. Before 2011, site reviews used the Basic Standards² as a reference and were composed of a narrative report and a variety of different specialty-specific workbooks and worksheets. The goals of creating uniform standardization of crosswalks were to increase the level of objectivity in site review and to make deficiencies clearer to the site reviewer and any subsequent readers of the report. The data suggest that uniform standardization and consistent use of crosswalks has improved site reviewers' abilities to catch deficiencies. In 2010, 59 of 157 programs (38%) needed a corrective action plan; the following year, the number was 75 of 126 (60%) (z score=−3.6745; P<.001). 

The AOA implemented another aspect of the uniform standardization process in 2013: professional site reviewers were enlisted to conduct the majority of reviews. (In past years, the task was entrusted to volunteers.) As a result of this change, 2013 showed the highest percentage of programs cited with deficiencies. In 2012, there were 68 of 132 programs (52%) that required a corrective action plan. That proportion increased in 2013 to 107 of 147 programs (73%), a difference that was statistically significant (z score=−3.6691; P<.001). 

The percentage of programs complying with the corrective action process has also improved since 2009. In 2008, the AOA started to track submission of corrective action plans, and in 2009 the AOA informed specialty colleges, OGME programs, and OPTIs that it was tracking compliance with the corrective action process. Also in 2009, the PTRC began receiving reports of compliance with the corrective action process. (Before 2008, compliance was neither tracked nor reported.) In 2009, the AOA also began regularly contacting OPTIs if 1 of its OGME programs failed to submit a corrective action plan in accordance with established AOA timelines and guidelines. Forty-eight of 96 programs (50%) in 2009 submitted a plan for deficiencies cited, a number that grew to 57 of 59 (97%) in 2010 (z score=−6.0274, P=0). This statistically significant difference indicates that tracking and sending notification to OPTIs about OGME program delinquency with the corrective action process improves compliance with the process. 

A strong corrective action plan should detail how the program will meet the standard, as well as highlight steps that the program has already taken to meet the standard. It is also important for a program submitting a reconsideration request to attach evidence that the standard was met during the time of the site review. 

Compliance tracking has revealed some key insights. First, a more uniform standardization of site reviews and site review documents has optimized the AOA's ability to determine if a standard has been met. This 5-year trend can be observed in the increase in the percentage of programs in which at least 1 deficiency has been cited. Standardizing processes for evaluation of education programs is beneficial and improves the chance of finding educational deficiencies. Once deficiencies are identified, educational programs can work to correct those areas and improve the program. 

Second, compliance with the corrective action process has increased, an effect of the openness of the AOA's tracking process, which involves both OGME programs and OPTIs. Currently, the AOA sends each OPTI a quarterly report that shows (1) all of its programs that have been cited with a deficiency and (2) if and when a corrective action plan has been received by the AOA. Each OPTI and OGME program is expected to submit any missing plans in a timely fashion. This process helps ensure that programs are in compliance with education standards and are providing quality education to residents. 

I suggest that institutions with many OGME programs and OPTIs also do their own tracking of program compliance with the corrective action process, referring to the items cited in the Reasons for Denial of Corrective Action Plan section. Armed with these reasons, programs may avoid such pitfalls and work to create a solid corrective action plan from the beginning. Currently, no AOA rules or standards exist for OGME programs that have yet to submit a corrective action plan. Such standards, if created, may help programs to increase compliance with the corrective action process. By highlighting these areas for improvement, OGME leaders can increase quality and adherence to AOA standards so that osteopathic trainees have the best possible experience and education. 

A limitation of the present study is that the data that pertain to program details are confidential and thus cannot be released to the public; however, the AOA provides these data to each OPTI and specialty college for use in their own programs. Also, the present study was not intended to prove or disprove a hypothesis but rather to find patterns and make inferences from the data. 

The AOA process for approval and review of OGME programs has allowed a useful set of data to be obtained on citation and correction of educational deficiencies. Compliance tracking will continue to keep the AOA, OPTIs, and programs informed of individual program compliance and of the process for correction of deficiencies.