Abstract
The American Osteopathic Association Commission on Osteopathic College Accreditation (AOA COCA) is recognized by the US Secretary of Education as the only accrediting agency for predoctoral osteopathic medical education. To maintain its recognition with the US Secretary of Education, AOA COCA is required to adhere to all federal laws and regulations associated with recognition of accrediting agencies. Included in the regulations are requirements for training the evaluators of institutions and programs. The authors discuss AOA COCA's established procedures for evaluator training and site visit conduct. Also included is a review of class sizes of the accredited colleges of osteopathic medicine.
Since 2008, T
he J
ournal's annual osteopathic medical education theme issue has included a series of articles that address major areas of the American Osteopathic Association Commission on Osteopathic College Accreditation (AOA COCA). The series, prepared by the AOA's Department of Accreditation, has focused on several topics that may be of interest to the profession and the many stakeholders in the accreditation process. In the March 2008 article,
1 we focused on key changes in the accreditation process for colleges of osteopathic medicine (COMs) in the United States
2 and on the recent growth of new COMs. In March 2009, we addressed AOA COCA procedures regarding “substantive changes” to institutional missions and programs—the chief example of such change currently being class size increases.
3 In March 2010, we discussed AOA COCA's process of continual evaluation of COMs.
4
The US Secretary of Education recognizes AOA COCA as the only accrediting agency for predoctoral osteopathic medical education. To maintain its recognition with the US Secretary of Education, AOA COCA is required to adhere to all federal laws
5,6 and regulations
7 associated with recognition of accrediting agencies. Included in the regulations are requirements for training the evaluators of institutions and programs.
8 The commission has established procedures for the selection and training of evaluators as well as for the conduct of site visits.
9,10 In the present article, we discuss AOA COCA's established procedures for evaluator training and conduct for site visits to COMs.
Evaluation Overview
All required site visits to COMs are conducted by a team of site evaluators appointed by AOA COCA.
9 The role of this team of evaluators is to inspect or assess COMs to ensure that COMs are in compliance with any or all accreditation standards that are stipulated for review. Site visits follow a predetermined and structured schedule, which may include interviews with COM administrators, faculty, students, and affiliated clinical site personnel as well as a review of facilities, finances, and governance structure. Evaluators hold positions similar to those of the individuals that they meet with during site visits. This type of peer review model is commonly used by other accrediting agencies recognized by the US Department of Education.
Each visit is directed by a team chair. The team chair is responsible for approving the agenda; providing leadership before, during, and after the visit; acting as the team's official spokesperson; approving the site visit report; and delivering the results of the site visit report to AOA COCA.
Identifying Evaluators
The identification of evaluators is an ongoing process. Evaluators are either nominated by members of the osteopathic medical community or self-nominated. The Commission reviews the credentials for each individual that has been nominated and then votes on whether to accept him or her as an AOA COCA evaluator in his or her specific area of expertise. Before participating in the evaluation process, evaluators must agree with the standards, policies, and procedures of AOA COCA and disclose any conflicts of interest or bias toward COMs that might affect their ability to make a fair decision. The Commission's evaluators are categorized by the following 4 areas of expertise:
Administrative/Finance—Evaluators are usually presidents, deans, provosts, and chief financial officers of COMs.
Student Services—Evaluators have expertise working as senior administrators in areas such as student financial aid, admissions, and student counseling.
Preclinical Education—Evaluators are usually associate or assistant deans or faculty who are primarily basic scientists at COMs or other institutions of higher education.
Clinical Education—Like preclinical education evaluators, clinical education evaluators are typically associate or assistant deans or clinical faculty at COMs. However, this group of evaluators may also be practitioners or directors of medical education at clinical training sites.
Training for New Evaluators
Before evaluators are qualified to participate in a site visit, they are required to complete a training program developed and facilitated by AOA COCA staff. The Web-based training program, developed for new evaluators in 2008, consists of two 90-minute, interactive modules. Because new evaluators are assumed to already have the expertise in the areas they are approved to review, the training program focuses on the role of AOA COCA in accreditation and site visit activities and procedures.
First Module
Accreditation overview—In the first module, new evaluator training begins with an overview of accreditation in general. The training program then covers the role of AOA COCA in accreditation, including the process of granting, withdrawing, and denying accreditation status. This process involves monitoring COM compliance with the published standards,
Accreditation of Colleges of Osteopathic Medicine: COM Accreditation Standards and Procedures.
2 An overview of the types of visits—pre-accreditation, provisional accreditation, comprehensive, focused, and interim progress review—is presented in the standards document so that evaluators understand the purpose of the various categories of visits to which they may be assigned.
Role of the evaluator—The Web-based training addresses the role of evaluators during site visits, including the process of selecting evaluators for visits and evaluators' responsibilities as team members. New evaluators learn that the AOA COCA chair is responsible for approving members of a site visit team. Site visit team members are selected based on the areas of expertise that are to be reviewed on a particular visit (
Figure). Conflict of interest is clearly defined for new evaluators and examples are presented to help the new evaluator understand what types of situations or circumstances constitute a conflict of interest in order to assist them in making decisions of self-disclosure. If an evaluator self-discloses a conflict of interest with a particular COM or if a conflict of interest is perceived, an evaluator will not be considered for a visit.
Expectations for site visits—In the final part of the first training module, new evaluators develop an understanding of what will be expected of them onsite. An important element of site visit etiquette is confidentiality of all information. Evaluators must use discretion in keeping any information about the visits—COM materials that they review, team deliberations, interviews with personnel, and the outcome of the site visit report—from being discussed in third party conversations. This module concludes with components of a site visit agenda, including the review of documents and interviews with COM personnel, to ensure that the COM is in compliance with the standards that are assessed on the visit. The agenda also reflects time allotted for team discussion and deliberation as well as writing the draft site visit report and delivering the results of the report to the COM prior to the end of the visit.
Evaluation—The first module contains questions preceding each section to test the participants' knowledge of what will be discussed in the section as well as a posttest to see what they have learned. In addition, each participant is asked to complete an evaluation of the presentation to provide feedback to AOA COCA staff. New evaluators also receive a follow-up evaluation later in the training process on whether the training modules were effective in preparing them for a site visit. This feedback is important in order to make changes to the training that will improve its quality and effectiveness.
Second Module
The second new evaluator training module focuses on 2 areas: use of the “DOC Pad” and writing the site visit report.
DOC Pad—The “DOC Pad,” or documentation pad, is the tool used by evaluators on all site visits. New evaluators must be trained to work with this tool because it provides the findings of the site visit review that are incorporated into the site visit report. For every visit, each evaluator on the site visit team has a DOC Pad, in either electronic or hard copy format, that contains standards and guidelines to be reviewed, a list of the people to be interviewed, and documents to be reviewed in order to determine compliance with each standard. Evaluators have space in the DOC Pad to write the current findings for each standard reviewed; current findings include whether or not COMs are in compliance with each standard as well as any requirements, recommendations, or commendations. The current findings are ultimately used to write the site visit report and then are reviewed during team deliberations and discussion with the team chair before becoming final.
Writing the site visit report—Because site visit reports are evidence-based, new evaluators learn about the characteristics of evidence-based report writing; in addition to being clear, concise, cogent, direct, evaluative, and objective, evidence-based reports must substantiate all conclusions and findings. This training segment includes interactive practice exercises in which new evaluators assess examples of evidence-based site visit reports and write and present their own examples for review by other new evaluator participants. The second module is also augmented by pretest questions, a posttest, and evaluation at the end of the program. These 2 modules have been used successfully on a yearly basis since 2008 and have provided training to approximately 30 new evaluators.
Shadowing
After new evaluators have completed both training modules, they are ready to participate in a site visit as a “trainee.” Trainees are assigned to experienced evaluators in their areas of expertise and shadow the experienced evaluators throughout comprehensive or provisional site visits. Trainees participate in interviews, the review of documents and materials, team discussions, deliberations, and report writing under the guidance of the experienced evaluator. At the successful completion of this component of the new evaluator training, the new evaluators are ready to be assigned to participate in a site visit as a regular member of the site visit team. The cost of training a new evaluator, which includes participation in the Web-based training, travel, and other expenses for the onsite trainee, is covered by AOA COCA.
Continuing Training for Evaluators
Evaluator training is a continual process and does not end once the trainee has assumed the role of an experienced evaluator. For the purposes of developing and implementing continual educational accreditation training for evaluators, the AOA COCA Committee on College Accreditation (CCAT) plans and conducts workshops. These workshops are face-to-face training programs and are held on a biannual basis.
The most recent workshop, which took place in January 2011, had the general theme of achieving consistency in standards review. Included in the workshop were plenary sessions on reaching consensus as a team. Evaluators participated in an interactive program that reviewed existing criteria and aimed to develop consistency in reviewing standards. Specifically, the program was designed to help evaluators determine a uniform response in reviewing standards that are frequently cited as being out of compliance by COMs on site visits. All evaluators also took part in an interactive presentation on developing competence in writing evidence-based site visit reports.
The workshop also included breakout sessions for training of evaluators in specific areas of expertise. Team chairs and administrative/finance evaluators had an opportunity to develop an enhanced understanding of financial statements. Preclinical and clinical evaluators received training on the importance of student and institutional assessment and the use of data to drive institutional improvement. Student services evaluators met to evaluate the standards that they review and to make recommendations for revisions to those standards to be sent to the AOA COCA Standards Review Committee. Team chairs were also trained in the further development of their leadership and team management skills. Workshop participants completed an evaluation of the program and made recommendations for additional training topics. This feedback assists staff in creating the needs-based program at future workshops.
Important accreditation issues often arise between workshops. For example, when new standards became effective on July 1, 2010, a teleconference was held with preclinical and clinical evaluators and team chairs to discuss the changes that affected the standards they must review. Staff, including the AOA COCA secretary (K.C.M.) and AOA COCA assistant secretary (A.W.), facilitated the new standards review teleconference and answered questions that helped evaluators interpret the new standards consistently.
Evaluators on a Site Visit
On comprehensive or provisional site visits, in which all standard domains are reviewed, evaluators are assigned to review specific standards according to their areas of expertise (
Figure).
Typically, a comprehensive or provisional site visit would have a team chair, 1 administrative/finance evaluator, 1 student services evaluator, 1 preclinical evaluator, and 2 clinical evaluators. Ideally, 1 clinical evaluator would be an educator and 1 would be a practitioner. The team members for a focused visit or interim progress review would be chosen according to the standards designated for review. For example, a focused visit that monitored clinical education standards in the third year of a COM class size increase (Standard 6) would have a team chair and a clinical evaluator participating in the site visit. A focused visit to monitor COM governance (Standard 1) would have a team chair and an administrative/finance evaluator on the site visit team. Currently, the AOA COCA's evaluator registry has a total of 121 approved evaluators comprising 18 administrative/finance evaluators, 19 student services evaluators, 26 preclinical evaluators, and 58 clinical evaluators. Nineteen clinical evaluators are practitioners.
Recent Changes to Approved Class Sizes
For the past 10 academic years, AOA COCA–approved class sizes have steadily increased. During the 2000-2001 academic year, 19 COMs operating on 19 campuses were approved for 2953 total matriculants. During the current 2010-2011 academic year 5067 students were approved for the aggregate class sizes at 26 COMs, operating 3 branch campuses and 3 additional locations for a total of 32 sites (
Table).
Table.
AOA COCA–Approved Class Sizes for Academic Years 2001-2002 Through 2010-2011
|
Academic Year
| | | | | | | | | |
|---|
COM*
| 2001-2002
| 2002-2003
| 2003-2004
| 2004-2005
| 2005-2006
| 2006-2007
| 2007-2008
| 2008-2009
| 2009-2010
| 2010-2011
|
|---|
| ▪ ATSU-KCOM | 160 | 160 | 160 | 160 | 160 | 160 | 160 | 160 | 160 | 160 |
| ▪ ATSU-SOMA | NA | NA | NA | NA | NA | NA | 100 | 100 | 100 | 100 |
| ▪ DMU-COM | 205 | 205 | 205 | 205 | 205 | 205 | 205 | 205 | 205 | 205 |
| ▪ KCUMB-COM | 220 | 220 | 220 | 220 | 250 | 250 | 250 | 250 | 250 | 250 |
| ▪ LECOM (LDP) | 144 | 144 | 144 | 144 | 144 | 144 | 180 | 180 | 180 | 180 |
| □ LECOM (PBLP) | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 |
| □ LECOM (ISP) | | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 |
| □ LECOM-Bradenton† (PBLP) | NA | NA | NA | 150 | 150 | 150 | 150 | 150 | 150 | 150 |
| □ LECOM-Seton Hill‡ | NA | NA | NA | NA | NA | NA | NA | NA | 104 | 104 |
| ▪ LMU-DCOM | NA | NA | NA | NA | NA | NA | 150 | 150 | 150 | 150 |
| ▪ MSUCOM | 185 | 185 | 200 | 200 | 200 | 200 | 200 | 200 | 200 | 200 |
| □ MSUCOM-DMC‡ | NA | NA | NA | NA | NA | NA | NA | NA | 50 | 50 |
| □ MSUCOM-MUC‡ | NA | NA | NA | NA | NA | NA | NA | NA | 50 | 50 |
| ▪ MWU/AZCOM | 125 | 140 | 140 | 140 | 140 | 140 | 140 | 250 | 250 | 250 |
| ▪ MWU/CCOM | 160 | 160 | 160 | 160 | 160 | 160 | 160 | 160 | 206 | 206 |
| ▪ NSU-COM | 180 | 180 | 200 | 200 | 230 | 230 | 230 | 230 | 230 | 230 |
| ▪ NYCOM | 260 | 260 | 260 | 260 | 260 | 260 | 260 | 260 | 260 | 260 |
| □ NYCOM (Émigré Physicians Program) | 40 | 40 | 35 | 35 | 35 | 35 | 35 | 35 | 35 | 35 |
| ▪ OSU-COM§ | 88 | 88 | 88 | 88 | 88 | 88 | 88 | 88 | 97 | 106 |
| ▪ OU-COM | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 140 | 140 | 140 |
| ▪ PCOM | 250 | 250 | 250 | 250 | 250 | 250 | 250 | 250 | 250 | 250 |
| □ GA-PCOM† | NA | NA | NA | NA | 80 | 80 | 80 | 80 | 80 | 80 |
| ▪ PCSOM | 60 | 60 | 75 | 75 | 75 | 75 | 75 | 75 | 75 | 75 |
| ▪ PNWU-COM | NA | NA | NA | NA | NA | NA | NA | 70 | 70 | 70 |
| ▪ RVUCOM | NA | NA | NA | NA | NA | NA | NA | 150 | 150 | 150 |
| ▪ TouroCOM | NA | NA | NA | NA | NA | NA | 125 | 125 | 125 | 125 |
| ▪ TUCCOM | 125 | 125 | 125 | 125 | 125 | 125 | 125 | 125 | 125 | 125 |
| □ TUNCOM† | NA | NA | NA | 125 | 125 | 125 | 125 | 125 | 125 | 125 |
| ▪ UMDNJ-SOM | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 150 | 150 |
| ▪ UNECOM | 115 | 115 | 115 | 115 | 115 | 115 | 115 | 115 | 115 | 115 |
| ▪ UNTHSC/TCOM§ | 115 | 115 | 125 | 125 | 125 | 175 | 175 | 175 | 200 | 225 |
| ▪ VCOM-VC | NA | NA | 150 | 150 | 150 | 150 | 150 | 175 | 175 | 175 |
| ▪ WesternU/COMP | 176 | 176 | 176 | 176 | 176 | 176 | 176 | 176 | 176 | 176 |
| □ WesternU/COMP∥ (NWT) | NA | NA | NA | 30 | 30 | 30 | 30 | 30 | 30 | 30 |
| ▪ WCUCOM | NA | NA | NA | NA | NA | NA | NA | NA | NA | 100 |
| ▪ WVSOM | 75
| 75
| 96
| 96
| 96
| 200
| 200
| 200
| 200
| 200
|
| ▪ Total | 2953 | 2968 | 3194 | 3499 | 3639 | 3793 | 4204 | 4599 | 4933 | 5067 |
□ Change from previous year, No. (%)
| 140 (5.0) | 15 (0.5) | 226 (7.6) | 305 (9.5) | 140 (4.0) | 154 (4.2) | 411(10.8) | 395 (9.4) | 334 (7.3) | 134 (2.7) |
Table.
AOA COCA–Approved Class Sizes for Academic Years 2001-2002 Through 2010-2011
|
Academic Year
| | | | | | | | | |
|---|
COM*
| 2001-2002
| 2002-2003
| 2003-2004
| 2004-2005
| 2005-2006
| 2006-2007
| 2007-2008
| 2008-2009
| 2009-2010
| 2010-2011
|
|---|
| ▪ ATSU-KCOM | 160 | 160 | 160 | 160 | 160 | 160 | 160 | 160 | 160 | 160 |
| ▪ ATSU-SOMA | NA | NA | NA | NA | NA | NA | 100 | 100 | 100 | 100 |
| ▪ DMU-COM | 205 | 205 | 205 | 205 | 205 | 205 | 205 | 205 | 205 | 205 |
| ▪ KCUMB-COM | 220 | 220 | 220 | 220 | 250 | 250 | 250 | 250 | 250 | 250 |
| ▪ LECOM (LDP) | 144 | 144 | 144 | 144 | 144 | 144 | 180 | 180 | 180 | 180 |
| □ LECOM (PBLP) | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 |
| □ LECOM (ISP) | | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 |
| □ LECOM-Bradenton† (PBLP) | NA | NA | NA | 150 | 150 | 150 | 150 | 150 | 150 | 150 |
| □ LECOM-Seton Hill‡ | NA | NA | NA | NA | NA | NA | NA | NA | 104 | 104 |
| ▪ LMU-DCOM | NA | NA | NA | NA | NA | NA | 150 | 150 | 150 | 150 |
| ▪ MSUCOM | 185 | 185 | 200 | 200 | 200 | 200 | 200 | 200 | 200 | 200 |
| □ MSUCOM-DMC‡ | NA | NA | NA | NA | NA | NA | NA | NA | 50 | 50 |
| □ MSUCOM-MUC‡ | NA | NA | NA | NA | NA | NA | NA | NA | 50 | 50 |
| ▪ MWU/AZCOM | 125 | 140 | 140 | 140 | 140 | 140 | 140 | 250 | 250 | 250 |
| ▪ MWU/CCOM | 160 | 160 | 160 | 160 | 160 | 160 | 160 | 160 | 206 | 206 |
| ▪ NSU-COM | 180 | 180 | 200 | 200 | 230 | 230 | 230 | 230 | 230 | 230 |
| ▪ NYCOM | 260 | 260 | 260 | 260 | 260 | 260 | 260 | 260 | 260 | 260 |
| □ NYCOM (Émigré Physicians Program) | 40 | 40 | 35 | 35 | 35 | 35 | 35 | 35 | 35 | 35 |
| ▪ OSU-COM§ | 88 | 88 | 88 | 88 | 88 | 88 | 88 | 88 | 97 | 106 |
| ▪ OU-COM | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 140 | 140 | 140 |
| ▪ PCOM | 250 | 250 | 250 | 250 | 250 | 250 | 250 | 250 | 250 | 250 |
| □ GA-PCOM† | NA | NA | NA | NA | 80 | 80 | 80 | 80 | 80 | 80 |
| ▪ PCSOM | 60 | 60 | 75 | 75 | 75 | 75 | 75 | 75 | 75 | 75 |
| ▪ PNWU-COM | NA | NA | NA | NA | NA | NA | NA | 70 | 70 | 70 |
| ▪ RVUCOM | NA | NA | NA | NA | NA | NA | NA | 150 | 150 | 150 |
| ▪ TouroCOM | NA | NA | NA | NA | NA | NA | 125 | 125 | 125 | 125 |
| ▪ TUCCOM | 125 | 125 | 125 | 125 | 125 | 125 | 125 | 125 | 125 | 125 |
| □ TUNCOM† | NA | NA | NA | 125 | 125 | 125 | 125 | 125 | 125 | 125 |
| ▪ UMDNJ-SOM | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 150 | 150 |
| ▪ UNECOM | 115 | 115 | 115 | 115 | 115 | 115 | 115 | 115 | 115 | 115 |
| ▪ UNTHSC/TCOM§ | 115 | 115 | 125 | 125 | 125 | 175 | 175 | 175 | 200 | 225 |
| ▪ VCOM-VC | NA | NA | 150 | 150 | 150 | 150 | 150 | 175 | 175 | 175 |
| ▪ WesternU/COMP | 176 | 176 | 176 | 176 | 176 | 176 | 176 | 176 | 176 | 176 |
| □ WesternU/COMP∥ (NWT) | NA | NA | NA | 30 | 30 | 30 | 30 | 30 | 30 | 30 |
| ▪ WCUCOM | NA | NA | NA | NA | NA | NA | NA | NA | NA | 100 |
| ▪ WVSOM | 75
| 75
| 96
| 96
| 96
| 200
| 200
| 200
| 200
| 200
|
| ▪ Total | 2953 | 2968 | 3194 | 3499 | 3639 | 3793 | 4204 | 4599 | 4933 | 5067 |
□ Change from previous year, No. (%)
| 140 (5.0) | 15 (0.5) | 226 (7.6) | 305 (9.5) | 140 (4.0) | 154 (4.2) | 411(10.8) | 395 (9.4) | 334 (7.3) | 134 (2.7) |
×
The aggregate number of authorized matriculants in the approved class sizes increased by 2114 over the academic years 2001-2002 through 2010-2011, inclusive, for a 71.6% increase since the 2001-2002 academic year. The larger fraction of this increase—1129 authorized matriculants (53.4%) of the 2114 increase—was the result of establishment of new COMs, additional locations, and branch campuses authorized by AOA COCA involving 9 COMs. The remaining fraction of 985 authorized matriculants (46.6%) is accounted for by the expansion of class size at existing COMs.
During the 10-year period under discussion, the net increase in approved positions ranged from 15 in the 2002-2003 academic year (0.5% more than the previous year) to 411 in the 2007-2008 academic year (10.8% more than the previous year). In the 2010-2011 academic year, a total of 134 new approved positions were added to the approved class sizes at 3 schools (
Table) for a 2.7% increase compared with the 2009-2010 academic year's total of 4933 students. The class size increases and additions were as follows:
9 (6.7%)—Oklahoma State University College of Osteopathic Medicine in Tulsa, Oklahoma
25 (18.7%)—University of North Texas Health Science Center Texas College of Osteopathic Medicine in Fort Worth, Texas
100 (74.6%)—William Carey University-College of Osteopathic Medicine in Hattiesburg, Mississippi
Conclusion
The training of AOA COCA evaluators is a continuous process, just as the evaluation of COMs is a continuous process. The development of additional campus sites has expanded the need for evaluators. Individuals who are interested should contact the AOA's Department of Accreditation at
predoc@osteopathic.org or at (312) 202-8097 to learn about opportunities that are available.
Financial Disclosures: None reported.
Williams A, Miskowicz-Retz KC. Evolution of colleges of osteopathic medicine: a discussion of the COCA's “substantive change” policies.
J Am Osteopath Assoc. 2009;109(3)128-134.
http://www.jaoa.org/cgi/content/full/109/3/128. Accessed February 15, 2011.
Higher Education Act of 1965, 20 USC §1001 (1965).
Higher Education Opportunity Act, Pub L No. 110-315, 122 Stat 3078 (2008). To be codified at 20 USC §1001.1 et seq.
The Secretary's Recognition of Accrediting Agencies, 34 CFR §602 (2010).
Administrative and Fiscal Responsibilities, 34 CFR §602.15(a)(2) (2010).