Free
JAOA/AACOM Medical Education  |   April 2017
Evidence-Based Redesign of the COMLEX-USA Series
Author Notes
Article Information
Evidence-Based Medicine / Medical Education / Graduate Medical Education / COMLEX-USA
JAOA/AACOM Medical Education   |   April 2017
Evidence-Based Redesign of the COMLEX-USA Series
The Journal of the American Osteopathic Association, April 2017, Vol. 117, 253-261. doi:10.7556/jaoa.2017.043
The Journal of the American Osteopathic Association, April 2017, Vol. 117, 253-261. doi:10.7556/jaoa.2017.043
Web of Science® Times Cited: 2
Abstract

To ensure that the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) reflects the evolving practice of osteopathic medicine, the National Board of Osteopathic Medical Examiners has developed new content and format specifications for an enhanced, competency-based examination program to be implemented with COMLEX-USA Level 3 in 2018. This article summarizes the evidence-based design processes that served as the foundation for blueprint development and the evidence supporting its validity. An overview is provided of the blueprint’s 2 dimensions: Competency Domains and Clinical Presentations. The authors focus on the evidence that supports interpretation of test scores for the primary and intended purpose of COMLEX-USA, which is osteopathic physician licensure. Important secondary uses and the educational and catalytic effect of assessments are also described. This article concludes with the National Board of Osteopathic Medical Examiners’ plans to ensure that the COMLEX-USA series remains current and meets the needs of its stakeholders—the patients who seek care from osteopathic physicians.

Keywords: COMLEX-USA, competencies, licensure

As a licensing examination, a primary purpose of the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) is to assure the public that candidates who have passed the examination series have met an established national standard of osteopathic medical knowledge and clinical skills, making them eligible to apply for the privilege of practicing osteopathic medicine in all 50 states. Examination scores have also been used for important secondary purposes, such as a screening tool for residency program directors1 and as a graduation requirement. 
Since their initiation in 1995, the COMLEX-USA series has been developed around a patient presentation–based, 2-dimensional blueprint2 that has since been used by other medical licensing organizations.3,4 The Federation of State Medical Boards endorsed COMLEX-USA and the United States Medical Licensing Examination as valid and reliable for their intended and respective purposes and concluded that the validity of COMLEX-USA is exemplary. A summary of published evidence for the validity of the current COMLEX-USA series for licensure and other secondary uses can be found on the National Board of Osteopathic Medical Examiners (NBOME) website.5 
The purpose of an examination blueprint is to outline the content and format specifications for the examination, including the percentage of total test questions and other content associated with each topic.6 In the years since its implementation, the COMLEX-USA blueprint has been reviewed and revised regularly to reflect the practice of osteopathic medicine and to be consistent with the recommendations of the Standards for Educational and Psychological Testing (ie, Standards) established by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education in 2014.7 The addition of a performance assessment of fundamental clinical skills, COMLEX-USA Level 2-Performance Evaluation, was added to the series in 2004. Additional test item formats have been developed and incorporated into the examinations. In 2018-2019, the NBOME plans to implement the enhanced test blueprint and specifications described herein. 
Blueprint Development
To ensure that COMLEX-USA is most effectively positioned to anticipate new trends and changes in osteopathic medical practice and education, the NBOME commissioned the Blue Ribbon Panel (ie, Panel) in 2010. The Panel was charged with outlining a plan to implement a 2-decision–point, competency-based COMLEX-USA that is consistent with the NBOME’s mission to protect the public by providing the means to assess competencies for osteopathic medicine and related health care professions. Decision Point One was defined as the grouping of predoctoral assessments leading to successful promotion along the licensure pathway for entry into supervised practice (ie, graduate medical education), and Decision Point Two was defined as a series of postdoctoral assessments leading to unsupervised practice (ie, eligibility for licensure). The Panel included representatives from the American Osteopathic Association (AOA), American Association of Osteopathic Examiners, Federation of State Medical Boards, American Association of Colleges of Osteopathic Medicine (AACOM), American Colleges of Graduate Medical Education (ACGME), Organization of Program Director Associations, Association of Osteopathic Directors and Medical Educators, Educational Council on Osteopathic Principles, Educational Commission for Foreign Medical Graduates, NBOME Board of Directors, and NBOME National Faculty. 
Further, the Coalition for Physician Accountability, of which the NBOME is a charter member, along with the AOA and AACOM, issued the Consensus Statement on a Framework for Professional Competence8 in 2013 that called its members to “align medical education, training and assessment with the competency framework to help physicians demonstrate mastery and excellence throughout their careers.” The Coalition solicited the input of the public (nonphysician) members of its 12 member organizations regarding physician competencies and competency domains. This consensus was reviewed by the Panel in its deliberations. 
One of the primary accomplishments of the Panel was to recommend a revised COMLEX-USA blueprint consisting of 2 dimensions—Competency Domains and Clinical Presentations—updated from the current dimensions of Patient Presentations and Physician Tasks. As a high-stakes summative examination, COMLEX-USA follows NBOME’s Standards for Quality Assurance9 that are aligned with the aforementioned Standards, which state that validity, the fundamental consideration of a test, is “…the degree to which all the accumulated evidence supports the intended interpretation of test scores for the proposed use.”7(p14) As a result, a major focus of the Panel’s work in developing the new blueprint was gathering and integrating multiple sources of evidence based on established standards and psychometric principles. 
These Standards describe the importance of examination blueprints being job-related and incorporating the knowledge, skills, and abilities required for effective practice for the profession. The requirement of being job-related is met in part by conducting a practice analysis, which is “the systematic study of a profession undertaken to identify and describe the job responsibilities of those employed in the profession.”10(p29) Practice analyses may include a variety of procedures, including a comprehensive review of relevant research, such as national survey data and other relevant literature; collection and evaluation of expert opinion in that profession; and completion of surveys, such as task inventories designed to elicit information from stakeholders.11 
The Panel’s initial step was to focus on the competency domains, with a starting point of previous NBOME research that defined the osteopathic medical competency domains and described them primarily from the licensure assessment perspective.12,13 To supplement these data, the Panel conducted a literature survey that included review of national and international medical competency models that have influenced the expectations of good medical practice. Among these frameworks were AACOM’s Osteopathic Core Competencies for Medical Students,14 CanMEDS 2005 Physician Competency Framework,15 and the ACGME Outcomes Project,16 as well as current research on physician revalidation.17 The result of this research was the updated Fundamental Osteopathic Medical Competency Domains (FOMCD) 2011,18 which delineated 7 competency domains comprising required elements and measurable outcomes. 
A second outcome of the competency research conducted by the Panel was the publication of FOMCD 2016,19 revised from FOMCD 2011 to provide a clarified presentation of required elements and outcomes that comprise the competency domains. The required elements detail the essential foundational specifications, including specific, definable knowledge, skills, experiences, attitudes, values, or behaviors that make up the standards for the competency domain. Each required element includes 1 or more outcomes and more explicit description statements of desired abilities, which are further classified as anticipated to be measured, attested, or not measured in the COMLEX-USA series. 
To obtain current data regarding the clinical practices of osteopathic physicians (ie, DOs), the Panel studied data from the 2010 National Ambulatory Medical Care Survey20 that reports medical care statistics related to DOs’ practice, such as patient demographics and visit characteristics including the principal reason for visit. Comparing data from DOs and allopathic physicians (ie, MDs) confirmed that DOs encounter more patients with musculoskeletal (13.4% DOs vs 9.2% MDs) and respiratory (13.5% DOs vs 9.9% MDs) complaints than MDs. 
The Panel also evaluated data from the 2010 National Hospital Ambulatory Medical Care Survey,21 the 2010 National Hospital Discharge Survey,22 and the 2004 National Nursing Home Survey.23 These analyses demonstrated some of the distinctive features of osteopathic medical practice and confirm that a higher percentage of patients with musculoskeletal and respiratory system complaints present to DOs than to MDs.24 This finding is reflected in the blueprint weighting. 
Panel Research and Surveys
In addition to reviewing current research and national survey data, the Panel sponsored research projects that contributed to the practice analysis, including a survey of the clinical procedures taught during residency. Other research efforts investigated the relationship between competency ratings of residents and COMLEX-USA scores,25,26 contributing convergent evidence of validity.7 
With the intention of adding broad stakeholder opinions to the practice analysis, the Panel sponsored a test development survey that was distributed to more than 15,000 persons in the osteopathic medical profession, including faculty and deans of colleges of osteopathic medicine, the Association of Osteopathic Directors and Medical Educators, the American Association of Osteopathic Examiners, and osteopathic medical students from the graduating classes of 2009 to 2013. 
This survey asked respondents to rate the importance of the specific competency domain required elements at Decision Point One (ie, entry into supervised practice, residency training) and Decision Point Two (ie, entry into unsupervised practice, unsupervised practice/licensure). These domains included the 7 Competency Domains: Osteopathic Principles, Practice, and Manipulative Treatment; Osteopathic Patient Care and Procedural Skills; Application of Knowledge for Osteopathic Medical Practice; Practice-Based Learning and Improvement in Osteopathic Medical Practice; Interpersonal and Communication Skills in the Practice of Osteopathic Medicine; Professionalism in the Practice of Osteopathic Medicine; and Systems-Based Practice in Osteopathic Medicine. The elements receiving the highest averages among DO respondents for Decision Points One and Two are listed in Table 1. Information from the survey was evaluated by the Panel and considered in the final determination of competency-based content for the enhanced blueprint. 
Table 1.
NBOME Test Development Survey Outcomes: Ratings of Required Elements
Required Element Average Ratinga
Competency Domains: Decision Point One
  Commitment to continuous learning 4.51
  Ability to work effectively with other members of the interprofessional collaborative team 4.40
  Ability to gather data from all sources, including patients’ secondary sources, records, and physical examination results 4.37
  Ability to formulate a prioritized differential diagnosis 4.33
  Humanistic behavior 4.32
  Rapport building with patients 4.27
  Effective written and electronic communication 4.27
  Ability to give information to patients clearly, including collaboration on treatment, informed consent, and lifestyle modification 4.26
  Ability to counsel and educate patients consistent with their ability to understand 4.18
  Treatment of patients using best medical evidence 4.16
  Ability to develop a safe, evidence-based, cost-effective, patient-centered care plan 3.84
  Ability to perform basic clinical procedures essential for a general osteopathic medical practice 3.77
Clinical Presentations: Decision Point Two
  Ability to formulate a prioritized differential diagnosis 4.78
  Ability to gather data from all sources, including patients’ secondary sources, records, and physical examination results 4.70
  Ability to counsel and educate patients consistent with their ability to understand 4.68
  Ability to give information to patients clearly, including collaboration on treatment, informed consent, and lifestyle modification 4.68
  Rapport building with patients 4.66
  Ability to develop a safe, evidence-based, cost-effective, patient-centered care plan 4.63
  Treatment of patients using best medical evidence 4.60
  Commitment to continuous learning 4.59
  Ability to work effectively with other members of the interprofessional team 4.58
  Effective written and electronic communication 4.56
  Humanistic behavior 4.45
  Ability to perform basic clinical procedures essential for a general osteopathic medical practice 4.37

aItems were rated on a scale of 1 (“not important at all”) to 5 (“extremely important”).

Abbreviation: NBOME, National Board of Osteopathic Medical Examiners.

Table 1.
NBOME Test Development Survey Outcomes: Ratings of Required Elements
Required Element Average Ratinga
Competency Domains: Decision Point One
  Commitment to continuous learning 4.51
  Ability to work effectively with other members of the interprofessional collaborative team 4.40
  Ability to gather data from all sources, including patients’ secondary sources, records, and physical examination results 4.37
  Ability to formulate a prioritized differential diagnosis 4.33
  Humanistic behavior 4.32
  Rapport building with patients 4.27
  Effective written and electronic communication 4.27
  Ability to give information to patients clearly, including collaboration on treatment, informed consent, and lifestyle modification 4.26
  Ability to counsel and educate patients consistent with their ability to understand 4.18
  Treatment of patients using best medical evidence 4.16
  Ability to develop a safe, evidence-based, cost-effective, patient-centered care plan 3.84
  Ability to perform basic clinical procedures essential for a general osteopathic medical practice 3.77
Clinical Presentations: Decision Point Two
  Ability to formulate a prioritized differential diagnosis 4.78
  Ability to gather data from all sources, including patients’ secondary sources, records, and physical examination results 4.70
  Ability to counsel and educate patients consistent with their ability to understand 4.68
  Ability to give information to patients clearly, including collaboration on treatment, informed consent, and lifestyle modification 4.68
  Rapport building with patients 4.66
  Ability to develop a safe, evidence-based, cost-effective, patient-centered care plan 4.63
  Treatment of patients using best medical evidence 4.60
  Commitment to continuous learning 4.59
  Ability to work effectively with other members of the interprofessional team 4.58
  Effective written and electronic communication 4.56
  Humanistic behavior 4.45
  Ability to perform basic clinical procedures essential for a general osteopathic medical practice 4.37

aItems were rated on a scale of 1 (“not important at all”) to 5 (“extremely important”).

Abbreviation: NBOME, National Board of Osteopathic Medical Examiners.

×
The Blueprint Workgroup
As this research was being conducted, the Panel established the Blueprint Workgroup with members from representative specialties from the osteopathic medical profession who were tasked to focus on additional practice analysis activities. The Blueprint Workgroup was charged with developing recommendations to the Panel based on empirical research and their expert opinion for the development of the blueprint, including its nomenclature and weighting, particularly as related to the Clinical Presentations dimension. A clinical presentation was defined as the manner in which a particular patient, group of patients, or community presents to DOs. Clinical presentations are high-frequency, high-impact categories based on evidence from osteopathic medical practice and are further classified as categories and topics. 
Using the current blueprint’s Patient Presentations dimension as a starting point for their research, members of the Blueprint Workgroup completed the Blueprint Workgroup Survey that asked them to identify topics that should be retained as is, retained but modified, eliminated, or added. By presenting the results of the survey, members of the workgroup used data to achieve consensus and arrive at a focused direction for both modernizing the language and reorganizing the categories to propose the new Clinical Presentations dimension. The Blueprint Workgroup continued their research by evaluating national survey data.20-23 To supplement these, they reviewed DO and MD medical school curricular programs in the United States, international medical school curricular programs, graduate medical education objectives, and competencies and milestones to arrive at consensus regarding the evolving best practices and standard content specifications across the educational spectrum. Using their clinical expertise and a consensus-driven process focusing on how patients present to DOs, the Blueprint Workgroup further defined the topics and subtopics that comprised the categories of the Clinical Presentations dimension. These recommendations were presented back to the Panel for their consideration. 
The 2018-2019 Master Blueprint
The outcome of the research and analysis conducted by the Panel and the Blueprint Workgroup was the development of an enhanced, competency-based, 2-dimensional blueprint that is aligned with the evolving practice of osteopathic medicine. It includes as its introductory preamble the tenets of osteopathic medicine27: 
  • ■ The body is a unit; the person is a unit of body, mind, and spirit.
  • ■ The body is capable of self-regulation, self-healing, and health maintenance.
  • ■ Structure and function are reciprocally interrelated.
  • ■ Rational treatment is based upon an understanding of the basic principles of body unity, self-regulation, and the interrelationship of structure and function.
The Competency Domains dimension (Table 2) in the new blueprint was defined as “related sets of foundational abilities and represent the required elements and outcomes that define the knowledge, skills, experience, attitudes, values, and behaviors of established professional standards”19 and provide an overall framework for the practice of osteopathic medicine. 
Table 2.
Weighting for COMLEX-USA Competency Domains and Clinical Presentations
Dimensions Minimum Weighting, %
Competency Domains: Dimension One
  Osteopathic Principles, Practice, and Manipulative Treatment 10
  Osteopathic Patient Care and Procedural Skills 25
  Application of Knowledge for Osteopathic Medical Practice 30
  Practice-Based Learning and Improvement in Osteopathic Medical Practice 5
  Interpersonal and Communication Skills in the Practice of Osteopathic Medicine 10
  Professionalism in the Practice of Osteopathic Medicine 5
  Systems-Based Practice in Osteopathic Medicine 5
Clinical Presentations: Dimension Two
  Community Health and Presentations Related to Wellness 12
  Patient Presentations Related to:
    Human Development, Reproduction, and Sexuality 5
    Endocrine System and Metabolism 5
    Nervous System and Mental Health 10
    Musculoskeletal System 13
    Genitourinary System 5
    Gastrointestinal System and Nutritional Health 10
    Circulatory and Hematologic Systems 10
    Respiratory System 10
    Integumentary System 5

Abbreviation: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA.

Source: Reprinted with permission from the National Board of Osteopathic Medical Examiners.

Table 2.
Weighting for COMLEX-USA Competency Domains and Clinical Presentations
Dimensions Minimum Weighting, %
Competency Domains: Dimension One
  Osteopathic Principles, Practice, and Manipulative Treatment 10
  Osteopathic Patient Care and Procedural Skills 25
  Application of Knowledge for Osteopathic Medical Practice 30
  Practice-Based Learning and Improvement in Osteopathic Medical Practice 5
  Interpersonal and Communication Skills in the Practice of Osteopathic Medicine 10
  Professionalism in the Practice of Osteopathic Medicine 5
  Systems-Based Practice in Osteopathic Medicine 5
Clinical Presentations: Dimension Two
  Community Health and Presentations Related to Wellness 12
  Patient Presentations Related to:
    Human Development, Reproduction, and Sexuality 5
    Endocrine System and Metabolism 5
    Nervous System and Mental Health 10
    Musculoskeletal System 13
    Genitourinary System 5
    Gastrointestinal System and Nutritional Health 10
    Circulatory and Hematologic Systems 10
    Respiratory System 10
    Integumentary System 5

Abbreviation: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA.

Source: Reprinted with permission from the National Board of Osteopathic Medical Examiners.

×
The Competency Domains dimension of the blueprint includes the measured outcomes of FOMCD 2016. Measured outcomes are considered statements of well-defined abilities, including knowledge, skills, experiences, attitudes, values, or behaviors that are observable and measurable and can be directly assessed in a reliable manner in the assessments that make up the COMLEX-USA series. Although attested outcomes and not-measured outcomes, which typically would require workplace-based assessment, are not currently expected to be assessed in the new COMLEX-USA, these outcomes could provide opportunities for further collaboration and research and eventually be included in an expanded portfolio for licensure. The Clinical Presentations dimension of the new blueprint defines the manner in which a particular patient, group of patients, or community presents to DOs (Table 2). 
Ultimately, the weighting for competency domains and clinical presentations, based on the comprehensive analyses of osteopathic medical practice, was determined by the Panel.5 
Primary and Secondary Uses of COMLEX-USA
The Panel affirmed that the primary and intended purpose of COMLEX-USA is for the licensure of DOs. However, the Panel acknowledged the evidence for important secondary uses of the examination results, including screening applicants for residency programs, summative evaluation for graduation from a college of osteopathic medicine, and promotion and academic progress evaluation in colleges or within residency programs. Although the NBOME supports holistic admission reviews in the residency application process and has cautioned residency program directors to avoid the overuse or sole use of any examination program for screening or hiring of residents, the evidence supporting these secondary uses is relevant. The ACGME recognizes and supports the use of COMLEX-USA for DOs in its programs,5 stating that “COMLEX and USMLE [United States Medical Licensing Examination] are both acceptable to the ACGME….We recognize the important role that COMLEX-USA plays in quality for osteopathic medical education and training.”28 
Next Steps
In developing the Master Blueprint 2018-2019, the Panel sponsored new research, evaluated existing research, considered expert opinion, and relied on psychometric best practices to create an evidence-based blueprint design for the COMLEX-USA series. Because the updated blueprint calls for new content with different weighting, its implementation may result in changes to the content of other assessments currently being administered in osteopathic medical education. With this in mind, the NBOME has begun communicating the Master Blueprint 2018-2019 to educators.29 
The new blueprint will be implemented beginning with Decision Point Two (COMLEX-USA Level 3) in 2018. With the goal of assessing across a broader competency subset, COMLEX-USA Level 3 will be extended from 1 day to 2 days and will include multiple-choice items as well as novel item formats. The new blueprint will be implemented for Decision Point One examinations (COMLEX-USA Level 1, Level 2-Cognitive Evaluation, and Level 2-Performance Evaluation) beginning in 2019. These examinations will remain similar in format but will follow the new blueprint’s content outline. Test specifications for each level and component of COMLEX-USA will be available publicly in 2017 to provide more detailed direction to stakeholders. Updated format specifications are currently available on the NBOME’s website.30 
In keeping with its focus on maximizing the validity evidence for COMLEX-USA, the NBOME continues to develop novel item formats for use on COMLEX-USA to assess a wider range of competencies and to increase the authenticity of its assessments. Among them are expanded use of multimedia test items, items that incorporate patients’ heart and lung sounds, and the Key Features/Clinical Decision Making31 format that was introduced in COMLEX-USA Level 3 in 2014 and is currently in use on other medical licensing examinations, notably by the Medical Council of Canada.32 Currently, the NBOME is investigating the potential of point-of-care item formats that would assess candidates’ use of resources such as clinical decision-making tools and resource applications typically available to physicians in actual clinical practice. 
During the past 5 to 7 years, there has been a focus on “milestones,”33 the developmental landmarks that mark the stepwise progression of a resident toward competency development acquired progressively during his or her residency training. As conceptualized by the ACGME, the milestones are delineated by specialty for each competency in terms of 5 levels of proficiency, from novice to expert, and require performance judgments by trained observers in the workplace. With the advent of the single accreditation system, the collaboration of the ACGME, AOA, and AACOM has resulted in the creation of osteopathic milestones for residency programs that apply for “osteopathic recognition” status within the ACGME.34 These osteopathic milestones incorporate osteopathic distinctiveness and competency-based assessment frameworks outlined for DOs in residency training. 
The ACGME’s milestones are intended for use in the formative assessment of learner development, and the NBOME’s competency domains with their required elements and outcomes are considered the behaviors, skills, values, and attitudes that all DOs are expected to demonstrate, and as such are for use as summative assessment. Based on work undertaken by AACOM,14 some alignment is possible between the competency domains, required elements and outcomes, Entrustable Professional Activities, and other formative assessments. As underscored in FOMCD 2016, the NBOME has attempted to harmonize the language and descriptions of competency domains, competencies, milestones, and Entrustable Professional Activities to facilitate alignment among learners both in undergraduate and graduate medical education. Despite these overall differences, the NBOME is seeking future opportunities for harmonization with the expectation that assessment of some attested or not-measured outcomes may someday contribute valuable information to physicians and trainees, state licensing boards, deans, and program directors. 
Conclusion
Now under the governance of NBOME’s COMLEX-USA Composite Examination Committee and its Board of Directors, final work on score reporting and continuance of systematic standard-setting processes are underway. The NBOME continues to study ways to limit construct-irrelevant variance in COMLEX-USA test scores, with appropriate attention to fairness and the paramount importance of protecting the public and enhancing the quality of patient care and public health. In this regard, the NBOME recognizes the educational effect that the COMLEX-USA series has in motivating students and residents to prepare in a manner that has educational benefit, as well as the catalytic effect such assessment can have in driving future learning and improving the care our patients receive. 
Acknowledgments
We thank Melissa Turner, MS, and Laura Barrett from the Strategy, Quality and Communications Department at the National Board of Osteopathic Medical Examiners, for their review and editorial assistance. 
References
Sandella JM, Gimpel JR, Smith LL, Boulet JR. The use of COMLEX-USA and USMLE for residency applicant selection. J Grad Med Educ. 2016:8(3):358-363. doi:10.4300/JGME-D-15-00246.1 [CrossRef] [PubMed]
Osborn GG, Meoli FG, Buser BR, Clearfield MB, Bruno JP, Sumner-Truax L. The Comprehensive Osteopathic Medical Licensing Examination, COMLEX-USA: a new paradigm in testing and evaluation. J Am Osteopath Assoc. 2000;100(2):105-111. [PubMed]
Blueprint Project Team. Blueprint ProjectQualifying Examinations Blueprint and Content Specifications. Ottawa, Ontario: Medical Council of Canada; 2014. http://mcc.ca/wp-content/uploads/Blueprint-Report.pdf. Accessed February 9, 2017.
Step 1. United States Medical Licensing Examination website. http://www.usmle.org/step-1/. Accessed February 9, 2017.
National Board of Osteopathic Medical Examiners. Residency program director’s guide to COMLEX-USA. https://www.nbome.org/docs/RPD.pdf. Accessed February 9, 2017.
Downing SM. Twelve steps for effective test development. In: Downing SM, Haladyna TM, eds. Handbook of Test Development. Mahwah, NJ: Lawrence Erlbaum; 2016.
Standards for Educational and Psychological Testing. 2nd ed. Washington, DC: American Educational Research Association; 2014.
Coalition for Physician Accountability. Consensus statement on a framework for professional competence by the Coalition for Physician Accountability. http://www.physicianaccountability.org/pdf/Coalition_Competencies_Consensus_Statement.pdf. Accessed February 9, 2017.
National Board of Osteopathic Medical Examiners. Standards for Quality Assurance. Chicago, IL: National Board of Osteopathic Medical Examiners; 2015.
Raymond M. Developing and Administering Practice Analysis Questionnaire [module]. Philadelphia, PA: National Council on Measurement in Education; 2005.
Knapp J, Knapp L. Practice analysis: building the foundations for validity. In: Impara JC, ed. Licensure Testing: Purposes, Procedures, and Practices. Lincoln, NE: Buros Institute of Mental Measurement; 1995:91-116.
National Board of Osteopathic Medical Examiners. The Seven Osteopathic Medical Competencies: Considerations for Future Testing and the Practice of Osteopathic Medicine. Chicago, IL: National Board of Osteopathic Medical Examiners; 2006. https://license.k3systems.com/LicensingPublic/docs/OsteopathicCompetenciesSeptember2006.pdf. Accessed February 16, 2017.
National Board of Osteopathic Medical Examiners. The Fundamental Osteopathic Medical Competencies: Guidelines for Osteopathic Medical Licensure and the Practice of Osteopathic Medicine. Chicago IL: National Board of Osteopathic Medical Examiners; 2009. https://www.aacom.org/docs/default-source/core-competencies/nbome6core-comp2009.pdf?sfvrsn=4. Accessed February 16, 2017.
Osteopathic Considerations for Core Entrustable Professional Activities (EPAs) for Entering Residency. Chevy Chase, MD: American Association of Colleges of Osteopathic Medicine; 2016. https://www.aacom.org/docs/default-source/med-ed-presentations/core-epas.pdf?sfvrsn=10. Accessed February 16, 2017.
The Royal College of Physicians and Surgeons of Canada. The CanMEDS 2005 Physician Competency Framework. Ottawa, Ontario: The Royal College of Physicians and Surgeons of Canada; 2005. http://www.ub.edu/medicina_unitateducaciomedica/documentos/CanMeds.pdf. Accessed February 16, 2017.
Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach. 2007;29(7):648-654. doi:10.1080/01421590701392903 [CrossRef] [PubMed]
Medical Council of Canada. Recalibrating for the 21st Century: Report of the Assessment Review Task Force of the Medical Council of Canada. Ottawa, Ontario: Medical Council of Canada; 2011. http://mcc.ca/wp-content/uploads/Reports-assessment-review-task-force.pdf. Accessed February 16, 2017.
National Board of Osteopathic Medical Examiners. The Fundamental Osteopathic Medical Competency Domains: Guidelines for Osteopathic Medical Licensure and the Practice of Osteopathic Medicine. Chicago, IL: National Board of Osteopathic Medical Examiners; 2011. https://www.nbome.org/docs/NBOME%20Fundamental%20Osteopathic%20Medical%20Competencies.pdf. Accessed February 16, 2017.
National Board of Osteopathic Medical Examiners. The Fundamental Osteopathic Medical Competency Domains: Guidelines for Osteopathic Medical Licensure and the Practice of Osteopathic Medicine. Chicago, IL: National Board of Osteopathic Medical Examiners; 2016. http://www.nbome.org/docs/Flipbooks/FOMCD/index.html#p=1. Accessed February 16, 2017.
Centers for Disease Control and Prevention. National Ambulatory Medical Care Survey: 2010 Summary Tables. Atlanta, GA: US Dept of Health and Human Services, Centers for Disease Control and Prevention; 2010. https://www.cdc.gov/nchs/data/ahcd/namcs_summary/2010_namcs_web_tables.pdf. Accessed February 16, 2017.
Centers for Disease Control and Prevention. National Hospital Ambulatory Medical Care Survey: 2010 Emergency Department Summary Tables. Atlanta, GA: US Dept of Health and Human Services, Centers for Disease Control and Prevention; 2010. http://www.cdc.gov/nchs/data/ahcd/nhamcs_emergency/2010_ed_web_tables.pdf. Accessed February 16, 2017.
National hospital discharge survey: data highlights - selected tables. Centers for Disease Control and Prevention website. http://www.cdc.gov/nchs/nhds/nhds_tables.htm. Accessed February 16, 2017.
Jones AL, Dwyer LL, Bercovitz AR, Strahan GW. The National Nursing Home Survey: 2004 overview. Vital Health Stat. 2009;13(167):1-155.
Boulet JR, Gimpel JR, Errichetti AM, Meoli FG. Using National Medical Care Survey data to validate examination content on a performance-based clinical skills assessment for osteopathic physicians. J Am Osteopath Assoc. 2003;103(5):225-231. [PubMed]
Langenau EE, Pugliano G, Roberts WL. Relationships between high-stakes clinical skills exam scores and program director global competency ratings of first-year pediatric residents. Med Educ Online. 2011;16:7362. doi:10.3402/meo.v16i0.7362
Langenau EE, Pugliano G, Roberts WL, Hostoffer R. Summary of ACOP (American College of Osteopathic Pediatricians) program directors’ annual reports for first-year residents and relationships between resident competency performance ratings and COMLEX-USA test scores. Electronic J Am Coll Osteopath Physicians. 2010;2(7).
Tenets of osteopathic medicine. American Osteopathic Association website. https://osteopathic.org/inside-aoa/about/leadership/Pages/tenets-of-osteopathic-medicine.aspx. Accessed February 16, 2017.
Joint AACOM & AODME 2015 conference program. American Association of Colleges of Osteopathic Medicine website. http://www.aacom.org/news-and-events/conference/past-conferences/2015-AC/program/wednesday. Accessed February 16, 2017.
COMLEX-USA master blueprint. National Board of Osteopathic Medical Examiners. https://www.nbome.org/docs/COMLEX_Master_Blueprint_2018-2019.pdf. Accessed February 16, 2017.
Level 1, level 2, and level 3 computer-based cognitive examinations. National Board of Osteopathic Medical Examiners website. http://www.nbome.org/comlex-cbt.asp#a1. Accessed March 1, 2017.
Farmer EA, Page G. A practical guide to assessing clinical decision-making skills using the key features approach. Med Educ. 2005;39(12):1188-1194. [CrossRef] [PubMed]
Medical Council of Canada. Guidelines for the Development of Key Features Problems & Test Cases. Ottawa, Ontario: Medical Council of Canada; 2012. http://mcc.ca/wp-content/uploads/CDM-Guidelines.pdf. Accessed February 16, 2017.
Milestones. Accreditation Council for Graduate Medical Education website. http://www.acgme.org/what-we-Do/Accreditation/Milestones/Overview. Accessed February 16, 2017.
Accreditation Council for Graduate Medical Education and the American Osteopathic Association. The Osteopathic Recognition Milestone Project. Chicago, IL: Accreditation Council for Graduate Medical Education; 2015. https://www.acgme.org/Portals/0/PDFs/Milestones/OsteopathicRecognitionMilestones.pdf. Accessed February 16, 2017.
Table 1.
NBOME Test Development Survey Outcomes: Ratings of Required Elements
Required Element Average Ratinga
Competency Domains: Decision Point One
  Commitment to continuous learning 4.51
  Ability to work effectively with other members of the interprofessional collaborative team 4.40
  Ability to gather data from all sources, including patients’ secondary sources, records, and physical examination results 4.37
  Ability to formulate a prioritized differential diagnosis 4.33
  Humanistic behavior 4.32
  Rapport building with patients 4.27
  Effective written and electronic communication 4.27
  Ability to give information to patients clearly, including collaboration on treatment, informed consent, and lifestyle modification 4.26
  Ability to counsel and educate patients consistent with their ability to understand 4.18
  Treatment of patients using best medical evidence 4.16
  Ability to develop a safe, evidence-based, cost-effective, patient-centered care plan 3.84
  Ability to perform basic clinical procedures essential for a general osteopathic medical practice 3.77
Clinical Presentations: Decision Point Two
  Ability to formulate a prioritized differential diagnosis 4.78
  Ability to gather data from all sources, including patients’ secondary sources, records, and physical examination results 4.70
  Ability to counsel and educate patients consistent with their ability to understand 4.68
  Ability to give information to patients clearly, including collaboration on treatment, informed consent, and lifestyle modification 4.68
  Rapport building with patients 4.66
  Ability to develop a safe, evidence-based, cost-effective, patient-centered care plan 4.63
  Treatment of patients using best medical evidence 4.60
  Commitment to continuous learning 4.59
  Ability to work effectively with other members of the interprofessional team 4.58
  Effective written and electronic communication 4.56
  Humanistic behavior 4.45
  Ability to perform basic clinical procedures essential for a general osteopathic medical practice 4.37

aItems were rated on a scale of 1 (“not important at all”) to 5 (“extremely important”).

Abbreviation: NBOME, National Board of Osteopathic Medical Examiners.

Table 1.
NBOME Test Development Survey Outcomes: Ratings of Required Elements
Required Element Average Ratinga
Competency Domains: Decision Point One
  Commitment to continuous learning 4.51
  Ability to work effectively with other members of the interprofessional collaborative team 4.40
  Ability to gather data from all sources, including patients’ secondary sources, records, and physical examination results 4.37
  Ability to formulate a prioritized differential diagnosis 4.33
  Humanistic behavior 4.32
  Rapport building with patients 4.27
  Effective written and electronic communication 4.27
  Ability to give information to patients clearly, including collaboration on treatment, informed consent, and lifestyle modification 4.26
  Ability to counsel and educate patients consistent with their ability to understand 4.18
  Treatment of patients using best medical evidence 4.16
  Ability to develop a safe, evidence-based, cost-effective, patient-centered care plan 3.84
  Ability to perform basic clinical procedures essential for a general osteopathic medical practice 3.77
Clinical Presentations: Decision Point Two
  Ability to formulate a prioritized differential diagnosis 4.78
  Ability to gather data from all sources, including patients’ secondary sources, records, and physical examination results 4.70
  Ability to counsel and educate patients consistent with their ability to understand 4.68
  Ability to give information to patients clearly, including collaboration on treatment, informed consent, and lifestyle modification 4.68
  Rapport building with patients 4.66
  Ability to develop a safe, evidence-based, cost-effective, patient-centered care plan 4.63
  Treatment of patients using best medical evidence 4.60
  Commitment to continuous learning 4.59
  Ability to work effectively with other members of the interprofessional team 4.58
  Effective written and electronic communication 4.56
  Humanistic behavior 4.45
  Ability to perform basic clinical procedures essential for a general osteopathic medical practice 4.37

aItems were rated on a scale of 1 (“not important at all”) to 5 (“extremely important”).

Abbreviation: NBOME, National Board of Osteopathic Medical Examiners.

×
Table 2.
Weighting for COMLEX-USA Competency Domains and Clinical Presentations
Dimensions Minimum Weighting, %
Competency Domains: Dimension One
  Osteopathic Principles, Practice, and Manipulative Treatment 10
  Osteopathic Patient Care and Procedural Skills 25
  Application of Knowledge for Osteopathic Medical Practice 30
  Practice-Based Learning and Improvement in Osteopathic Medical Practice 5
  Interpersonal and Communication Skills in the Practice of Osteopathic Medicine 10
  Professionalism in the Practice of Osteopathic Medicine 5
  Systems-Based Practice in Osteopathic Medicine 5
Clinical Presentations: Dimension Two
  Community Health and Presentations Related to Wellness 12
  Patient Presentations Related to:
    Human Development, Reproduction, and Sexuality 5
    Endocrine System and Metabolism 5
    Nervous System and Mental Health 10
    Musculoskeletal System 13
    Genitourinary System 5
    Gastrointestinal System and Nutritional Health 10
    Circulatory and Hematologic Systems 10
    Respiratory System 10
    Integumentary System 5

Abbreviation: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA.

Source: Reprinted with permission from the National Board of Osteopathic Medical Examiners.

Table 2.
Weighting for COMLEX-USA Competency Domains and Clinical Presentations
Dimensions Minimum Weighting, %
Competency Domains: Dimension One
  Osteopathic Principles, Practice, and Manipulative Treatment 10
  Osteopathic Patient Care and Procedural Skills 25
  Application of Knowledge for Osteopathic Medical Practice 30
  Practice-Based Learning and Improvement in Osteopathic Medical Practice 5
  Interpersonal and Communication Skills in the Practice of Osteopathic Medicine 10
  Professionalism in the Practice of Osteopathic Medicine 5
  Systems-Based Practice in Osteopathic Medicine 5
Clinical Presentations: Dimension Two
  Community Health and Presentations Related to Wellness 12
  Patient Presentations Related to:
    Human Development, Reproduction, and Sexuality 5
    Endocrine System and Metabolism 5
    Nervous System and Mental Health 10
    Musculoskeletal System 13
    Genitourinary System 5
    Gastrointestinal System and Nutritional Health 10
    Circulatory and Hematologic Systems 10
    Respiratory System 10
    Integumentary System 5

Abbreviation: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA.

Source: Reprinted with permission from the National Board of Osteopathic Medical Examiners.

×