Entrustable professional activities were introduced as a novel outcomes-based model for assessing essential, observable work tasks that define a specialty or profession.³ They are described as tasks observed in daily practice rather than specific knowledge, skills, and attitudes assessed in isolation. For example, in an oral case presentation, a trainee may require excellent skills in eliciting a history, performing a physical examination, determining a differential diagnosis, developing a plan, and communicating with others. While each of these individual skills can be assessed, the task and how the various knowledge, skills, and so on are used can vary depending on the workplace context (eg, outpatient vs emergency department) and a number of other factors. The overall performance on the task and all affiliated competencies measured requires documentation in a variety of patient contexts and settings to qualify as an entrustable activity. 

The key principles underlying EPA assessment and decision making (ie, workplace learning and trust) are generalizable to the continuum of physician training.⁴ EPAs offer medical educators a workplace-relevant approach to the assessment of physicians-in-training. At the same time, the structuring of EPAs requires the development of new measures capable of addressing the underlying elements of entrustment decision making. Entrustment decisions must be based on clear definitions of trust, specific types and levels of targeted entrustments, and explicit definitions of what each development stage means in terms of moving students from novice learners to physicians capable of providing safe, unsupervised, and professional care.^4,5 

In the process of analyzing EPAs for use in medical education, researchers have identified several challenges, including the need for valid and reliable EPA assessment strategies.^6,7 Assessment guidelines outlined by ten Cate et al⁵ call for the use of targeted assessment activities that are consistently observable, measurable, time-limited, and geared toward practice-relevant tasks suitable for entrustment. If assessments are not designed in this manner, validity and reliability may prove challenging.⁸ 

In this article, we provide an overview of existing assessment strategies that can be used as a basis for conceptualizing potential approaches to the assessment of EPAs in osteopathic medical education. The aim is to assist osteopathic medical schools with the development of robust, evidence-based assessment methods and encourage the design and implementation of innovations in EPA assessment. 

During an initial meeting of the AACOM EPA Assessment Planning Subcommittee in August 2016, the group identified the lack of tools to support assessment planning as a critical gap in resources. Members felt that several types of information could be assembled to assist osteopathic medical schools with the development of robust, evidence-based assessment tools. The group agreed that some of the existing research on clinical assessment could prove valuable in supporting the design and customization of EPA assessment methods initiated across campuses. A decision was made to research the evidence in support of EPA-related assessment tool development and implementation with a focus on EPA 1 through EPA 6 because a higher level of entrustment should be targeted for these EPAs before entry into clinical training for all medical students. The other EPAs, although important, may require a lower level of entrustment or be assessed on a smaller scale before entry into clinical training. Each subcommittee member investigated assessment approaches affiliated with competencies embedded within 2 different EPAs and worked with his or her campus librarian to conduct the search of major publication databases. 

Three overarching questions were used to guide the literature review: 

Using advice from the academic librarians, the research team developed a strategy to search MEDLINE, ERIC, PubMed, and other relevant databases. MeSH keywords included sets of EPA-related terms such as “history,” “medical history taking,” “students, medical,” and “educational measurement,” among others (eAppendix 1). Articles with “Entrustable Professional Activities,” “EPAs,” or EPA-related competency wordings in the title were also included. All searches were conducted between August 2016 and March 2017. While the scope of the search was primarily focused on the most recent 10-year timeframe, several relevant articles that fell outside of this timeframe provided useful information related to EPA 2 and EPA 4 and were, therefore, included in the results. Articles were excluded from the review if the following was true: the article originated from a nonmedical field (eg, veterinary, pharmacy, dentistry), the full text was not accessible, the focus of the article was on how to design and implement an EPA rather than on how to assess learner entrustability on EPA-related competencies, the article assessed curricula or practitioners rather than learners, an English version of the article could not be obtained, it described a self-assessment solely (not including self-assessments used as part of a multisource feedback or objective structured clinical examination), or the article described the process used to develop the tool only. Also excluded from the review were editorials, commentaries, interviews, debates, and book reviews. The researchers then revisited the searches in January 2018 to update the findings before final publication. 

The group systematically searched the key journals and publication databases. To ensure consistency and accuracy during the data collection process, the subcommittee developed a classifying system to extract the most relevant information from each article. The categories and categorical elements were selected from analysis of pertinent literature, conference presentations, committee discussion, etc. The classification system ultimately included the following categories: assessment category, EPA relevance, affiliated competency domains, level of learner, targeted specialty area(s), skills assessed, assessment type, feedback mechanisms, content validity, internal structure, and type of internal structure (Figure). Articles were not placed into the classification system/database until they had met the selection criteria. 

Although most journals examined in this review clearly stated each article's primary topic in the title, abstract, or key words, every article was reviewed in detail by subcommittee members with expertise in medical education research. Each reviewer (M.L., L.W., M.S., J.P., S.M., and E.K.) undertook a primary review of competencies embedded in 2 different EPAs and independently coded the data using the classification categories. Each reviewer also served as a secondary reviewer and cross-checked the codings of the primary reviewer. Any disagreements were resolved by discussion, consensus, and consultation with a third member of the review team. Finally, the researchers perused the Association of American Medical Colleges’ Toolkits for the 13 Core EPAs⁹ to ensure that key assessment tools were not missed in this compilation. 

A total of 55 articles were included in the content analyses and reporting. All but 1 of the articles were conducted in an undergraduate or graduate medical education setting. Although our review focused on empirical articles, important conceptual and theoretical findings were included in the reporting to enhance understanding of the theoretical and conceptual frameworks that form the basis for the development of EPA assessment tools. 

The majority of articles were related to the assessment of competencies affiliated with EPA 2 (16 articles [29.1%]) and EPA 4 (15 articles [27.3%]). Four articles (7.3%) focused on EPA 3-related competency assessment, 6 (10.9%) on EPA 1, 8 (14.5%) on EPA 5, and 6 (10.9%) on EPA 6. A majority of the articles targeted the assessment of medical students (34 [61.8%]). In contrast, 12 (21.8%) related to the assessment of residents, 7 (12.7%) of both medical students and residents, and 2 (3.6%) did not specify the learner level. 

All but 3 articles focused on 2 or more competency areas. Learner competency related to patient care was the focus of assessment in all of the articles except 3. At least half of the articles targeted interpersonal and communication skills and/or medical knowledge, and slightly fewer than half targeted practice-based learning and improvement. The 2 competencies addressed the least were systems-based practice and professionalism. 

All but 3 articles identified the type of assessment tool used to measure learner competency. Assessment tools most frequently reported in the literature were knowledge tests (n=21), simulation (n=16), and assessment rubrics (n=11). In contrast, product evaluations (n=7), long practice observations (n=3), short practice observations (n=2), and case-based discussions (n=2) were least often identified. The reviewers noted inconsistencies in the reporting of feedback to learners and inconsistencies in the provision of reliability and validity evidence across articles. In most instances, it was not clear how (or whether) postassessment feedback was provided to learners. When the feedback method was mentioned, written feedback was the most commonly used approach. Twenty-three articles included a description of how instrument reliability was determined, and 15 provided validity evidence. 

Although the type of response scale was not itemized in our final report, we noted that response scales embedded with the various assessment tools varied considerably, within a wide range of behavioral anchors, dichotomous measures (eg, yes/no; performed/didn't perform), and agreement, satisfaction, and/or confidence scales represented. This finding highlighted to the subcommittee the importance of selecting appropriate response scales when the focus of assessment is intended as an entrustment decision. Entrustment scales will be needed to frame assessment results in terms of the type and designated level of supervision required. In this sense, the language of entrustment scale construction represents a shift in approach from competency-based “person-descriptors” to EPA-based “work-descriptors” as described by ten Cate et al,⁶ who suggested using 5 levels of supervision to reflect increasing trust in trainee autonomy. 

The subcommittee encountered several entrustment and co-activity scales commonly found in the undergraduate and graduate medical education literature (Table 1) but not yet widely reported in research on EPA-related competency assessment.^4,5,10 Additionally, the toolkits document⁹ provides a thorough review of scales and suggested modifications to scales to better fit the undergraduate medical education system. As mentioned previously, the content and format of EPAs represent a distinct shift toward more authentic assessment of work-based skills. During the design and development phase, the existing literature can be used to help map out EPAs as shown in Table 2 using an EPA description and EPA matrix.^6,11 A detailed description of all 55 articles included in this review are shown in eAppendix 2. 

This literature review provides insight into key structural elements and processes identified as important in the assessment of EPA-related competencies. Taken together, the results indicate that several challenges should be taken into account when designing and assessing EPAs: 

Many experts advocate for matrix-type assessment planning that maps EPAs to competency frameworks and milestones (matched to appropriate levels of entrustment).^6,30 Narrative statements included as descriptors can also be used to guide decisions about entrustment level. These approaches can serve as a mechanism for feedback and reporting by providing a common language or map for discussing learner performance and progress over time. 

We noted several limitations related to this review. For example, literature related to the assessment of EPAs is still in its infancy, making it difficult to support findings with previously published research. The lack of an established set of EPA-related search terms mandated the use of a relatively complex set of search algorithms, which may require refinement as research in this area advances. While a wide range of articles were located, the diversity of study populations and methodologies, represented in the various articles, made the meaningful comparison of instrument psychometric properties difficult. The results of this review may be outdated quickly as assessment of EPAs progress and more articles are published. Therefore, additional searches will be necessary to locate emergent findings. It is also possible that some exclusion criteria (especially limiting to the medical field) may have resulted in oversight of relevant assessments. 

Future EPA assessment subcommittee research will focus on assessment tools affiliated with EPA 7 through EPA 13 and refining search terms to better identify articles (eg, filtering out articles related to specific tests and finding articles related to the ability to recommend or interpret appropriate tests for EPA 3). The committee will develop similar MeSH terms for EPA 7 through EPA 13. The MeSH terms developed and used in this review (eAppendix 1) can be applicable to institutions interested in continuing to search and develop EPA-related assessment tools going forward. A continued goal of the subcommittee research will be to create practical tools for use by medical schools interested in developing assessments that uniquely align with their organizational contexts and settings. 

This article provides a unique frame of reference by summarizing the limited research literature to date. Given that the core set of medical schools charged with EPA piloting have not yet completed their initial piloting process nor extensively published on EPA assessment tool development, it is the hope of the subcommittee that this information will provide a useful benchmark against which future EPA assessment research and reporting can be measured. 

A review of the extant literature over the past year allowed the AACOM EPA Assessment Planning Subcommittee to denote key gaps in the existing literature, as well as identify several considerations worth keeping in mind when designing assessments for use in entrustment decision making. In addition to the current findings, educators should note that institutional context will be an important covariant in determining the effectiveness of EPA assessment in osteopathic medical education because each institution has a different mission, curricular resources, and support structures. Therefore, this article may assist institutions by providing a baseline set of assessment tools that can be used to guide the development of EPA-specific assessment plans. Each medical school is urged to consider how the tools presented in this article can be adapted to meet the needs of its learners. The featured articles can also be reframed to address key entrustment decisions using deliberately assigned levels of supervision. Measures should address not only ability, but also integrity, reliability, and humility.³¹ Medical educators may find it useful to implement several of the entrustment scales (Table 1)^4,5 and EPA planning tools (Table 2)^6,11 as a way of ensuring that all major decision points and assessment components are taken into consideration. By developing multiple measures and making consistent use of entrustment scales, instruments will be better able to document evidence of progress toward summative entrustment decisions. 

We acknowledge Stephen C. Shannon, DO, MPH, president of the American Association of Colleges of Osteopathic Medicine, for his vision and support in advancing the EPA initiative, as well as Mary Essig, MLS, library director at the West Virginia School of Osteopathic Medicine in Lewisburg; Rebecca Hines, MLKS, education librarian and assistant professor at Des Moines University in Iowa; and Kelli Hines, MLIS, librarian at Western University of Health Sciences in Pomona, California, for their insights, advice, and efforts in the literature search process. 

MeSH (Medical Subject Heading) terms: strategies to guide entrustable professional activity (EPA) assessment tool searches for EPAs 1 through 6. ^aEPA 3 will take more time because diagnostic tests comprise a broad category. 

(“history and physical”[tw] OR “physical exam”[tw] OR “physical examination”[tw] OR “physical examination”[MeSH Terms] OR “history taking”[tw] OR “medical history taking”[MeSH Terms] OR “interviews as topic”[MeSH Terms]) AND (“Schools, Medical”[MeSH] OR “Students, Medical”[MeSH] OR “Education, Medical, Undergraduate”[MeSH] OR “Clinical Clerkship”[MeSH]) AND (“Curriculum/standards”[MeSH] OR “Educational Measurement”[MeSH] OR instrument[tw] OR rubric[All Fields] OR tool[tw] OR “checklist”[MeSH Terms] OR “checklist”[All Fields]) AND (“Validation Studies”[Publication Type] OR reliability[tw] OR validate[tw] OR validity[tw] OR validation[tw] OR “Reproducibility of Results”[MeSH Terms]) 

(“diagnosis”[MeSH] OR “Diagnosis, Differential”[MeSH] OR diagnosis[tw] OR “Clinical Examination”) AND (“Schools, Medical”[MeSH] OR “Students, Medical”[MeSH] OR “Education, Medical, Undergraduate”[MeSH] OR “Clinical Clerkship”[MeSH]) AND (“Curriculum/standards”[MeSH] OR “Educational Measurement”[MeSH] OR instrument[tw] OR rubric OR tool[tw] OR checklist) AND (“Validation Studies” [Publication Type] OR reliability[tw] OR validate[tw] OR Validity[tw] OR validation[tw] OR “Reproducibility of Results”[MeSH Terms]) 

(“Diagnosis, Differential”[MeSH] OR diagnosis[tw] OR “Clinical Examination”) AND (“Schools, Medical”[MeSH] OR “Students, Medical”[MeSH] OR “Education, Medical, Undergraduate”[MeSH] OR “Clinical Clerkship”[MeSH]) AND (“Curriculum/standards”[MeSH] OR “Educational Measurement”[MeSH] OR instrument[tw] OR rubric OR tool[tw] OR checklist) AND (“Validation Studies” [Publication Type] OR reliability[tw] OR validate[tw] OR validity[tw] OR validation[tw] OR “Reproducibility of Results”[MeSH Terms]) 

“Diagnostic Techniques and Procedures”[MeSH] AND (“Schools, Medical”[MeSH] OR “Students, Medical”[MeSH] OR “Education, Medical, Undergraduate”[MeSH] OR “Clinical Clerkship”[MeSH]) AND (“Curriculum/standards”[MeSH] OR “Educational Measurement”[MeSH] OR instrument[tw] OR rubric OR tool[tw] OR checklist) AND (“Validation Studies” [Publication Type] OR reliability[tw] OR validate[tw] OR validity[tw] OR validation[tw] OR “Reproducibility of Results”[MeSH Terms]) AND “humans”[MeSH Terms] 

(prescribing[tw] OR “Drug Therapy”[MeSH] OR “Medication Errors”[MeSH] or pharmacology[tw] OR “Pharmacology, Clinical”[MeSH] OR “Pharmacology/education”[MeSH]) AND (“Schools, Medical”[MeSH] OR “Students, Medical”[MeSH] OR “Education, Medical, Undergraduate”[MeSH] OR “Clinical Clerkship”[MeSH]) AND (“Curriculum/standards”[MeSH] OR “Educational Measurement”[MeSH] OR instrument[tw] OR rubric OR tool[tw] OR checklist) AND (“Validation Studies” [Publication Type] OR validate[tw] OR validity[tw] OR validation[tw] OR reliability[tw] OR “Reproducibility of Results”[MeSH Terms]) 

(Documentation[tw] OR “Health record”[tw] OR “Medical record”[tw] OR “soap note”[tw] OR “computerized provider order entry”) AND (“Schools, Medical”[MeSH] OR “Students, Medical”[MeSH] OR “Education, Medical, Undergraduate”[MeSH] OR “Clinical Clerkship”[MeSH]) AND (“Curriculum/standards”[MeSH] OR “Educational Measurement”[MeSH] OR instrument[tw] OR rubric OR tool[tw OR checklist]) AND (“Validation Studies” [Publication Type] OR validate[tw] OR validity[tw] OR validation[tw] OR reliability[tw] OR “Reproducibility of Results”[MeSH Terms]) 

(“case presentation” OR “oral presentation” OR “patient presentation”) AND (“Schools, Medical”[MeSH] OR “Students, Medical”[MeSH] OR “Education, Medical, Undergraduate”[MeSH] OR “Clinical Clerkship”[MeSH]) AND (“Curriculum/standards”[MeSH] OR “Educational Measurement”[MeSH] OR instrument[tw] OR rubric OR tool[tw] OR checklist) AND (“Validation Studies” [Publication Type] OR validate[tw] OR validity[tw] OR validation[tw] OR reliability[tw] OR “Reproducibility of Results”[MeSH Terms]) 

Articles to support entrustable professional activity (EPA) assessment tool development (as of January 2018) by EPA. ^aBrinkman et al mention these measures but no results were shown to verify reliability. Abbreviations: GME, graduate medical education; NA, not available/applicable; MSF, multisource feedback; NI, not included; OSCE, objective structured clinical examination; SOAP, subjective, objective, assessment, plan; SP, standardized patient; UME, undergraduate medical education.