The institutional review board considered this research exempt from approval. 

The first- and second-year classes (classes of 2019 and 2018, respectively) at RVUCOM both contained approximately 160 students at the time of the study. Each class participated in a set of 6 ultrasonography laboratory sessions during the 2015-2016 academic year. First-year students focused on ultrasonography physics, image acquisition, and interpretation of normal anatomy as it related to the Gross Anatomy systems blocks and dissections (Table 1; Figure). Second-year students had the same learning objectives; however, the content was delivered with a greater focus on clinical application in a case-based discussion format as integrated into the Principles of Clinical Medicine course (Table 1). For this inaugural year of NUSMED, the second-year students had not had any formal ultrasonography training and, therefore, focused on normal anatomy. In the future, second-year students will have completed the introduction during the first year and will have a greater emphasis on pathologic conditions and clinical applications. 

The NUSMED curriculum used a flipped classroom model in which students completed precourse assignments that incorporated instructional videos showing clinical relevance of the examination, probe placement, examination mechanics, and video clips of both normal anatomy and pathologic conditions before in-person hands-on sessions. The video learning modules were created by ultrasonography instructors at the University of Colorado School of Medicine (CUSOM) and are also used in the CUSOM USMED curriculum. The students then came to the laboratory sessions where they practiced image acquisition on each other. 

Each NUSMED laboratory session took place in a large, multipurpose room equipped with adjustable examination tables and 17 ultrasonography machines (Chison ECO5). An 8.5-foot projector screen was located at the front of the room, with 8 fifty-inch monitors (Sharp Aquos) dispersed throughout the room and synchronously showing the projected images. The projector (Christie LX380) was equipped for live video feed from the presenter's computer, a video camera (Datavideo EDI-HD1) and an ultrasonography machine (Chison Model X). The instructor was thus able to switch between projecting a didactic presentation with lecture content and pathologic images, the video camera to show probe placement and examination technique, and a live feed from the ultrasonography screen using a switcher (Extron ICP PRO 550 Switcher). Students in each class were divided into 2 two-hour sessions, such that there were 80 students in each session and 4 to 5 students scanning on each machine. The instructor reviewed the didactic content and then demonstrated the ultrasonography modality at the podium. Students then practiced at each station while the instructor was able to move about the room and answer individual questions. 

To assess ultrasonography knowledge, multiple choice examination questions were incorporated into the Principles of Clinical Medicine course examination. First-year students answered 9 questions, and second-year students answered 5. Questions focused on basic structure identification, asking students to identify a portion of the structures listed in each section's learning objectives, ultrasonography physics and probe selection, and clinical relevance of images. The questions were probing for mastery of material in the form of image identification, interpretation, and technique. For example, students were provided with a still image and asked to identify structures in the image, were given a clinical scenario and asked to identify which probe to use, or were given an example of poor image quality and asked to identify how to improve the image. Examination questions were administered on ExamSoft testing software by the Rocky Vista University testing center. There was not a separate NUSMED final examination. 

Faculty teaching hours were defined as hours spent during onsite student instruction, and preparation hours were defined as hours spent preparing content and learning slides for sessions (Table 2). Time commuting to and from the educational site from clinical duties was not included. 

Data collection for hours spent in NUSMED's curriculum was achieved through the primary individual instructor's tracking and reporting. Comparison teaching hours for a traditional teaching format were collected from published hours reported in a traditional USMED program at CUSOM. As it has been determined that a 1:8 faculty-to-student ratio is the minimum necessary to implement an effective ultrasonography curriculum for medical students,¹⁰ we adapted this guideline to fit our class size of 80 students per group and extrapolated the anticipated hours. Comparison preparation hours were set at 1 hour per instructor per scanning session, presuming that each instructor spent approximately 45 minutes watching the modules and an additional 15 minutes reviewing relevant anatomy. Raw data were tabulated and totaled using Microsoft Excel. 

Given that qualitative student feedback was our primary method for evaluating efficacy, standards for quantitative results and validity were difficult to apply. We instead applied principles adapted from Guba and Lincoln¹¹ to assess qualitative validity and credibility, transferability, dependability, and confirmability. Credibility establishes that the research participants can believe the results. In our case, these were the students who participated in the sessions and reported their feedback. Because we quoted some students’ comments directly, we presume that they would find them believable. Transferability refers to the degree to which our results could be generalized to other contexts or settings. For our results to be transferrable to another setting, that other setting would need to involve preclinical medical students, limited instructor resources, and access to technological resources necessary to use this curricular model. To the degree that these characteristics were present, we believe that our results would transfer. Dependability refers to the ever-changing environment within which any extended activity occurs. Because we collected data retrospectively, we had no opportunity to make use of the data and adapt and adjust the program midstream to student feedback. Thus, this concept is difficult to apply and assess in the scope of the current article. Confirmability refers to the degree to which our results could be confirmed or corroborated by others, and we believe our findings can be reproduced. 

We required a total of 12 days and 24 two-hour scanning sessions to teach 320 first- and second-year medical students (160 first- and 160 second-year students) ultrasonography. All 320 students took the test, and 82 completed the course evaluation. We used 1 primary ultrasonography instructor for all NUSMED laboratories, requiring a total of 96 hours (Table 2), which can be compared to the 600 hours extrapolated to fit a 1:8 instructor to student ratio (Table 2). 

Eighty-two students completed anonymous qualitative course evaluations at the end of each semester. They were asked to evaluate the instructor, course content, and written examinations. Our initial feedback showed common themes of student appreciation for early exposure to ultrasonography and repetition of integrating hands-on techniques with concurrent anatomy covered in the systems courses. For example, responses included “helped reinforce the material in a real-time hands-on 3D representation” and “it's a helpful addition to our curriculum. I'm relieved that we will be able to draw on this experience when we begin clinical rotations.” Another student wrote: “Excellent. Definitely keep it as part of the systems courses as it complements our learning and enhances our understanding. It is also a highly applicable technical skill to possess for our future careers.” Preliminary feedback also suggested that a significant limitation in NUSMED is a lack of hands-on/in-person teaching as the instructor was roving about the room: 47.5% (39/82) of students expressed an overwhelming desire for smaller groups and shorter sessions to allow more time for individual practice and 27.9% (23/82) of students requested additional instructors/teaching assistants to provide more direct feedback and instruction. For example, “I think the US labs are great and correlate well with our other courses, however, I wish that we had more proctors in the room to help us identify what we are seeing on our screens. In the 2 hours, we usually only get 2 proctors to come back for about 10 seconds each and the rest of the lab are guessing as to what we are seeing.” Another student wrote: “I'm really grateful to have ultrasound built into our curriculum, but I do think that we need more table trainers in order for the lab to be successful. The class sizes are just too large for her to manage, even with the help that she does have.” Finally, “I appreciate the hands-on aspects and being able to apply it to cases and the material we study in each unit. If the material could be covered in a one-hour session, I believe I would benefit more from the lab, be able to spend more time using the equipment, and have more time to study for the course.” 

The first-year class answered 9 questions, and, across the question set, 88% (140.9/160) of the students answered questions correctly. The lowest-performing question was answered correctly 66% of the time, and the highest, 99% of the time. The second-year class answered 5 questions, and, across the question set, 91% (145.6/160) of the students answered the questions correctly. The lowest-performing question was answered correctly 71% of the time, and the highest, 99% of the time. 

The use of a single instructor teaching 80 medical students during 1 scanning session represents a novel method in osteopathic USMED. Overall, we were successful in saving time and faculty resources, and students enjoyed their exposure to ultrasonography and performed well on written examinations. Compared with a traditional USMED curriculum, which requires an extrapolated 600 instruction hours annually, NUSMED required only 96 hours. Although students reported appreciating exposure to ultrasonography, they expressed desire for smaller group and greater individual instruction. Written test questions performed well, with the first-year class answering 88% correctly and the second-year class answering 90.6% correctly. 

Teaching modalities of point-of-care ultrasonography frequently include didactics, web-based modules, live demonstrations, and short videos in addition to hands-on scanning sessions.¹⁰ While didactics, modules, and live demonstrations can effectively introduce ultrasonography principles, hands-on sessions are necessary to gain skill.^12-14 We attempted to incorporate these methods into our curriculum. We feel this method of curriculum design has the potential to be adapted and used in other settings that may lack instructional resources, such as other new medical schools. Live ultrasonography demonstrations could even be done remotely by experienced educators, with peer instructors available to lead small groups in person. 

In our review of the literature on assessment of knowledge, the most frequent method of evaluation was self-assessment by the learner, with results showing high satisfaction and an increase in knowledge and technical skills.¹⁰ Our student feedback was corroborative, with most students reporting that they enjoyed their exposure and felt they were learning valuable clinical skills. The second most common method of evaluation involved image acquisition and technical skill observation, which is discussed in detail below. Further direct comparison of our curricular effectiveness to others is difficult because of the wide variety of curricula structure and assessment methods; however, we maintain that our curriculum may be appropriate for other settings lacking limited instructor resources. 

We considered a cost analysis to quantify cost savings of an extrapolated annual 600 instruction hours compared with our use of 96 hours, but the regional and institutional variations in salary and reimbursement make this difficult. It can reasonably be inferred that the significant reduction in teaching hours represents our cost analysis and translates into intangible monetary savings. It would be important for individual institutions to take stock of their available technical resources before implementing a program like this. At RVUCOM, we were able to use equipment that had been purchased for our osteopathic manipulative treatment sessions, as a similar curricular modality with flipped classroom, live demonstrations, and hands-on practice is used during those sessions. 

However, NUSMED had several limitations and uncovered areas for future research and improvement. First, our course evaluations were limited to commentary rather than being quantitative. This format did not facilitate standardization of feedback, as there were no specific evaluation points. For example, 27.9% of students reported desire for more instructors and 47.5% of students requested shorter laboratory sessions and smaller group sizes. We suspect that a greater proportion of students felt this way, but they did not provide commentary in the write-in section. In the future, ultrasonography sessions will be evaluated using a Likert scale that assesses the instructor, precourse videos, session content, and whether the learning objectives were achieved. Second, we evaluated the efficacy of this educational format on written examinations; however, we were not able to evaluate the students’ ability to acquire and interpret images. Ideally, ultrasonography performance should be evaluated under direct observation of skills. However, this would require significant additional faculty resources. As this NUSMED program expands, we hope to integrate these additional methods of evaluation. 

Another significant limitation in the curriculum is students’ desire for individual instruction during scanning sessions. While the instructor could deliver didactic content using the flipped classroom and demonstrate scanning skills to the group, it was difficult for her to provide direct feedback and guidance at each station. To this end, we are developing a peer instructor model for future sessions that allows upper-level students to assist with instruction in the hands-on scanning sessions to produce a smaller instructor-to-student ratio. Peer-assisted course models have been effective in delivering undergraduate ultrasonography education¹⁵ and would allow us to increase available instructors while maintaining a single-instructor model. Other studies have also shown that POCUS skills and understanding did not differ between students taught by faculty or peer instructors.^16-18 

Students in the current study also reported that laboratory sessions were too lengthy and there were too many students per group. They felt that downtime allowed for off-task activities like studying for other classes or talking among themselves. One strength of our open commentary method of evaluation was that students felt free to leave constructive criticism and advice about their experience with the curriculum. While approximately 48% of students requested smaller class sizes, 70% of these students directly suggested that along with a smaller class size, ultrasonography laboratories should be reduced to 1 hour rather than 2. They felt that this time would provide (1) a straightforward way of reducing group sizes and increasing time spent using the ultrasonography machines, (2) immediately increase the instructor-to-student ratio, and (3) keep students on task, as class could move more quickly with fewer students needing a turn on the probe. As we move into future curriculum development, smaller class sizes and shorter laboratory periods will be a priority. 

Given the limitations found in real-time feedback on scanning technique and evaluation of students’ ability to acquire and interpret images, NUSMED may not be best implemented during the clinical years of medical school. However, as an introductory ultrasonography course for students in the preclinical years, we found that students had an overall positive experience and demonstrated that they had acquired basic ultrasonography knowledge. For more advanced learners or health care providers who will be using ultrasonography clinically, a more traditional instruction model is likely a better fit. 

We acknowledge John Kendall, MD, and David Ross, DO, for their contributions to the editing process.