Telemedicine training was not a substantial element of most residency programmes prior to the COVID-19 pandemic. Social distancing measures changed this. The Cleveland Clinic Internal Medicine Residency Programme (IMRP) is one of the largest programmes in the USA, which made the task of implementing a telemedicine curriculum more complex. Here we describe our experience implementing an effective, expedited telemedicine curriculum for our ambulatory resident clinics. This study was started in April 2020 when we implemented a resident-led curriculum and training programme for providing ambulatory telemedicine care. The curriculum was finalised in less than 5 weeks. It entailed introducing a formal training programme for residents, creating a resource guide for different video communication tools and training preceptors to safely supervise care in this new paradigm. Residents were surveyed before the curriculum to assess prior experience with telemedicine, and then afterward to assess the curriculum’s effectiveness. We also created a mini-CEX assessment for residents to solicit feedback on their performance during virtual appointments. Over 2000 virtual visits were performed by residents in a span of 10 weeks. Of 148 residents, 38% responded to the pre-participation survey. A majority had no prior telemedicine experience and expressed only slight comfort with the modality. Through collaboration with experienced residents and faculty, we expeditiously deployed an enhancement to our ambulatory care curriculum to teach residents how to provide virtual care and help faculty with supervision. We share our insights on this experience for other residency programmes to use.
- medical education & training
- internal medicine
This article is made freely available for personal use in accordance with BMJ’s website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.https://bmj.com/coronavirus/usage
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Purpose of study
The first documented cases of COVID-19 in Ohio occurred on 9 March 2020.1 One day later, the Cleveland Clinic’s leadership announced that all in-person meetings were cancelled and ambulatory patient visits would be virtual. This was a major paradigm shift for our Internal Medicine Residency Programme (IMRP), where one-third of training time is spent in ambulatory clinics. Teaching residents at one of the largest programmes in the country to provide virtual visits on short notice was an enormous endeavour. In addition, we needed to make sure that all faculty had the tools they needed to precept residents in a completely virtual format.
A cohort of Cleveland Clinic faculty physicians have been performing virtual patient visits since 2014 using the American Well platform.2 They conducted 41 000 virtual visits in 2019 and over 100 000 such visits by February 2020.3 Residents, however, had not been trained in delivering virtual care prior to the COVID-19 pandemic. In any given week, our IMRP residents complete around 840 ambulatory patient encounters. The avoidance of in-person visits for routine care as a result of social distancing measures would have deprived our community of considerable primary care, and it would have limited learning opportunities for residents.
Our IMRP’s clinician-educator track (CET) residents began building a telemedicine curriculum before the start of the pandemic. The original plan was for a slow, stepwise rollout of the curriculum with the new interns in fall 2020. However, the urgent need created by the pandemic called for an accelerated implementation of this nascent telemedicine curriculum.
The Accreditation Council for Graduate Medical Education (ACGME) had planned to integrate telemedicine into the Common Program Requirements in July 2020. Considering the pandemic, precepted telemedicine encounters were permissible immediately.4 Our telemedicine curriculum allowed us to quickly train residents and preceptors to serve our community with virtual visits. We first assessed resident confidence in providing telemedicine care before the curriculum started in April 2020. In June and July 2020, we surveyed residents regarding the effectiveness of the curriculum. These surveys were determined to be exempt by the Cleveland Clinic Institutional Review Board (IRB).
Our ambulatory sites accommodate 148 residents and vary in size and patient populations. Residents function as primary care physicians during their clinic week (4+1 model) in nine different clinic sites. Implementing this programme involved preparing and standardising clinic workflow documents for residents and a precepting guide for faculty across the different ambulatory sites (online supplemental appendix A). We created an orientation presentation to standardise virtual visits inspired by external resources from the American College of Physicians (ACP), the Ohio State Medical Association (OSMA) and the ACGME. Telemedicine training for residents was conducted virtually by Zoom, a practice that has been widely used at other institutions to implement virtual curricula.5
Trainee pre-participation and post-participation surveys
Residents were surveyed before and after the telemedicine curriculum. The pre-participation survey was distributed by email starting in early April 2020 and asked about prior telemedicine experience and comfort level in providing telemedicine care (online supplemental appendix B). The post-participation survey was sent by email in June and July 2020 at the end of the academic year. It asked for feedback on resident satisfaction with and benefit from the telemedicine curriculum (online supplemental appendix C). Surveys were anonymous and not matched.
We considered it paramount to teach our residents how to communicate and perform remote examinations with virtual tools, which require unique communication skills.6 Residents watched a recording of a simulated encounter prior to their first real encounter (online supplemental appendix D). This was done at the start of each ambulatory clinic week for all residents in a cohort (approximately 30 physicians) via Zoom and then was repeated for 5 weeks to reach all trainees. We demonstrated potential pitfalls of a virtual encounter (from technology-specific issues such as audio or connectivity problems, to challenges in virtual verbal communication) and paused periodically for discussion. This gave residents an opportunity to share their impressions with the group and brainstorm possible solutions. The Cleveland Clinic Communication Tips for Virtual Visits (online supplemental appendix E) was used to teach resident essential skills (eg, conveying respect, setting an agenda and providing reflective listening using non-verbal communication). This tip sheet was developed based on the Relationship: Establishment, Development, and Engagement (REDE) model, a validated model for teaching healthcare communication.7 The presentation also reviewed practical tips such as looking at the camera instead of the screen to provide the appearance of direct eye contact. Initially, virtual visits were conducted from dedicated patient rooms on campus, but eventually many visits were conducted remotely to provide appropriate social distancing for trainees and staff. Medical decisions were always discussed with and approved by attending physician preceptors.
Virtual physical examination competency
Physical examinations in telemedicine must be augmented to fit the virtual nature of the encounter. Two inherent challenges with virtual visits are the physician’s ability to describe the examination he/she needs, and the patient’s ability to perform the examination on video. In our programme, we taught the virtual physical examination with Zoom. We used problem-based questions and interactive cases, addressing symptoms ranging from ankle pain to vision changes. During these scenarios, we emphasised creativity. For example, we showed residents how to teach a patient the Ottawa ankle rules to address ankle pain (online supplemental appendix F), thus saving them a visit to the emergency department.
Telemedicine technology platform competency
Virtual encounters were initially offered using a variety of platforms including American Well, Doximity, Skype and Apple FaceTime. We created a resource slide deck with practical tips on how to install and use each of those platforms (online supplemental appendix G). One month later, our institution started using a secure version of Zoom integrated into our Epic electronic medical record (EMR) to streamline virtual visits. Our slide deck was updated to incorporate this new platform. Residents also viewed an institutional video that walked them through using Zoom within Epic for virtual visits.
Teaching documentation of virtual encounters
Residents and preceptors were taught how to appropriately document virtual encounters. New documentation elements to the standard ambulatory note included a statement that the encounter was conducted by phone or video, an estimate of the time it took to provide care and a description of the findings of the virtual physical examination, if performed. Preceptors also needed to attest that the visit was provided either by phone or video, and indicate whether they were present for a portion of the encounter. These documentation changes were facilitated with new templated phrases in Epic that all residents and staff physicians could use.
Virtual encounter formative trainee evaluation
We created a formal evaluation process to help residents improve their practice. Residents were still required to precept their patient care with a staff physician. The staff physician could join the encounter at any time, especially during the visit wrap-up where the resident covers the plan with the patient and answers questions. Residents were also able to get direct observation and feedback during collection of the patient’s history and their performance of the physical examination in the virtual setting. This mirrored the process for in-person visits.
We created a mini-clinical evaluation exercise (mini-CEX) for telemedicine centred around the ACGME core competencies8 (online supplemental appendix H). Each resident was encouraged to send an electronic mini-CEX survey to faculty to get actionable feedback on their virtual visit, preferably before the encounter. Mini-CEX evaluations were geared to evaluate for the nuances of the virtual encounter. Residents in our programme are required to complete 12 mini-CEXs per year, and a telemedicine mini-CEX may be included.
Precepting the virtual encounter
Dynamic changes in recommendations for trainee virtual visits by the ACGME and billing and supervision rules from the Centers for Medicare & Medicaid Services (CMS) made regular updates for virtual care preceptors paramount. Our precepting team clinic leaders collaborated to identify champions for maintaining compliance with ACGME and CMS rules and providing as-needed updates to all faculty, residents and support staff. This incorporated changes to workflow for encounter consent and the location of preceptors (direct or indirect supervision). It also entailed identification of the types of encounters where the CMS primary care exception,9 stipulating that the preceptor does not need to interact with the patient, could be used.
Our team implemented the telemedicine curriculum within 4 weeks. All 148 ambulatory residents in our IMRP were trained in performing virtual visits and they completed more than 2000 such visits between early April and mid-June 2020. In total, 56 residents completed the orientation pre-participation survey with a total response rate of 38%. Forty-three residents completed the post-participation survey, with a total response rate of 29%. Of residents responding on the pre-participation survey, 76.8% had no prior telemedicine experience and indicated limited comfort with the modality (table 1). The post-participation survey assessed the residents’ perception of the quality of this curriculum in teaching telemedicine skills. Of these respondents, 46.5% were third-year residents (n=20) and 67.4% (n=29) reported learning new knowledge about telemedicine because of the curriculum that was offered (table 2). Residents selected strongly agree or agree to each of the eight statements on the post-participation by over 60% per question (table 2). Free response survey answers provided both positive and growth-oriented feedback. In regard to the content, many residents felt that it helped bridge the skill gap created by the pandemic. One respondent wrote “It was very helpful as I had no formal training in telemedicine before this time and I had to do virtual visits in my Resident PCP clinic during COVID-19.” Regarding the delivery of the content, some respondents wrote that the classroom course was ‘not interactive enough’ or that ‘too much of the presentation was obvious or common sense’. As of 1 December 2020, 62 of the 76 virtual visit/telemedicine mini-CEXs sent to preceptors have been completed.
Our experience demonstrates that teaching and precepting telemedicine in resident ambulatory clinics is feasible. Implementing any type of curriculum or training programme requires extensive preparation, involvement of multiple stakeholders and competing time constraints. The urgency of the COVID-19 pandemic and the resulting changes to healthcare delivery allowed us to move this process along quickly within a major tertiary medical centre, at one of the largest internal medicine residency programmes in the USA, and at an institution where faculty had significant prior experience providing telemedicine care before the pandemic.3 In addition, several authors (DJS, OG, BEM and ADS) were residents with a significant leadership role from the beginning. They were designing this curriculum before the pandemic and they collaborated with the IMRP ambulatory team to expedite its launch. These residents created the telemedicine training video (online supplemental appendix D), the telemedicine troubleshooting guide (online supplemental appendix G) and the first iteration of the PowerPoint slides for the telemedicine didactic course. They also helped design the pre-participation and post-participation surveys (online supplemental appendices B and C), and the mini-CEX form for residents to solicit faculty feedback on telemedicine encounters (online supplemental appendix H). They attended many of the virtual didactic courses to serve as facilitators and answer questions from their peers. Their leadership helped to accelerate the pace of implementation of the Cleveland Clinic IMRP telemedicine curriculum to meet an essential need precipitated by the pandemic.
Zoom was the modality for teaching the introductory telemedicine course due to social distancing needs, but other mainstream tools like Microsoft Teams, used by other programmes,5 would have also worked. We did not require advanced features such as topic-based threads or secure messaging, adding flexibility in the choice of a video platform. Trainings were led by a chief resident for groups of 30 learners on average, as this is the number of residents in each ambulatory week at our programme. In the future, this small size may permit the use of breakout rooms to simulate encounters and provide feedback. Recently, Sartori and colleagues demonstrated the use of an Objective Structured Clinical Examination (OSCE) to assess residents’ telemedicine-specific skills,10 and this would be a natural next step for our programme. We plan to build on this study to train more faculty to become facilitators for the simulation of the telemedicine ambulatory clinic encounters in the future.
In many regards, the process for performing, precepting and documenting a virtual visit parallels a traditional encounter. Our introductory course therefore focused on video communication skills, the virtual physical examination, documentation changes and technology troubleshooting to help build resident competency in this new framework. Many recent articles detail best practices for the virtual physical examination,11 and the neurological12 and musculoskeletal13 14 examinations in particular. One study showed non-inferiority of the virtual shoulder examination compared with the traditional ‘in person’ examination.15 In our experience, ailments that could be demonstrated by visual inspection were easier to assess in a virtual physical examination than problems of the heart, lung or gastrointestinal tract that may require a more detailed examination with auscultation, percussion and/or palpation. Another unexpected challenge was that it was cumbersome at first for residents to identify a video platform that would work for each patient. Early on, patients were more likely to adopt video calls when we offered a wide variety of choices. Our institution’s subsequent rollout of an Epic EMR update with embedded Zoom video made visual appointments easier and more preferred by patients.
To evaluate learner response to our curriculum, we used pre-participation and post-participation surveys modelled after the work of others in creating a longitudinal telemedicine curriculum.16 The pre-participation results indicated that a majority of residents had not had prior telemedicine experience, but that they felt it was an important skill to learn and it would be part of their future practice (table 1). Post-participation survey results indicated that this curriculum was perceived as effective at teaching telemedicine skills that could be applied immediately to serve patients (table 2). Free response answers indicated that while the information conveyed was beneficial, the presentation format should be reworked to make it more interactive. This feedback has already been used to restructure and improve the telemedicine curriculum for 2020–2021.
Now that our residents have had at least three ambulatory weeks of telemedicine encounters, our next step will be to assess their self-reported competency in providing telemedicine care by phone or video. Since most residents from the 2019–2020 academic year did not have prior telemedicine experience, it will be helpful to understand how residents’ skillsets have changed because of this curriculum and hands-on experience in clinic. We also plan to solicit preceptor feedback on the training programme. In addition, residents are now requesting formal objective feedback from their ambulatory preceptors on their performance with quarterly summative evaluations and on-demand mini-CEX evaluations specific to virtual visits. These feedback tools will help learners improve and provide a further assessment of this curriculum. The new interns in 2020–2021 will provide in-person visits exclusively for the first half of the year, and then will have at least one telemedicine slot per ambulatory week thereafter. We believe this will allow the newest trainees time to better understand the workflow at our institution and the tools available to serve their patients in a hands-on, supervised environment before those interns begin performing telemedicine encounters. They will also receive the Introduction to Telemedicine course that was offered to all residents at the start of the pandemic.
Our limitations were a low survey response rate and variable faculty experience with precepting telemedicine. The response rate for the pre-participation survey was higher than the post-participation survey, but the response rate on both instruments was low. This is likely because of the accelerated pace that this curriculum was rolled out to meet the needs of our learners and patients. We had the most success with getting pre-participation surveys completed by providing time at the start of the virtual didactic course for residents to fill them out with their phones. We also kept the number of survey questions short to encourage participation. It was even more challenging to get learner post-participation feedback because of their busy work schedules and the ongoing pandemic. The post-participation survey response rate was also impacted by timing. It was distributed at the end of the academic year when many residents were preparing to graduate or advance to the next training year.
Lack of staff physician experience with precepting telemedicine encounters was also a limitation early on, but this improved with time, practice and the creation of a faculty preceptor guide (online supplemental appendix A). Since preceptors supervise resident care, the biggest change in workflow has been at the end of each encounter. If a resident wants the preceptor to examine the patient or be available on the line to listen to the wrap-up with the patient, the preceptor needs to either join the resident in person or connect to the encounter by three-way video or phone. This was something that our faculty mentors and learners adapted to quickly, and this would be a very manageable barrier within any other residency programme, especially those of smaller size. Despite challenges in assessment and ambulatory precepting, this curriculum has served the needs of our residents well, and it will be a sustainable element of our training programme.
Our experience with implementing a comprehensive telemedicine curriculum for internal medicine residents and their preceptors is notable for its quick turnaround time in response to the COVID-19 pandemic, its breadth in reaching 148 trainees and 62 preceptors at nine clinic sites, and for its involvement of resident leaders for implementation. Having resident champions significantly broadened the scope of this project and helped create a support network for their peers so that the first iteration of the telemedicine curriculum was a success. We hope our experience can guide other programmes in developing a lasting telemedicine curriculum that prepares physicians-in-training for a practice landscape that includes virtual visits.
List of learning points
Prior to March 2020, telemedicine training for residents and fellows was limited, even though telemedicine was quickly emerging as a widespread tool for providing care.
Telemedicine curricula for residents and fellows are quite new, but their development was accelerated by the COVID-19 pandemic.
A comprehensive curriculum for telemedicine can be expeditiously and effectively implemented at a large multisite residency programme.
Trainee leadership in building a curriculum can accelerate implementation and increase buy-in from both faculty and peers.
Patient consent for publication
We would like to acknowledge our 2019-2020 Ambulatory Chief Residents (Drs Rabel Misbah Rameez and Christine Zayouna), our IMRP Ambulatory Team and Programme Director Dr Abby Spencer for empowering us to implement this program and then share it with programmes nationwide. We would also like to acknowledge the Cleveland Clinic Center for Excellence in Communication for its contribution to this project.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Twitter @DocDavidSavage, @ogutierrez89, @Achintya_singh4, @EYMD_, @JamalMaharH, @AndreiBrateanu, @JollyMD_GIM
Contributors DJS: Conceptualised project, prepared IRB proposal, wrote original outline and draft, edited final manuscript and prepared paper for submission. OG: Conceptualised project, data curation, methodology, project administration, contributed substantially to original draft and edited final manuscript. BEM, ADS: Conceptualised project, project administration, contributed substantially to original draft and edited final manuscript. EY: Data curation, methodology, edited initial draft and provided substantial feedback. JM: Conceptualised project, edited initial draft and provided substantial feedback. AB, LK, CF, SJ: Investigation, methodology, project administration, edited initial draft and provided substantial feedback.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.