Aim To describe how unannounced in situ simulation (ISS) was perceived by healthcare professionals before and after its implementation, and to describe the organisational impact of ISS.
Study design Ten unannounced ISS involving all staff were scheduled March–August 2007. Questionnaire surveys on staff perceptions were conducted before (2003–2006) and after (2007–2008) implementation of unannounced ISS. Information from the debriefing sessions following each ISS constituted a proxy measure of the organisational impact of the ISS.
Results Five out of ten of the unannounced ISS scheduled were conducted. Twenty-three members of the staff at work on a scheduled day for ISS were randomly selected to participate. Questionnaires before implementation revealed that 137/196 (70%) of staff members agreed or strongly agreed that ISS was a good idea and 52/199 (26%) thought it likely to be stressful and unpleasant. Questionnaires completed after implementation showed significantly more staff members, 135/153 (89%), thought ISS was a good idea. A significantly higher amount of staff members 50/153 (33%) found it to be stressful and unpleasant, and among midwives, 15/59 (25%) were anxious about ISS, whereas none of the obstetricians reported this. Information obtained through debriefing sessions generated learning points.
Conclusions The number of staff members with a positive perception of multiprofessional unannounced ISS increased after implementation; however, one-third considered ISS to be stressful and unpleasant and midwives more frequently so. The specific perception of ISS by each healthcare profession should be taken into account when planning ISS. The information from the debriefing sessions showed that implementation of ISS had an impact as it provided information required for organisational changes.
- MEDICAL EDUCATION & TRAINING
Statistics from Altmetric.com
Labour wards have the dual function of creating a relaxed atmosphere for normal childbirth and of simultaneously having to be constantly prepared to deal with life-threatening emergencies.1 As a result, labour wards are challenging workplaces, where patient safety and medical litigation are high on the agenda.2 ,3 Labour ward staff must be prepared to deal with unexpected emergencies. Clinical management of obstetric emergencies is difficult to learn in real life due to the rarity of emergency events, which is why simulation-based training is essential.4–9 In recent years, many wards have implemented simulation-based medical training. Health authorities such as the National Board of Health in Denmark and the Clinical Negligence Scheme for Trusts in Britain currently require all labour ward staff to participate in annual skill drills.3 ,10 Little is published on optimal training content, preferred educational and learning strategies or the optimal physical setting for simulation-based training.7 ,11
Simulation-based training has traditionally been conducted off site in simulation centres and some hospital departments also provide in-house training at the hospital in rooms specifically allocated to training.5 In situ simulation (ISS), introduced over the past decade, is defined by Riley et al12 as "a team-based simulation strategy that occurs on patient care units involving actual healthcare team members within their own working environment". Conduction of ISS can be either announced or unannounced.13 ,14 The term drill is sometimes used for unannounced ISS.13 ,14 An advantage of ISS is that it is conducted in the real workplace, thus making it possible to train staff and also identify systemic weaknesses, which in turn forms the basis for required organisational changes.15–18 Some have argued that ISS is more feasible and cost saving compared with traditional simulation conducted in simulation centres.19 ,20 Some studies concluded that ISS is met with widespread acceptance among participants and has resulted in a collaborative approach to patient safety.14 ,19 ,21 ,22
Emergency drills are described by Anderson et al as "as scenario-based training in obstetric emergencies conducted in ‘real time’ in the normal working environment, without the prior knowledge of the staff involved. This type of training is sometimes called ‘fire drill’. The aim of such drill is to test local systems and protocols for responding to emergencies, as well as to test professional teamwork and individual skills and knowledge."13 Inspired by this approach, we decided to focus on unannounced ISS. Anderson et al13 indicated that emergency drills can be perceived as intimidating by participants, but so far this has been poorly explored in the literature.
This is an observational study with the primary aim of describing how healthcare professionals perceive unannounced ISS in obstetric emergencies before and after its implementation and, secondarily, to analyse the organisational impact of ISS. The more specific objectives were to describe staff members’ perceptions of ISS and its likely usefulness and impact in terms of anxiety, stress and other perceptions and to describe any changes in these views once staff members had become familiar with ISS.
Materials and methods
See box 1 for the setting.
The study took place at the Department of Obstetrics, Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark.
During the study period, the department had approximately 3300 deliveries per year, approximately two-thirds of which came from a local catchment area in Copenhagen, while the remaining third comprised high-risk referrals from Eastern Denmark and a small number of special cases categorised as nationwide referrals.
Participants were recruited from the following staff groups: specialised obstetricians, trainee obstetricians, midwives, specialised midwives, auxiliary nurses, and nurses in antenatal and postnatal wards. With turnover in staff accounted for, the department employed an estimated 160–170 staff at any given time during the study period.
The research group consisted of one obstetrician (JLS), three midwives (PL, MS, KSA) and one nurse (PE). MS and PE were a part of the departmental management team, and KSA was the quality coordinator in the department and cooperated with the risk managers at the hospital. Everyone in the research group was ISS instructors. Obstetrician JLS and a midwife KSA were responsible for the debriefing. Two psychologists from the hospital's Development and Quality Improvement Department helped to design the structure of the debriefing session.
The research group laid down the prerequisites for the intervention, which was unannounced ISS, before its implementation (box 2). Since 2003, the department has regularly conducted planned in-house mandatory multiprofessional obstetric training at the hospital in dedicated training rooms. Hence, the staff was used to and had participated in planned mandatory training, which was part of a research project that involved the entire staff completing questionnaires from 2003 to 2006.23
Prerequisites for unannounced in situ simulation (ISS) established by the research group before planning and implementing the unannounced ISS
Before participating in an unannounced ISS, staff are required to attend the department's mandatory obstetric training programme, which was established in 2003 and is currently conducted quarterly or bimonthly.23
Staff must be informed in advance of planned unannounced ISS.
Instructors must be prepared to cancel scheduled unannounced ISS in the event of heavy patient loads, staff shortage or recent severe obstetric emergencies.
The unannounced ISS must not pose any risk to real-life patient care, which means extra staff must be available to replace staff participating in the unannounced ISS.
The planning of the unannounced ISS and the debriefing must be systematic and focus on the identification of systemic errors, quality improvement initiatives and the need for future training.
Immediately after taking part in an unannounced ISS, participants must be given the opportunity to retrain on manikins and/or be provided with written guidelines.
Facilitators of the debriefing must write a report based on the information that emerges. This report must be approved by ISS participants and subsequently distributed to all department staff.
If suggested changes are relevant, management and the department's quality coordinator must assist with problem solving and work to implement the recommended changes.
Unannounced ISS included the following emergencies: (1) shoulder dystocia (a complication of vaginal delivery, where the baby's shoulders are impacted behind the pubic bone after delivery of the foetal head), (2) postpartum bleeding and (3) severe preeclampsia with eclamptic convulsions.
Before implementation of the unannounced ISS, all staff members were informed at staff meetings by email and on notice boards that one would take place within a 6-month period (March–August 2007). Internet links to the relevant clinical guidelines for the three emergencies in focus were made easily available.
The research group planned the unannounced ISS in detail, wrote scripts for the ISS and selected manikins. The setting was either a labour ward delivery room (shoulder dystocia and postpartum bleeding) or an in-patient room on postnatal ward (severe preeclampsia). Online supplementary table S1 presents examples of ISS planning. Ten unannounced ISS were scheduled.
Each of the unannounced ISS was expected to involve an authentic team of 4–5 different healthcare professionals. The ISS participants were randomly selected among the staff at work on the day scheduled for the unannounced ISS. The ISS participants had to be replaced by other staff members during the ISS and subsequent debriefing sessions to ensure that the ISS did not interfere with regular patient care.
Scheduled to take place immediately after each unannounced ISS, all debriefing sessions were facilitated by an obstetrician (JLS) and a midwife (KSA). The debriefing sessions explored the participant's views on organisational learning, the individual and team learning, and proposals for organisational changes in practices could also be put forward. The debriefing sessions were oral, and the facilitators took notes. The debriefing sessions lasted 30–60 min and comprised three phases: description, analysis and application.24
The facilitators in the debriefing session (JLS, KSA) wrote a report that had to be approved by all the ISS participants. Upon approval, the report was distributed in the department by email, website and on notice boards. The department's managerial team (MS, PE) and the midwife working as the quality coordinator (KSA) were responsible for implementing any relevant changes suggested as a result of the unannounced ISS scenarios.
Participants eligible for questionnaires and as ISS participants
Eligible participants comprised staff from all healthcare professional groups working on the labour ward (figure 1) and they were all invited to complete questionnaires before (pre-Q) and after implementation (post-Q) of ISS. Staff members eligible for taking part in the unannounced ISS were recruited randomly on the day designated for the ISS from among the healthcare professional groups at work.
Pre-Q and post-Q questionnaires, administered as presented in figure 1, comprised items using a five-point Likert scale, ranging from 1=strongly disagree to 5=strongly agree or from 1=never to 5=very often.
The paper questionnaires were distributed via the internal mail system, and after filling them out by hand, staff members were asked to return them in either closed boxes placed in the wards or to two research group members (JLS or PL).
The pre-Q addressed issues like whether planned and mandatory obstetric training was perceived as a good idea and whether unannounced ISS was perceived as potentially stressful and unpleasant and/or as instructive and educational (table 1). The post-Q was given to all staff, that was, both ISS participants and staff members who did not participate in the ISS. Some questions addressed to all staff in the pre-Q were repeated after implementation. Hence, the post-Q asked all of the healthcare professionals whether ISS was a good idea and whether they perceived it to be stressful and unpleasant (table 2). Furthermore, after implementation (post-Q) they were also asked whether they knew about the ISS, were anxious about them or hoped more of them would follow. They were also asked about their view on the impact of ISS on future interprofessional collaboration, its perceived helpfulness in preparing healthcare professionals for real emergencies and whether the unannounced ISS encouraged staff to study the literature and guidelines, and finally on whether it would impact the level of individual practice (table 3). Tables 1⇓–3 present a translated, condensed version of the original questionnaire items in Danish.
Questionnaire data were transferred in coded form from the paper version to a 2003 Excel spreadsheet by JLS and PL and prepared for statistical analysis performed by SR.
After the debriefing session, the facilitators (JLS and KSA) documented the findings in five reports, which, based on the debriefing session, were then reanalysed using content analysis to condense the acquired information.25
The questionnaire responses based on a Likert scale were treated as ordinal data. For questions answered twice by the same participants (pre-Q and post-Q), the proportion of participants giving the same responses on the two occasions, reflecting marginal homogeneity of ordinal tables of agreement, was quantified by weighted κ statistics. Marginal homogeneity was also formally tested by a generalisation of the McNemar test for paired proportions appropriate to multicategory ordinal rating data.26 A value of κ close to zero means the paired proportions differed between the two measurement occasions, and close to one means agreement between the two measurement occasions. The Cochran–Armitage test for trend was used to determine whether there was an association between responses to ordinal questionnaire items and participation in unannounced ISS (yes/no).26 The Kruskal–Wallis test was used to determine whether responses to the questionnaire differed by healthcare professional groups. Our analyses have not been adjusted for multiple statistical testing, and due to the large number of tests, no conclusions were drawn on associations of borderline significance. SAS V.9.2 and R V.3.0.2 were used for the statistical analysis.
Since the study did not involve patients, no approval was required under Danish regulations. Questionnaire respondents were assigned identification numbers known only to JLS and PL. During the analysis and reporting phase, all data were treated as non-traceable information.
Response rates to questionnaires
The response rate to the pre-Q for all staff members was 196–207/220 (89–94%), which varied for different questions. For the individual staff groups, the highest response rate was obstetricians: 27/28; trainee obstetricians: 21/21; specialised midwives: 18/21; midwives: 80/84; auxiliary nurses: 21/24; and nurses: 40/42. The response rate to the post-Q for all staff members was 84–86% (149–155/178). For the individual staff groups, the highest response rate was obstetricians: 21/21; trainee obstetrician: 14/14; specialised midwives: 13/14; midwives 59/64; auxiliary nurses: 19/30; and nurses: 28/35.
Number of ISS conducted and ISS participants
Five out of ten of the scheduled ISS were conducted and analysed, while the other five were cancelled due to the heavy workload on the labour ward, a shortage of labour ward delivery rooms and/or shortage of staff. Altogether, 23 healthcare professionals out of a possible 178 eligible staff members participated in an ISS and distribution among the staff was as follows: three obstetricians, four trainee obstetricians, three specialised midwives (one participated twice), five midwives, four auxiliary nurses and four obstetric nurses. The response rate after implementation of ISS was 18/23 (79%) among ISS participants and 131–135/155 (85–88%) among non-participants.
Perceptions of ISS before and after implementation
Table 1 shows staff perceptions of planned and mandatory training and unannounced ISS before implementation of ISS. Table 2 shows a comparison of staff perceptions before and after implementation of ISS. Before implementation, 137/196 (70%) of all participants ‘agreed’ or ‘strongly agreed’ that unannounced ISS was a good idea and this increased to 135/153 (89%) after implementation (p<0.0001). In summary, 52/199 (26%) ‘agreed’ or ‘strongly agreed’ that unannounced ISS was stressful or unpleasant before implementation compared with an increase of 50/153 (33%) afterwards (p=0.0001).
Perception of ISS among non-participants and ISS participants
Table 3 summarises the perceptions of ISS among participating and non-participating staff. Questions on information and discussion about ISS showed that staff participating in ISS had heard about ISS and discussed ISS significantly more often. Participating staff also expected ISS to prepare them for real emergencies significantly more often, while both participating and non-participating staff found ISS to be important for future cooperation and for identifying changes in work processes to the same extent. Responses did not differ between staff participating in ISS and non-participating staff concerning anxiety about ISS and whether ISS was perceived to be stressful or unpleasant. Participant responses differed, however, depending on their profession. The question rating whether ISS was stressful or unpleasant was rated as ‘agree’ or ‘strongly agree’ by 1/13 (8%) specialised midwives; 2/21 (10%) obstetricians; 3/14 (21%) trainee obstetricians; 8/27 (30%) nurses; 7/19 (37%) auxiliary nurses; and 29/59 (50%) midwives (p=0.0082 using the Kruskal–Wallis test for the full set of ordinal responses). Responses to the question on perceived anxiety towards ISS also differed by profession. Midwives rated ‘agree’ or ‘strongly agree’ to feeling anxiety about ISS, 15/59 (25%) compared with 0/21 (00%) for obstetricians (p=0.0023). Participating and non-participating staff reported studying guidelines and reading the literature to the same degree. This was not the case for this question, however, with regards to profession, where 4/13 (31%) trainee obstetricians and 5/14 (36%) specialised midwives ‘agreed’ or ‘strongly agreed’ that they read educational material to prepare for ISS, which represents a higher proportion than the other professions (p=0.0019).
Analysis of the reports from the debriefing sessions resulted in several learning points at individual, team and organisational levels. Online supplementary table S2 presents a comprehensive list concerning organisational impact. Several of the practical changes recommended after ISS were implemented, for example, improvements in operating the telephone system, checking stopwatches in labour rooms, checking content of delivery room cabinets and the setup of blood pressure monitoring systems in the postnatal ward.
The unannounced ISS implemented over a 6-month period were considered to be a good idea by the majority of staff members, and acceptance increased after implementation. Nevertheless, approximately one-third of all staff members found unannounced ISS to be potentially stressful and unpleasant, and this number increased after implementation of ISS. In particular, midwives perceived unannounced ISS as unpleasant and stressful and reported related anxiety, whereas obstetricians reported no anxiety at all. When planning multiprofessional educational interventions, like unannounced ISS, it is therefore important to consider these differences in perceptions between groups of healthcare professionals.
Almost all staff members, and both ISS participants and non-participating staff, thought that unannounced ISS was important for future collaboration. Three-quarters of the staff participating in ISS indicated a belief that unannounced ISS would enhance their performance in future real emergencies, whereas the corresponding number among non-participants was only about one-third. Approximately half of the non-participating staff reported having studied the written reports on the unannounced ISS debriefing sessions. An ongoing question is how does ISS, involving only few staff members, influence all staff members and the organisation as a whole? The findings of the present study suggest that effects of the intervention (ISS) can be extrapolated to the entire staff to some extent, for example, to the non-participants.
Our results support the conclusion that only five unannounced ISS were sufficient to provide the organisation with valuable information on weaknesses in the system, including information that may prove difficult to obtain elsewhere. This is consistent with conclusions in other studies on the impact factor of ISS, thus emphasising the system perspective of ISS.12 ,14–19 ,22 Overall, the managerial team considered the organisational value of implementing unannounced ISS in the department to be of major importance. In this study, the research group included members of the department's managerial team and a midwife working as quality coordinator, which we considered useful in the process of evaluating the need for implementation of the proposed changes.
It could be argued that some of the system weaknesses could also have been identified with announced ISS, which is presumably easier to organise and implement. It seems unlikely, however, that problems with, for example, telephones and the calling system would have been exposed quite as noticeably as they were during the unannounced ISS. This may be due to the unpreparedness aspect of unannounced ISS, which potentially allows a more authentic simulation of the stress element experienced in real-life obstetric emergencies. This phenomenon warrants further investigation in future studies comparing announced and unannounced ISS. A potentially powerful contributor to the effects of simulation is the simulation's level of fidelity.27 The setting in which the simulation takes place may partially determine the fidelity, but more research is needed to clarify how announced and unannounced ISS influence the level of fidelity and learning.
Implementing unannounced ISS was time consuming and challenging for the research group, which is why strong support from the management team was of paramount importance. A shortage of staff members and delivery rooms meant we could only carry out 5 instead of the 10 unannounced ISS originally planned.
The study has limitations and the design of the study can be characterised as a compromise design, which is not unusual in educational research. A randomised design with control groups would have been preferable. We were unable to identify any randomised studies comparing ISS and simulation training in a simulation centre or in-house training in hospital facilities. We were also unable to identify studies comparing announced and unannounced ISS.14 Another limitation was the lack of validated questionnaires. Use of validated instruments to measure the level of anxiety, stress and motivation should be considered for future studies.28 Furthermore, this study did not take an explorative approach and hence was not designed to elaborate upon and explain why problems appeared during the ISS and the debriefing session. The study was small and resulted in the inclusion of only five unannounced ISS, all conducted in the same hospital. This provided the opportunity to measure local effects but also raised the issue as to whether the results can be transferred to other departments. All the unannounced ISS were carried out during daytime hours, which means our study does not address whether conducting unannounced ISS during night shifts or on weekends would produce different or new findings.
The five unannounced ISS in this study appear to have had a positive organisational impact by providing information to support organisational changes and changes in the clinical management of the department. To date, the ISS literature has primarily focused on organisational impact, neglecting the perceptions of participating and non-participating staff, which was the major focus of our study. Our findings indicate that further studies are warranted to explore the mechanisms at play in relation to the differences in perception of unannounced ISS between various healthcare professional groups that emerged during our study and their implications for learning.
Based on experiences from the present study, ISS has recently been implemented in the same department but involves also anaesthesia trainee and consultants, anaesthesia nurses and surgical nurses, in addition to the same obstetric staff members. Our recent work on implementing ISS has focused on applying experiences from our previous ISS work. Apart from appointing a working committee with representatives from each healthcare professional group, we have worked closely to involve all of the healthcare professionals and the management. A protocol article describes this new ISS intervention.28
Midwives perceived unannounced in situ simulation (ISS) as unpleasant and stressful and reported related anxiety, whereas the obstetricians reported no anxiety at all. When planning unannounced ISS in a multiprofessional environment, it is important to consider differences in the perception between healthcare professional groups.
Despite the fact that the number of staff members with an overall positive perception of unannounced ISS increased with its implementation, one-third still indicated unannounced ISS to be stressful and unpleasant.
Planning and implementation of unannounced ISS was time consuming and challenging. Strong support from the managerial team was of paramount importance.
Even after conduction of only five unannounced ISS, we were able to identify important areas for organisational development and improvement.
Current research questions
How does the setting, that is, ISS versus off-site simulation of obstetric training, affect the level of learning?
What role do factors like the fidelity of simulation setting, stress and motivation play in relation to the differences between healthcare professionals’ perceptions of the simulation setting and what implications do they have on learning?
Psychologists Morten Jack and Dorthe Degnegaard, Development and Quality Improvement Department, Rigshospitalet, University of Copenhagen, Denmark, helped to design the structure of the debriefing session after the unannounced ISS and provided feedback to the instructors. The authors would like to thank Mereke Gosira, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, the Netherlands, and obstetric consultant Marianne Johansen, Department of Obstetrics, Rigshospitalet, University of Copenhagen, for editing the manuscript.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
Contributors JLS conceived the idea, acquired data for the prequestionnaire, drafted this paper and obtained funding jointly with BO. PL acquired the data for the postquestionnaire. JLS, PL, MS, KSA and PE contributed to the practical design and practical completion of this study. SR performed in cooperation with JLS the statistical analysis. All authors contributed to the discussion and interpretation of the data as well as a critical revision of the manuscript. BO and CV discussed the manuscript in detail, and all authors approved the final manuscript.
Funding The Capital Region of Denmark, a non-profit public organisation, provided funding for this project.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.