The introduction of evidence based medical practice requires awareness of a large number of up-to-date guidelines and protocols,¹ and the availability of clinical data.² Guidelines are proliferating rapidly.³ Assimilating and using this information can be difficult, particularly in the busy environment of acute wards. Furthermore the information is often most needed when dealing with urgent problems late at night, when access to clinical information sources is difficult.

A previous systematic review has found the traditional paper based dissemination of guidelines to be ineffective in changing the behaviours of health care professionals.⁴ Alternative strategies for guideline dissemination need to be developed.

Morriston Hospital is a large district general hospital in Swansea, serving a population of around 200 000. Acute medical patients are admitted through casualty and by direct referral from general practitioners. Most patients are assessed in the admissions ward before full admission.

The Multidisciplinary Online Clinical Information Service (MOCIS) at Morriston Hospital gives access to hospital guidelines and protocols, pharmacy information, databases such as the Cochrane Library, National Poisons Information Service including Toxbase, the Health of Wales Information Service, and other information. It is available on terminals in all medical wards.

Application of guideline directed, evidence based practice in the NHS is an official priority for the government. Two national bodies, the National Institute for Clinical Excellence and Commission for Healthcare Audit and Inspection, are established to produce and disseminate clinical guidelines and to oversee the quality of clinical governance. However, dissemination and implementation of guidelines remains problematic and coherent strategies to promote the use of guidelines need to be developed and rigorously evaluated.⁵

The aim of the study was to evaluate the impact of MOCIS on guideline compliance.

METHOD

A portfolio of 12 guidelines was distributed on paper to all medical staff on 2 January 1999. To compromise between the needs for rapid dissemination of the guidelines and a systematic evaluation of the impact of MOCIS, all except three preselected guidelines were made available in electronic format on MOCIS from 1 February. The remaining three were then added to MOCIS one by one at three month intervals and the impact on practice monitored.

A multiple interrupted time series design was used to evaluate the impact of MOCIS on guideline compliance.^6,7 We measured guideline compliance for a period before and after the introduction of paper and electronic guidelines for the management of three conditions. The repeated measurements across multiple guidelines allowed us to control for threats to internal validity (such as increasing familiarity with departmental practice or historic events such as a sudden bed crisis). We introduced the electronic version of the three guidelines sequentially at three monthly intervals. If online presentation were effective, compliance with each guideline should increase stepwise in the weeks after its addition to MOCIS, while compliance with the other guidelines not yet added to MOCIS would remain stable at that time. The sequential introduction of the electronic guidelines across the three conditions would control for any interaction between the intervention (MOCIS), time, and different settings (specialties and wards). This approach used statistical instead of experimental controls and relied heavily on data quality.

Data on compliance with the guidelines were collected by scrutiny of patients’ notes, looking for adherence to preset criteria. As the project team considered documentation an essential part of patient care, the assessment was based on documented compliance. Although the study was not a randomised trial, we used the Consolidated Standards for Reporting of Trials flow diagram⁸ to monitor the case identification process, to identify impact of possible problems associated with the tracking of medical records reported in previous literature.⁹ An independent researcher extracted data on compliance from the medical records. Another researcher randomly checked 10% of the reviewed notes to ensure inter-rater reliability.

We selected guidelines for the management of suspected deep vein thrombosis, upper gastrointestinal bleeding, and stroke for evaluation. These guidelines were related to common conditions where variations in clinical practice were known to occur and usually dealt with by junior medical staff. These conditions were also covered by national guidelines with endpoints that can be measured.^10–16 A consultant haematologist (SAI), a gastroenterologist (PDD), and a geriatrician (MAW) led the local adaptations of the guidelines. Each local guideline comprised a summary algorithm and explanatory text incorporating the evidence for the guideline.

The medical directorate considered all the paper guidelines and agreed that they were appropriate to local circumstances. The directorate then issued the paper guidelines to all doctors working in the directorate and all medical wards on 2 January 1999.

Patients admitted to the medical admissions unit and other wards between 1 December 1998 and 31 October 1999 with presenting complaints suggestive of stroke, deep vein thrombosis, or upper gastrointestinal bleeding, were identified using ward admission books. Case notes were used to confirm appropriateness for inclusion, and to extract data. As twice as many patients with stroke or deep vein thrombosis were admitted than with gastrointestinal bleeding, our sampling strategy extracted alternate records for stroke and deep vein thrombosis but every record for gastrointestinal bleeding. Figure 1 summarises this process and the numbers analysed. We assessed inter-rater reliability with prevalence and bias adjusted kappa.¹⁷

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

 Case identification and inclusion and exclusion process (DVT, deep vein thrombosis; GI, gastrointestinal; PAS, patient administration system).

The guideline for suspected deep vein thrombosis advised the discharge of otherwise fit patients on subcutaneous, low molecular weight heparin (tinzaparin), while awaiting investigation by ultrasound. Compliance required that either the reason for admission was recorded or low molecular weight heparin had been prescribed before discharge.

Compliance with the guideline for upper gastrointestinal bleeding required that a severity assessment (Rockall score)¹³ had been recorded in the notes by a junior doctor on the basis of the initial clinical assessment.

Compliance with the guideline for stroke required that the initial clerking recorded consciousness and swallowing, and a request or plan to measure the erythrocyte sedimentation rate (ESR). If there was no evidence of these three criteria being assessed, the case scored 0. A general statement about consciousness, swallowing, or ESR measurement scored 1 for each criterion. A more detailed statement (Glasgow coma score, request for swallowing assessment, or dated ESR request form) scored 2 for each. The total score therefore ranged from 0 to 6. The median compliance score for the whole study was 2 and those scoring 2 or less was considered to be non-compliant.

The impact on guideline compliance rate of dissemination through paper and electronic format was assessed according to the following steps:

• Provisional specification of an autoregression integrated moving average (ARIMA)⁶ model for the series of observation for each of the three guidelines, including the timing of paper and electronic dissemination as dummy variables.

• Fine tune the model specification till there was no underlying pattern in the distribution of residuals (the differences between observed compliance and the level of compliance estimated from the model).

• Interpret the coefficients of the dummy variables as measures of the effect of the different formats of guideline dissemination.

Ideally, the ARIMA model should have been developed with observations made before the dissemination of guidelines in any format and then re-estimated with the whole series of observations. However, this was not possible as the number of observations made before the introduction of guidelines was small. To triangulate the findings, we also performed cumulative sum technique analysis.¹⁸

Compliance scores for each of the three guidelines in each study day were recorded in SPSS. For each successive compliance or non-compliance with guidelines, positive or negative increments were added to a cumulative score. The cumulative compliance scores were charted with an upward trend indicating more patients being managed according to guidelines and a downward trend indicating the opposite. Davies formulas were used to evaluate the statistical significance of observed changes in cumulative compliance.¹⁸

The results of the assessment were discussed with staff in a series of interviews after the study to understand differences in compliance rate with the three guidelines. Eight junior doctors who had admitted patients in the course of the study, and the three guideline authors were interviewed.

RESULTS

There were 1534 possible cases identified from medical admissions unit ward books, of which 879 cases were sampled and 829 sets of notes located. The assessment team excluded 304 cases, which showed early consultant involvement in patient care, as these cases would reflect guideline compliance by consultant rather than by junior medical staff. Other reasons for exclusion included inconsistent episode details between the records and clinical or administrative reasons precluding guideline compliance. A total of 470 cases were included (fig 1). Kappa analysis showed good inter-rater reliability (table 1).

The overall average rate of guideline compliance within the study period was low (deep vein thrombosis: 33%, gastrointestinal bleed: 2%, stroke: 28%). There was a slight increase in compliance rate with the deep vein thrombosis and gastrointestinal bleed guideline throughout the study period (table 2), but the differences were not significant (table 3).

Compliance with the stroke guideline decreased when the guideline was made available on paper but increased after the guideline was put on MOCIS (table 2). Difference in compliance rate with the stroke guideline before and after it was made available on paper was not significant. The introduction of the electronic version of the stroke guideline was associated with a significant increase in compliance (table 3).

The cumulative sum technique was used to illustrate the general trend of guideline compliance.¹⁸ This did not change significantly for deep vein thrombosis or upper gastrointestinal bleeding but showed a significant improvement for stroke after the guideline was made available in electronic form (fig 2).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

 Cumulative compliance score (standardised for series average) by stage of guideline introduction (DVT, deep vein thrombosis; GI, gastrointestinal).

These quantitative findings were corroborated by the views of junior doctors, who were interviewed when the results were known. These interviews revealed that junior doctors were confused as to whether they were supposed to manage patients with suspected deep vein thrombosis as inpatients or outpatients. This confusion was caused, in part, by uncertainty whether the policy to manage patients with suspected deep vein thrombosis as outpatients was in force, and also by difficulties in obtaining an early Doppler ultrasound. The guideline author confirmed this. Thus there was conflict between what was advocated in the guideline and what was possible to achieve in the service setting.

For upper gastrointestinal bleeding, most doctors felt that the guideline was useful and reported that they had consulted it. In spite of this, only one doctor claimed to have used the Rockall score. Others claimed that they assessed patients’ conditions using some of the components that make up the score, but admitted that they had not calculated a formal score. This was consistent with the view of the guideline author, who accepted that although the score was included to enable junior medical staff to assess when to call for urgent endoscopy, only doctors with experience in gastroenterology were familiar with the Rockall score.

By contrast, the stroke guidelines were widely supported and actively promoted throughout the study period. Junior doctors had highlighted some potential problems with the use of MOCIS but shared the view of guideline implementers that MOCIS was safer than paper for storing and distributing updated guidelines. They liked the flexibility of paper guidelines but felt overwhelmed by the amount of paper based information sent to them. This was particularly pertinent in the case of the stroke guideline as the hard copy of this guideline was double the thickness of the other two guidelines in the study. There was an observed decrease in compliance rate associated with the issue of the stroke guideline on paper, which was reversed when the guideline was made available on MOCIS.

These interviews after the study showed that the guidelines for deep vein thrombosis and gastrointestinal bleeding did not have full unequivocal support from opinion formers within the trust. Making these guidelines available online would not influence compliance. Success of MOCIS with the stroke guideline was intricately linked with the support for the guideline itself.

DISCUSSION

This study has shown that making guidelines available in an electronic form has the potential to improve compliance, providing there is consensus on the content and active support for guideline use. We chose a multiple, interrupted time series design to detect differences in compliance when the guidelines are presented in paper or electronic form. We chose this design to control for threats to internal validity (such as increasing professional expertise) or continuous and discrete historical events (such as undefined health trends or defined events such as a ward closure). Sequential introduction of electronic guidelines for different clinical conditions, controlled for any interaction between the intervention, time, and specialties. The multiple measurement points allowed us to use autocorrelation analyses to assess for any trends caused by confounding effects of associations between data points over time, unrelated to the intervention.

This design does not differentiate the impact of MOCIS on guideline compliance from its possible effect on improved documentation of those factors that indicate guideline compliance. However, appropriate documentation is an integral part of good patient care and, if affected, would be a positive benefit from the guideline.

This study has confirmed the need for active promotion of guidelines³ and the negative effect of confusion and practical difficulties, even when guidelines are made available in electronic form. With effective clinical leadership and adequate resources for the process of building support for the guidelines, online guidelines can help to alter practice and improve patient care.

Adherence to appropriate guidelines has important clinical governance implications. Taking the appropriate measures to ensure guidelines are effective will impact on patient safety, junior medical staff training, and job satisfaction. After the study, processes were put in place in the study site for the continuous promotion of the stroke guideline and the consensual development of guidelines for the management of suspected deep vein thrombosis and upper gastrointestinal bleeding.

Different strategies were adopted to promote guideline compliance for the management of the three conditions. Post-study interviews with junior medical staff revealed a conflict between what was advocated in the deep vein thrombosis guideline and what was possible in practice. Therefore, the guideline for deep vein thrombosis was reviewed, consensus between clinical and service departments built, and departmental policy in outpatient management was clarified. The deep vein thrombosis guideline was reissued and a clinical nurse specialist to coordinate the management of these patients has now been appointed by the trust. The failure of compliance with the gastrointestinal bleeding guideline was due to a lack of familiarity of junior doctors with the Rockall score, even though this was described clearly in the guideline. The recent appointment of a second consultant has enabled reorganisation of the emergency endoscopy services, and the guideline will be reissued when this is complete. The stroke guideline has been continuously promoted.

MOCIS has been expanded in content and availability. A multidisciplinary committee has been formed to supervise it and this reports to the trust clinical governance committee. There are now over 96 clinical guidelines available online and a virtual library gives access to CD textbooks and over 900 full text biomedical journals. Training on the use of MOCIS is part of the junior doctors’ induction programme and will soon be included in induction of nurses.

All new guideline authors are asked to ensure their guidelines have support within their department and they are obliged to update them every year. Regular newsletters are sent around electronically to staff, advising of the introduction of any new guidelines.