The purpose of this paper is to review the methods used to measure quality of stroke care. Relevant articles were searched for on Medline using the following key words: stroke, quality, outcome of care, process of care, structure of care. Articles that examined how to measure the quality of stroke care and that examined difficulties in the measurement of care outcomes, processes, and structures were selected. Selected articles were reviewed to summarise methods used to measure quality of stroke care and the primary outcome measures of the studies were extracted. Conclusions were drawn about the best ways to measure the quality of stroke care. Practical problems in using outcome measures to monitor quality of care include the consequences of case mix and difficulties in risk adjustment. Clinicians may use process measures to understand differences in outcome. Once a process of care has been linked to an outcome measure, this care process should be measured. The national sentinel audit for stroke is an audit tool used to examine the quality of the processes of stroke care.
- outcome of care
- process of care
- structure of care
Statistics from Altmetric.com
“Patients have a right to expect that . . .they get a first class service. And in a first class service there is no room for second best”.1 About 100 000 people in England and Wales have a first stroke annually.2 Stroke patients occupy 20% of acute medical beds in the NHS.3 Stroke patients require access to quality services appropriate to their needs. The purpose of this article is to review the methods by which clinicians can measure quality of stroke care and the limitations of such measures.
Lohr and Schroeder defined quality of care as “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge”.4 Donabedian developed guidelines for assessing quality of care.5 These guidelines take into account the outcome, process, and structure of care.
The outcome of care indicates the effects of care on individuals. In the context of stroke care, outcomes may include death or disability.
The process of care refers to the characteristics of available services.6 In stroke care, process measures may include access to radiological investigations.
The structure of care is “the facilities, equipment, services and manpower available for care and the credentials and qualifications of the health care professionals involved”.7 Availability of trained multidisciplinary staff is one measure of the care structure.
Data were drawn from published articles to prepare this paper. Relevant articles were found by searching for the following Medline keywords: stroke, quality, outcome of care, process of care, structure of care. Articles that examined how to measure the quality of stroke care and articles that examined difficulties in the measurement of care outcomes, processes, and structures were selected. The selected articles were reviewed to summarise the methods by which clinicians can measure quality of stroke care and to summarise the limitations of these methods.
The data from the studies were then put together. Issues related to the measurement of outcomes of stroke care were discussed and summarised in a section entitled “issues concerning outcome of care measures”. Conclusions about the best ways to measure the outcomes of stroke care were then drawn—in a section entitled “which outcome measure?”
This method was repeated in the sections related to the measurement of processes and structures of stroke care.
ISSUES CONCERNING OUTCOME OF CARE MEASURES
Using healthcare outcomes in quality assessment has a number of advantages: outcomes are crucial to the definition of quality and they are free of predetermined ideas about how we should provide care.8 They also have high face validity. The gold standard in clinical research is the randomised controlled trial and outcome measures are intrinsic to these trials.
However, there are limitations to the measurement of healthcare outcomes. Mortality is a commonly used outcome. However, its relative rarity often results in inadequate power. More than 20 000 patients were required in the Chinese Acute Stroke Trial to demonstrate a statistically significant difference in mortality between stroke patients who received aspirin and those that did not.9 Equally in a patient with a severe brainstem stroke, death is almost an inevitable outcome and may come as a relief to the patient. The palliative care of such a patient may be high quality but still lead to death.
Other problems in using outcome measures to monitor performance include problems of definition (for example, accuracy of case definition) and the consequences of case mix, co-morbidity, and chance.10
Jesse and Schranz examined the relationship between hospital quality and mortality rates and as measured by accreditation survey performance.11 They made comparisons for three mortality groups: overall, stroke, and acute heart disease. No statistically significant differences were found in survey performance for any of the mortality groups (p = 0.05).
Box 1: Issues concerning outcome of care measures
Outcomes are crucial to the definition of quality and they are free of predetermined ideas about how we should provide care.
Variations in case mix may exert a confounding influence on clinical outcome.
Risk adjustment may be used to account for differences in outcome in patients with stroke.
The sample of stroke patients audited must be is adequately powered to detect clinically significant differences between different units.
The time interval of the patient's improvement is essential to the measurement of the outcome of stroke care.
Gompertz et al compared outcomes and quality of care for patients with stroke in different healthcare districts.12 They did not find major differences in patients' outcome or quality of care, possibly due to the non-random nature of the comparison; they urged caution when using such non-random comparisons.
Mant and Hicks investigated the ability of outcome measurements to discover real differences in performance between units.13 They looked at the quality of care for patients with myocardial infarction. They compared the relative sensitivity of measures of process and measures of outcome in detecting differences in quality of care. They found that process measures were able to detect significant differences between units that would not be identified by comparing units' mortality rates.
Outcome measures may be misconstrued. Shi Wu Wen analysed the outcomes of patients who had an incidental appendicectomy during cholecystectomy.14 Initial analysis showed an unexpected reduction in mortality when appendicectomy was performed. However, when confounding factors were adjusted for, patients who underwent incidental appendicectomy showed a significant increase in complications. The authors felt that their pre-existing clinical suspicions had strongly informed their analysis.
The average of the success rates of individual clinicians at a hospital form the success rate for that hospital. Comparisons between hospitals mask greater variations that exist between clinicians.15
Davenport et al examined the influence of case mix on clinical outcomes for stroke patients.18 They showed that variations in case mix (for example, age) have a crucial influence on the interpretation of outcome data.
Comparisons between services are prone to bias because of differences in prognostic factors between the populations studied.19 There is no ideal way to adjust for such differences; this may result in investigators missing important but modest differences or producing false associations.20
Shah et al investigated prognostic factors in patients with stroke.21 They found that multiple factors (including social factors) could affect the outcome.
In 1996, the Intercollegiate Working Party for Stroke established a minimum dataset to measure the severity of case mix in stroke.22 Items in the dataset were specific to stroke and predictive of outcome; they included age, pre-stroke function, and acute status after stroke. Allowance for case mix using the dataset can inform the interpretation of outcome data. For example, poor pre-stroke function may affect the outcome of rehabilitation.23,24
Poloniecki et al examined the value of hospital league tables in measuring the outcome of cardiac surgery.25 They found these tables of limited value because year-to-year differences in death rate can be substantial even when the underlying case mix does not change. They felt that “statistical quality control of a single series with adjustment for case mix is the only way to take recent performance into account when informing a patient of the risk of surgery at a particular hospital”.
Due to imperfect adjustments for differences in case mix, differences in performance figures for hospitals do not inevitably indicate differences in risk to individual patients.15
The characteristics of stroke patients admitted to hospital influence the outcome of care.26 Many of these characteristics (for example, co-morbid conditions) are not modifiable by the healthcare provided. Thus, it is crucial that we take into account these characteristics during outcome measurement. Risk adjustment may be used to account for differences in outcome in stroke patients.27 The quantification of patient risk and its use in the adjustment of outcome should be done thoroughly and accurately. If done thoroughly, it is more likely that healthcare outcomes will be a reflection of quality of care.8
However, there remains controversy regarding the means of risk adjustment. Fink et al reviewed the literature on the use of risk adjusted hospital mortality as a measure of quality of care.28 They found significant deficiencies in more than 75% of published studies.
In cardiac surgery, a risk stratification system has been developed to adjust for differences in a patient's perioperative risk.29 No such risk stratification system exists for stroke care.
Frequently the sample of stroke patients audited is inadequately powered to detect significant differences between different units.30
The time interval of the patient's improvement is essential to the measurement of the outcome of care.31 Outcome measured one year after the stroke may reflect events that have occurred long after discharge. Conversely, outcome measurement taken immediately after discharge may not take into account events that occur shortly after discharge. For example, a patient may be discharged home inappropriately and then readmitted. The immediate outcome may thus be wrongly judged a success. The institutionalisation rate after stroke may be used as an outcome measure. Some authorities feel that outcomes are too distant from the process of care to be efficient measures of quality.32
The optimal time to measure outcome depends on the outcome that we are measuring (for example, motor function improves sooner than speech). The Copenhagen Stroke Study showed that 90% of patients reached their plateau for improvement at 12 weeks and that the improvement rate was greatest between three and six weeks.33 High quality care during this period may have the biggest effect on outcome.
Single outcome measurements are far less informative than measures that clinicians have assessed repeatedly over time.6
WHICH OUTCOME MEASUREMENT?
When outcome measures are chosen their validity, reliability, relevance, simplicity, and resource implications should be intrinsic to the choice.39 We should choose outcome measures that are applicable to as many patients as possible and that are agreed upon by the multidisciplinary team.39 We should measure outcomes in the first month after hospital discharge.40 This is because clinical status beyond this time is difficult to measure and factors such as depression and recurrent ischaemic events will influence the clinical status.41,42
Box 2: Issues surrounding measures of process of care
Once a link has been demonstrated between a process of care and an outcome measure, then this care process should be measured.
Process measures that ignore differences in individual needs may result in standardised care that has little relevance to individual patients.
High quality audit of the process of care requires high quality coding, record keeping, and record retrieval.
We should interpret outcome measurements cautiously as many factors besides the structure or process of care can affect outcome. Until we make improvements in methods of risk adjustment for case mix, quality assessment should concentrate on assessing process and structure of care measures.11,13,15,21,25
Pound et al demonstrated that greater patient satisfaction was associated with greater service delivery and better outcome.34 Most authors agree that service evaluations should incorporate the views of patients 34,35,39
ISSUES SURROUNDING MEASURES OF PROCESS OF CARE
Once a link has been demonstrated between a care process and an outcome measure, then this care process should be measured.
Measures of process of care have usually been chosen arbitrarily without proven linkages to outcomes. There is a need for more evidence of linkages between processes of care and outcomes of care.43 In a study by the RAND Corporation, no differences were found in hospital level processes between hospitals for risk adjusted mortality due to stroke, pneumonia, or myocardial infarction.44
However in another study from RAND, significant relationships were found between risk adjusted outcome and process of care for stroke: the unit of analysis was the patient not the hospital.45,46 Until now our ability to adjust for risk factors such as severity of illness has been crude.
We should measure quality of care by measuring what is done instead of how much is done. This would involve a change from purchasing activity to purchasing protocols.47 High quality care is care that is consistent with care provided in clinical trials and, in the past, clinicians have sometimes failed to incorporate research findings into clinical practice.48
There are other advantages to measuring care processes. Patients often desire good processes (for example, dignity) as well as favourable outcomes.32 However, there is no good way to measure this; patient satisfaction surveys are crude measurement instruments.49
In patients with complex conditions, many variables that influence outcome cannot be measured: in such patients, process measures may be used as proxies for outcome measures.4
Individuals with stroke have individual needs. Most treatment programmes attempt to tailor their efforts to the patients' deficits. Process measures that ignore differences in individual needs may result in standardised care that has little relevance to individuals. Equally, processes of care tailored to individual needs are very difficult to measure. Assessing process measures in patients who share common needs may overcome this problem.6
Lee et al analysed geographic variability in the utilisation of post-stroke care in the US using multiple linear regression.50 They found that patient variables rarely explained more than one third of practice variation. Much of the practice variation in post-stroke rehabilitation remained unexplained.
Coding and record retrieval
Poor diagnostic coding may bias the sample of records retrieved for audit.49 A biased sample may result if medical records designated for audit are not located.51 It is often difficult to locate the records of deceased patients,52 and if these records are not found the audit will be biased.
Studies have demonstrated the limited recording of care processes in medical records.53 This inadequate documentation does not necessarily mean poor care but does make retrospective chart audit difficult. Record keeping must be of high quality to allow high quality audit.
WHICH PROCESS MEASUREMENT?
Clinicians should use valid, reliable process measures that are appropriate to their patients.39 Processes of care should be measured repeatedly at appropriate intervals. When deficits are identified in the process of care, these deficits should be corrected.
Whichever process measure is used, it should not be restricted to individual professions but should cover the quality of the whole service.39 Clinicians should take responsibility for the data collection and audit.39 A possible caveat is that this may increase the risk of bias. Clinicians should keep a stroke register, as routine hospital statistics do not accurately count stroke patients.54
The Royal College of Physicians of London developed an audit package to identify the quality of management of stroke patients.55 Gompertz et al performed inter-rater reliability studies on this audit package with acceptable results, most items showing “good” or “fair” reliability.56 Inter-rater reliability is better when auditors are medically qualified.56
The national sentinel audit for stroke is a multidisciplinary audit tool developed to examine the quality of assessment, investigation, treatment, and services for stroke patients. It has gained acceptance as a tool for raising the standard of care afforded to stroke patients.57 Gompertz et al demonstrated that it is a valid and reliable tool for assessing stroke care.58 They emphasised that careful piloting of audit tools and clear instructions to auditors were vital to the reliability of data. The national sentinel audit of stroke was used to audit the care received by stroke patients in 197 trusts (80%) in Great Britain. Regional workshop sessions that followed the reporting of the audit added to the consensus that the results accurately reflected what was happening on the ground. It is a genuinely multidisciplinary tool and can be used in a wide variety of care settings and allows an extensive review of many aspects of stroke care. It examines after-care in the community as well as the hospital care.
The American Heart Association proposed that measures of the process of care should include initial evaluation, acute and preventive therapy, and rehabilitation.40
Tracers are prevalent, well defined conditions with significant functional impact; there are accepted procedures for their management and these are likely to positively influence the patient's condition.59 The tracer approach assumes that the manner in which care is provided for the specific condition chosen as a tracer is an indicator of the quality of care in other areas.60
Box 3: Issues surrounding measures of structure of care
Demonstrations of linkages between structures and outcomes of care are limited.
The relationship between measures of structures of care and patient outcomes is inconclusive.
Outcome measurement in the assessment of quality of care should proceed cautiously as many factors besides the structure or process of care can affect outcome.
Until we improve methods of risk adjustment for case mix, quality assessment should concentrate on assessing process and structure of care measures.
The national sentinel audit for stroke is a multidisciplinary audit tool used to examine the quality of the process of stroke care.
Clinicians should use measures of structure of care that have high face validity and that are applicable to a variety of settings.
High quality care is care that is consistent with the care provided in clinical trials.
Ring et al used urinary incontinence as a tracer to assess and improve the quality of care on a stroke rehabilitation ward.59
ISSUES SURROUNDING MEASURES OF STRUCTURE OF CARE
Demonstrations of linkages between structures and outcomes of care are limited. Researchers attempting to link structures and outcomes of care have examined only a few structures including hospital procedure volume, teaching status, physician caseload, and physician certification. The relationship between patient outcome and many of these structural variables is inconclusive: in any case, translation of results into practice changes is difficult.8
WHICH STRUCTURE MEASUREMENT?
A report from the American Heart Association sought to identify measures of the structure of care.40 They suggested standards that had high face validity and that were applicable to many settings including local competence in stroke care and access to brain imaging and neurosurgical expertise. The presence of a dedicated stroke unit improves the functional outcome of patients cared for in such units.61
There are important areas for future research within this field. Most importantly, new quality of care measures should be developed and tested. These measures should take into account the interests of patients and caregivers.
We should attempt to develop valid, reliable, simple, and relevant outcome measures. Risk adjustment models should be developed for allow for differences in case mix.
In the future, measuring the process of care may allow us to delineate which aspects of care contribute to improved outcomes seen in stroke units.
Practical problems in using outcome measures to monitor quality of care include the consequences of case mix and difficulties in risk adjustment. Clinicians may use process measures to understand differences in outcome. Once a process of care has been linked to an outcome measure, this care process should be measured. The national sentinel audit for stroke is a multidisciplinary audit tool used to examine the quality of the processes of stroke care. It measures parameters including assessment, investigation, and therapy.
Questions (answers at end of paper)
What are the disadvantages of using healthcare outcomes in measuring the quality of stroke care?
What are the advantages of using healthcare processes in measuring the quality of stroke care?
What criteria should clinicians use in assessing process of care measures?
What measures of the structure of stroke care should be used in assessing stroke care quality?
Practical problems in using outcome measures to monitor performance include problems of definition (such as reliability of case definition) and the consequences of case mix, severity, co-morbidity, and chance.
Process measures may enable us to understand differences in outcome. Once a link has been demonstrated between a process of care and an outcome measure, then this care process should be measured. In addition, patients often desire good processes of care (for example, dignity) as well as favourable outcomes. In patients with complex medical conditions, many variables that influence outcome cannot be measured: in such patients, process measures may be used as proxies for outcome measures.
Clinicians should use process measures that have been proved in terms of validity and reliability and that are appropriate to their patients.
Whichever process measure is used, it should not be restricted to individual professions but should cover the quality of the whole service.
Structure of care measures may include quality improvement efforts, local competence in stroke care, and access to a stroke unit.