Article Text

Download PDFPDF
Republished: Which questions of two commonly used multidimensional palliative care patient reported outcome measures are most useful? Results from the European and African PRISMA survey
  1. Irene J Higginson1,
  2. Steffen T Simon1,2,
  3. Hamid Benalia1,
  4. Julia Downing3,
  5. Barbara A Daveson1,
  6. Richard Harding1,
  7. Claudia Bausewein1,
  8. on behalf of PRISMA
  1. 1King's College London, Cicely Saunders Institute, Department of Palliative Care, Policy and Rehabilitation, London, UK
  2. 2Department of Palliative Medicine and Clinical Trials Centre Cologne (BMBF 01KN1106), University of Cologne, Cologne, Germany
  3. 3African Palliative Care Association, Kampala, Uganda
  1. Correspondence to Irene J Higginson, Cicely Saunders Institute,Department of Palliative Care, Policy and Rehabilitation, King's College London, Bessemer Road, London SE5 9PJ, UK; irene.higginson{at}


Aim To evaluate the views of clinicians and researchers on their use of outcome measures and which questions are most important in palliative and end-of-life care.

Methods Online survey of professionals working in clinical care, clinical audit and research in palliative care across Europe and Africa identified through national and international associations and databases. Questions focused on measures used, reasons and which questions were important in two commonly used multidimensional measures, the Palliative care Outcome Scale (POS) and the Support Team Assessment Schedule (STAS).

Results The overall completion rate was 59% (392/663). Three outcome measures were commonly used by over one in four respondents for clinical practice and over one in 10 for research: the Karnofsky Performance Scale (KPS), followed by the Edmonton Symptom Assessment Scale (ESAS) and the POS. Measures were used twice as often in clinical practice as in research. The main uses were similar: assessing patients' symptoms/needs (88% and 85% of POS and STAS users, respectively), monitoring changes (62%, 58%), evaluating care (61%, 48%) and assessing family needs (59%, 60%). Respondents rated the most important questions as pain, symptoms, emotional and family aspects. There were no differences in the choice of the most important questions between doctors and nurses or between researchers and clinicians.

Conclusions In palliative care, outcome measures often used in clinical practice are also often used in research. Questions relating to pain, symptoms, emotional needs and family concerns are consistently considered the most useful and important in palliative patient reported outcome measures (PROMs).

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


In healthcare today there is a growing emphasis on capturing patients' voices and experiences. This involves using patient reported outcome measures (PROMs) and patient reported experience measures (PREMs). Traditionally, PROMs were used in clinical trials to measure treatment effects but increasingly are employed in clinical practice to prioritise problems, communicate, screen, monitor change, record treatment response and train new staff, and in clinical audit and clinical governance.1–3 Their use helps to ensure care is orientated towards the needs of patients and families, rather than focusing on test reports or process measures, which are arguably not patient or family orientated. PREMs are closely related to PROMs, as the experience of care, especially when patients are very ill, is part of the outcome.

What is measured subsequently defines care, especially if results are used by clinicians to help their practice or by policy makers, funders or patient advocates. Therefore, it is imperative that palliative and end-of-life care PROMs capture those aspects which are most relevant, and helpful to practice and to patient and family concerns. This is especially critical when healthcare budgets are constrained. Herein lies the challenging paradox of PROMs in palliative care. Because of the illnesses and frailty of most patients and family distress, PROMs must be short and easy to use. But they must also capture complex aspects, such as quality of life and quality of dying. Clinicians favour outcome measures that consist of 6–10 questions,4 but in such a short measure it is challenging to capture the essence of palliative care. Although there are psychometric methods to reduce the number of items in measures, these focus on reducing questions that relate to a single construct and rarely consider clinical priorities and use.5 ,6 No study has analysed multidimensional palliative care measures to determine if and how questions might be reduced, based on clinician views of utility.

To determine which items are most important, we chose to focus on two widely used (in all continents) multidimensional PROMs/PREMs in palliative care: the Palliative care Outcome Scale (POS)7–14 and the Support Team Assessment Schedule (STAS).15–17 This allowed us to provide specific, rather than hypothetical, examples of measures, and to use the experience of individuals who had used the measures in practice and/or research. The POS and STAS are freely available, with resources at and

This survey aimed to evaluate the views of clinicians and researchers on their use of outcome measures and which questions of multidimensional measures are most useful and important in palliative and end-of-life care. A secondary aim was to determine if there was any difference between practitioners with different backgrounds or experience. The survey was developed as part of PRISMA (Positive diveRsities of European priorities for reSearch and Measurement in end-of-life cAre), a 3-year collaboration funded by the European Commission with a special focus on outcome measurement in palliative care.18



This was a web-based online survey. Additional details were published previously.4

Survey procedures

The Centre for Evaluation and Methods (ZEM) at the University of Bonn, Germany ( conducted the survey.4 Strategies to increase response rates included two reminders, no password required (also to ensure anonymity) and a low cost prize draw. The prize draw offered book tokens for 10 of the first 100 respondents to the invitation email (a token worth €50/£40) and the first reminder (token worth €25/£20), if respondents opted to enter their details at the end of the questionnaire. To maintain anonymity, the details of these respondents were separated from their questionnaire responses. We did not use cookies or collect internet protocol (IP) addresses.


We surveyed palliative care professionals (doctors, nurses, other professionals) working in clinical care, clinical audit or research in Europe or Africa. Our main sources for invitees were the large databases of national associations in the eight European countries within the PRISMA collaboration, and the African Palliative Care Association (APCA). Invitees were emailed by the associations directly. We also sought participants by searching smaller databases of individuals with specific outcomes and research interests (see Box 1).

Box 1

Methods of identifying survey participants

Databases used to identify survey participants

  1. National palliative care associations

    Twelve national palliative care associations across the eight countries of the PRISMA collaboration (Belgium, Germany, Italy, Netherlands, Norway, Portugal, Spain and the UK). To adjust for the different sizes of countries and associations, a weighted random sample of 20% of each group was drawn if the total number of clinicians (physicians, nurses and others) in the associations was over 500 members and a sample of 33% if the total number was less than 500 members.

  2. APCA

    APCA is a membership organisation for individuals and organisations interested in and working in palliative care in Africa with 475 individual contacts from 25 African countries. As the list does not contain information on the professional background of contacts, invitation emails were sent out to all contacts but those who were not clinicians were asked to feed this information back to the APCA office. This procedure should have helped to assess the number of non-respondents, taking into account those who are non-clinicians.

  3. POS database

    The Department of Palliative Care, Policy and Rehabilitation, King's College London, hosts a database of registered users of the POS. This database was updated and cleaned before the survey. Invitations to participate in the survey were sent to 211 users.

  4. POS and STAS authors

    A scoping exercise in scientific journals on the use of POS and STAS identified 61 authors for whom contact details were available. All authors were invited to participate in the survey.

  5. European palliative care academics

    PRISMA provides a list of 61 chairs and researchers in palliative medicine in Europe. All contacts on this list were invited to participate in the survey.

  6. UK palliative care nurses

    As there is no organisation representing palliative care nurses in the UK, two email lists were used to invite this group to participate in the survey and give them the opportunity to share their views. One list originated from a forum of the Palliative Nursing Group of the Royal College of Nursing (this was also listed as a national organisation). As sampling was not possible with this list, all members of the forum received an invitation email asking them to request the link to the online questionnaire. The second list contained 28 nurse consultants who were also invited to participate in the survey.

  7. PRISMA members

    All 38 members of the PRISMA consortium were invited to participate in the online survey.

APCA, African Palliative Care Association; POS, Palliative Care outcome Scale; PRISMA, Positive diveRsities of European priorities for reSearch and Measurement in end-of-life cAre; STAS, Support Team Assessment Schedule.


The questionnaire was developed to respond to the aims of the study. The team reviewed other questionnaires on outcome measurement use. Potential questions were reviewed by the team and revised and further reviewed, until a final version was agreed. The electronic version was piloted in seven European countries (Austria, Germany, Italy, Netherlands, Norway, Portugal and the UK) with 20 professionals working in palliative care. The final version contained 59 questions in total, but as adaptive questioning was used, several questions were displayed based only on specific responses. The survey was conducted in English due to restricted resources.

The survey included general questions on measurement use in clinical care, clinical audit and research (purpose, choice, frequency of use). Participants were asked to identify which measures they had or were currently using. We developed a list of 25 potential measures based on an initial scoping exercise among clinicians, literature review and piloting, including those measures most mentioned. We left open space for participants to add measures. Participants were asked to identify which measures they used for research and for clinical practice. Those participants using the POS and/or STAS were asked to rate the utility of individual POS and STAS items (using a four-point Likert scale of very useful, fairly useful, neutral, not useful), and then to rank order the three most useful questions. Items in the POS, APCA version of the POS19 ,20 (culturally adapted and validated for Africa) and STAS are shown in table 1.

Table 1

Items in the different measures which were assessed for utility


Following the CHERRIES (Checklist for Reporting Results of Internet E-Surveys) guidelines,21 the participation rate was calculated by dividing the number of replies to the first question by the number of unique site visitors (defined as those visiting the first page of the online survey); the completion rate was calculated by dividing the number of participants answering the last question by the number answering the first. We conducted simple descriptive analysis and tested for differences between professional groups using Fisher's exact test; a two tailed p<0.05 was taken as significant.


Ethics approval was provided by the Research Ethics Committee at King's College London (BDM/08/09-102). Further ethics approval from other European countries was not needed as this survey did not include patients, only professionals identified via national organisations. For Africa, ethics approval was obtained from the Ethics Committee of the Ugandan National Council for Science and Technology (IS 62) as APCA (our PRISMA partner agency) is based in Kampala, Uganda.


There were 1592 visitors (1291 in Europe and 301 in Africa) to the site: 663 answered the first question (overall participation rate 42%) and 392/663 completed the last question (completion rate 59%). In Europe, the participation rate was 38% (495/1291) and the completion rate was 63% (311/495); in Africa the participation rate was 56% (168/301) and the completion rate was 48% (81/168). The mean age of respondents was 46 years (SD 9) and two-thirds (63.4%) were women. Both in Europe and in Africa, the majority (72% Europe, 65% Africa) of respondents were clinicians (see table 2). In both continents, about two-thirds of participants had more than 5 years experience in palliative care.

Table 2

Characteristics of respondents

Use of measures in clinical practice and in research

Overall, 68.1% of respondents from Europe and 73.3% from Africa had experience with outcome measures in palliative care. Use (ever or current) of the different measures is shown in table 3. Use in clinical practice of the top 18 measures was usually two to three times more prevalent than their use in research.

Table 3

Use of different tools reported in the survey (n=434 respondents)

Three measures were commonly used by over one in four respondents for clinical practice and over one in 10 for research: the Karnofsky Performance Scale (KPS; a single item measure which assesses performance status), followed by the Edmonton Symptom Assessment Scale (ESAS; a measure of the severity of nine symptoms) and the POS (with order reversed for research). The fourth most common measure – the Palliative Performance Scale (PPS) – is also a single item measure of performance status which was adapted from the KPS. With few exceptions, measures used commonly in clinical practice were also used commonly for research. There was a modest overlap between research and clinical practice use, with around a quarter of those using the measures in clinical practice also using the same measure for research. This overlapping use was present across all respondents, not just those who were both clinicians and researchers. The exceptions were the European Organisation for Research into the Treatment of Cancer (EORTC) 30-item (QLQ-C30) and 15-item (QLQ-C15-Pal) questionnaire measures, which were used more in research than in clinical practice. There was also an overlap for these measures – around half of those using the QLQ-C30 were also using the QLQ-C15-Pal.

The prevalence of POS use (original or African versions) was 112/434 (25.8%) for clinical care, and 63/434 (14.5%) for research (31 respondents were using the POS in both clinical care and research). The STAS was the seventh ranking measure, used by 45/434 (10.4%), for clinical care and was the equal eighth ranking measure, used by 20/434 (4.6%), for research (six respondents were using the STAS in both clinical care and research). In addition, 30% of respondents using the POS also used it for teaching and training others.

Of those using the POS (there was no difference between the results for the original and African versions), 67% and 71% of those using it for clinical care or for research, respectively, reported their experience as good or very good. For the STAS, the figures were 53.6% and 84.6%, respectively.

Of those applying the POS and STAS in clinical practice, the most common uses were similar and related mainly to patient assessment and monitoring changes. The top two uses were to assess patient symptoms and needs (88% and 85% of POS and STAS users, respectively) and monitor changes in health status or quality of life (62%, 58%). Other main uses were to evaluate care (61%, 48%), assess family needs (59%, 60%), document problems (41%, 46%) and assess care against a given standard in clinical audit (38%, 27%). Use in facilitating communication diverged between the measures. The STAS was used much more often to facilitate communication within the team than the POS (38% POS, 71% STAS; Fisher's exact test, p<0.001). The POS appeared to be used more often to facilitate communication with the patient and family, but this was not significant (43% POS, 29% STAS; Fisher's exact test, p=0.17).

There were no differences between the professional groups in their use of the measures, except that nurses had a borderline trend towards more often using the POS to evaluate care compared to doctors (73% of nurses vs 53% of doctors; Fisher's exact test, p=0.07). There were no differences between staff with more or less than 5 years of experience in palliative care in their use of the POS or STAS.

Importance of items

For the POS (95 respondents), every item was rated as at least very or fairly useful by 59% or more of respondents, and very useful by at least 30% of respondents (figure 1). The most useful and important item was ‘pain control’, followed by ‘other symptom control’, and then ‘life worthwhile’ and ‘patient anxiety’. ‘Main problems’ (open question) was in the middle/lower end of the rankings, being the eighth most important of 11 questions. The APCA version of the POS showed similar results (figure 2; 41 respondents), again with most items being rated very or fairly useful. The main additional item in this measure (‘at peace’) was in the lower end of the rankings. For the STAS, the items showed a similar pattern, with ‘pain control’, ‘patient anxiety’, ‘other symptom control’ and ‘family needs’ being the most important (figure 3; 46 respondents). The communication items in the STAS (which are not found in the POS) were in the middle ranking of importance. Some of the STAS items were of lower importance than corresponding POS items, suggesting that the POS had appropriately focused on the more important aspects.

Figure 1

Number of respondents who ranked the Palliative care Outcome Scale item as most, second most or third most important and who rated the item as very useful (95 respondents). *Question 7 has two versions, but results for the usefulness of the items ‘depression’ and ‘life worthwhile’ were similar.

Figure 2

Number of respondents who ranked the African Palliative Care Association's Palliative care Outcome Scale item as most, second most or third most important and who rated the item as very useful (41 respondents).

Figure 3

Number of respondents who ranked the Support Team Assessment Schedule item as most, second most or third most important and who rated the item as very useful (46 respondents). Com, communication.

There was no difference between doctors and nurses, or between staff with more or less experience in palliative care, or between those using the measures for research or clinical care, in their rating of the relative usefulness of the different items in the measures. However, the numbers were small in some categories.


‘If you can measure it, you can manage it’ is a maxim commonly attributed to McKinsey, an international company which advises on the management of healthcare and private organisations. You cannot manage for improvement if you do not measure to see what is getting better and what is not. If process and activity are recorded, these then may become the standards of palliative care. If symptoms are recorded, then these may drive how palliative care is understood and supported. Therefore, the challenge for palliative care is to be defined not by the aspects ‘easy’ to measure, but by aspects which are important for patients and families, and the practice of palliative and hospice care, that is by the holistic multiprofessional outcomes of care. Measures in palliative care must perform the complex job of capturing individual needs and how they change and help palliative care services to address patient/family problems.

What this study adds

This study is the first international study of what measures are being used and for what purposes. and what items within the measures are of most value. We were able to reach a wide range of clinicians by using online survey methods, and achieve a large response from 392 individuals. There are five important findings.

First, we found that similar measures were used commonly in clinical practice and in research. If measures are clinically useful, then it is likely that they will collect meaningful data. This is important for research as well as clinical practice, as clinical trials and epidemiological studies also need to measure what is important for patients and families, albeit that in research further assessment of specific aspects might be needed.

Second, three measures stood out as being most commonly used in palliative care, both in clinical practice and in research: the KPS, the ESAS and the POS. If these measures were used to define palliative care, what would this suggest? The KPS is a single item scale which allows patients to be classified according to their functional impairment. The fourth most common measure (the PPS) is a similar performance scale to the KPS. The KPS and PPS range from 100 (normal, no complaints) to 0 (dead). They are quick and useful for assessment of prognosis, but in palliative care patients are likely to have floor effects – that is, most patients have poor function and their scores will tend to all be at the lowest values. The KPS and PPS may be particularly useful in assessing care needs and prognosis, but may have some limitations as outcome measures because of floor effects among very ill or elderly patients.

The ESAS comprises nine symptoms – pain, tiredness, nausea, depression, anxiety, drowsiness, appetite, well-being, shortness of breath and an open question about ‘other problems’.22 Each symptom is assessed on a visual analogue scale, although more recent versions offer an 11-point numerical rating scale. The frequency of use of this scale highlights the importance of symptoms to palliative care, both in clinical practice and in research. Symptoms have multidimensional aspects and well-being may relate to quality of life. A review of the ESAS (1991–2006) identified 13 studies on the reliability and/or validity of the ESAS.23 Further development through a think-aloud study conducted in 20 advanced cancer patients has led to a revised version of the ESAS, called the ESAS-r, which has been tested in a prospective multicentre study conducted in 160 patients in different palliative care settings, comparing the ESAS with the ESAS-r.24 This revised scale retains the core elements of the ESAS, with five main revisions: the timeframe for symptom ratings is specified as ‘now’; brief definitions have been added for tiredness (lack of energy), drowsiness (feeling sleepy), depression (feeling sad), anxiety (feeling nervous) and well-being (how you feel overall); ‘appetite’ has been changed to ‘lack of appetite’; the order of symptoms has been changed so that related symptoms are grouped together, with ‘well-being’ at the end; the example of ‘constipation’ has been added to the 10th item, ‘other symptom’; and horizontal lines over the numbers have been removed.23 ,24

Our study found that the POS is the most commonly used multidimensional measure which captures symptoms and specific aspects beyond symptoms. The POS indicates a broad definition of palliative care, including symptoms, psychological, social and spiritual well-being, and the needs of the family, consistent with the WHO definition and that of Dame Cicely Saunders.

Third, all of the most common measures are short. The KPS is a single item, while the ESAS and POS have 9–10 items, suggesting that palliative care clinicians and researchers favour short questionnaires. This is not surprising. Patients in palliative care are often too ill to complete or maintain concentration for even standard length questionnaires. Longer questionnaires may introduce a problem of missing data. The POS was developed from the STAS to be more flexible in terms of offering patient, care giver or staff completion, and having fewer items. Our findings suggest that this development was successful as the POS is more often used than the STAS, although both measures had positive reports.

Fourth, the POS and STAS were used most often to assess patients and to monitor changes in outcomes and need, although they had a wide range of other uses. Assessment is a cornerstone of palliative care. Uses were often multiple, the measure being used both to assess and monitor change, and/or for teaching, or for evaluation or communication. In clinical practice, many tests and measures are used both for assessment – to guide treatment – and for outcome measurement. Assessment of routine bloods, blood pressure, even CT or other tests is used to both assess patients and measure outcomes in response to treatment. PROMs and PREMs should be able to be used in the same way. This is also efficient. However, the use of the POS or other measures for assessment could be further enhanced by more clearly developed guidance on the best clinical responses to particular scores. Work on this issue has begun in Africa with the APCA African POS.19 ,25

Finally, our results show which items respondents working in palliative care find most useful. All items of the POS were rated as being very or fairly useful by 59% or more of respondents, but there was a pattern across the POS and the STAS, of pain, symptoms, depression, anxiety and family needs being the most useful. Items in the POS which factor analysis have shown are assessing the quality of care (financial/practical and information) were assessed as being less useful by respondents.26 Future work should consider whether these items could be dropped or made optional. The STAS contains some items which respondents found useful but which are not included in the POS because they are based on clinician assessments. These relate to communication. Work on the staff completion version of the POS could consider whether to reintroduce these items. We were surprised that the open question was not rated as being more important by respondents. This needs exploration.

Strengths and limitations

The web-based survey allowed us to reach professionals in a large number of countries in Europe and in Africa, and clinicians and researchers with a variety of medical, nursing and other backgrounds. This reflects daily practice as PROMs are used in a variety of ways and by various professionals. In particular, the views of clinicians are not often heard in the context of outcome measurement, and our survey specifically included their views.

However, our respondents may have differed from non-respondents in their views, and we do not know how this might have affected the results. Respondents with more extreme views on outcome measurement (positive or negative) may have been prompted to reply. Sampling was challenging, as we approached national palliative care associations which had slightly different categories of membership and methods for contacting their members. To ensure anonymity we did not collect IP addresses or use cookies, which may mean our participation rates are underestimated, as people may have viewed the survey and decided to complete it later. Equally, we cannot be sure that people did not complete the survey twice, although we think this is unlikely. Our survey was conducted in English, and so responses from individuals who were not able to write in English would have been missed. We considered translating the questionnaire into other languages but this was beyond our resources. Wording was kept simple. Participants were invited to respond to open-ended questions in their native language. Nevertheless our data are biased to the UK, although we did not find differences between countries in responses to items of importance or uses of measurement.

We focused on two commonly used multidimensional measures that capture symptoms and other aspects of palliative care. It may be that other aspects not included in these measures would have been deemed important. The numbers of people assessing specific items was a subgroup of the whole, and we did not separate those who had ever used from those currently using a measure. However, one measure (the POS) was the most commonly used multidimensional holistic measure, and the STAS was in the top seven measures, and was longer. Therefore, it is likely that these results can be applied to other measures.


In this first large international survey of professionals working in palliative care, we found three measures were commonly used in both clinical practice and research. All commonly used measures were short (ranging from one to 10 items). Measures were most often used for assessing patients and families and measuring changes over time, but multiple purposes were described. The most useful items identified by respondents related to pain, symptoms, and emotional and family aspects, which reflect the holistic nature of palliative care.


The authors would like to thank all participants for their time and engagement in the project and a workshop to review results. The contribution of all Work Package 4 PRISMA members is also kindly acknowledged.



  • This is a reprint of a paper that first appeared in BMJ Supportive & Palliative Care, 2012, Volume 2, pages 36–42.

  • Funding PRISMA is funded by the European Commission's Seventh Framework Programme (contract number: Health-F2-2008-201655) with the overall aim of co-ordinating high quality international research into end-of-life cancer care.

  • Competing interests Higginson conducted initial development of POS and STAS, and is interested in the results to improve outcome measures in the future. She receives no financial income from POS or STAS which are freely available

  • Ethics approval The Research Ethics Committee at King's College London (BDM/08/09-102) and the Ethics Committee of the Ugandan National Council for Science and Technology (IS 62) approved this study.

  • PRISMA Members Gwenda Albers, Barbara Antunes, Emma Bennett, Ana Barros Pinto, Claudia Bausewein, Dorothee Bechinger-English, Hamid Benalia, Lucy Bradley, Lucas Ceulemans, Barbara A Daveson, Luc Deliens, Noël Derycke, Martine de Vlieger, Let Dillen, Julia Downing, Michael Echteld, Natalie Evans, Dagny Faksvåg Haugen, Nancy Gikaara, Barbara Gomes, Marjolein Gysels, Sue Hall, Richard Harding, Irene J Higginson, Stein Kaasa, Jonathan Koffman, Pedro Lopes Ferreira, Johan Menten, Natalia Monteiro Calanzani, Fliss Murtagh, Bregje Onwuteaka-Philipsen, Roeline Pasman, Francesca Pettenati, Robert Pool, Tony Powell, Miel Ribbe, Katrin Sigurdardottir, Steffen Simon, Franco Toscani, Bart van den Eynden, Jenny van der Steen, Paul Vanden Berghe and Trudie van Iersel. PRISMA aims to provide evidence and guidance on best practice to ensure that research can measure and improve outcomes for patients and families. PRISMA activities aim to reflect the preferences and cultural diversities of citizens, the clinical priorities of clinicians and appropriately measure multidimensional outcomes across settings where end–of-life care is delivered.

  • Provenance and peer review Not commissioned; externally peer reviewed.