Reply to Reply In reply to De Wets and Bowie's reply to my criticism:

De Wet and Bowie state 1. Our paper makes it very clear that this is a study of primary care records and not of consultations, GPs or of primary care per se.

Reply Although this is true, nowhere does the paper explicitly explain that is looking in primary records for errors across the totality of healthcare. This is not mentioned until page 179 (4th page of article) and then only in passing. The clear assumption is that this is a study of primary care records and is looking for error and harm in primary care and this impression is not overtly refuted in the article.

De Wet and Bowie state 3. The harm rate per number of records and consultations is provided to make the results more meaningful for primary care clinicians who have multiple patient contacts.

Reply No primary care clinician I have spoken to thinks including secondary care events makes the results more meaningful to primary care clinicians. Primary care clinicians want to know what their own error/harm rate is. They do not want secondary care error/harm rates added to their own figures and inflating the 'primary care' error/harm rate.

De Wet and Bowie state 4. Johnstone suggests the harm rate should be expressed as '...47 per 2251 consultations (2.1%)...' We provided this measure i.e. 1 per 48 consultations (a ratio that equates to 2.1%). We could calculate at least six different measures involving the following variables: error, harm, preventability, setting, record and consultation. All provide useful but limited information, none is superior. We focused on harm per record because that is what we judged as most relevant to patient safety.

Reply I have no problem with this, but it must be made explicit and this is not the case in this paper. The paper does not explicitly state that secondary care errors are being included. Spinal surgery errors are mentioned in passing and on page 179 the paper mentions 'procedures in secondary care.' The paper continually mentions primary care, using the tool to provide a 'primary care equivalent'. On page 2 the paper's aims are to 'test a validated trigger tool for the primary-car setting'. It does not mention a secondary care setting. This lack of clarity invalidates this paper.

De Wet and Bowie state 5. We detected harm that originated in secondary care, which concurs with previous research. (4) We discuss the implications of this and that most 'severe' cases originated in this setting. We do not state that the harm rate of 9.4% is "for" primary care but make it clear that it is the harm rate found in the records, regardless of source of origin or preventability judgements.

Reply Although De Wet and Bowie mention secondary care once in the discussion section on page 179, the rest of the paper talks about harm and error in primary care. The whole paper is about using the trigger tool to detect and quantify error and harm in primary care, not in secondary care.

De Wet and Bowie state 6. Definitions of error and adverse event are outlined in our Methods section. Careful reading of Tables 3 and 4 shows that matching study data are presented for each category using the NCCMERP Error Index employed (which defines near misses, error and harm). We use harm and adverse event interchangeably, which is accepted practice. (5)(6)(7)

Reply I accept this, but harm is regularly used when error should be used. This occurs in the titles of tables 3 and 4. The NCCMERP tool detects error, not just harm, not all errors lead to harm and not all harm is due to error. This suggestsharm is more common than it is. On page 179 is the sentence Most cases of harm were graded as category A-E (temporary harm to a patient requiring an intervention). This is inaccurate as only category E is harm, categories A-D are errors with the potential for harm, but with no actual harm occurring.

De Wet and Bowie state 7. The NCCMERP index was selected in the absence of a universally agreed system and because it is used in hospital care. We did not adapt it to reduce bias. We provide a rationale: "In secondary-care studies, only categories E to I have been used, as they correlate with actual patient harm occurring. We included all categories because we assumed that relatively minor events may be more common in primary care". This assumption was proved correct. We also made the following recommendation: "It may be reasonable to use only categories E to I in future to target patient harm rather than just error".

Reply De Wet and Bowie have misread their own paper, on page 180 they state ' Few adverse events were in categories A-D despite the expectation of high error incidence.' Their assumption was proven wrong not correct.

De Wet and Bowie state 8. Johnstone states that he "almost believes" that we "confuse" these terms "on purpose". This comment is lacking in reason and evidence.

Reply True

De Wet and Bowie state 10. Johnstone queries the category E harm example in Table 2 (refer also to our No. 6 and 7 responses). Some harm events may arise from normal or evidence-based care. There may be no associated error (as in this example) or obvious error. To illustrate: a patient is appropriately prescribed an antibiotic but develops an allergic reaction. Record review reveals a similar prescription and outcome previously, but no allergy code was added. Arguably, this is preventable harm.(9)

Reply I agree, the second prescription is preventable harm, the first prescription is unavoidable harm.

De Wet and Bowie state 11. Johnstone is factually incorrect in stating that of "...the 47 events in primary care, 11 were in the range of possible harm rather than actual harm". 47 records contained evidence of some harm i.e. categories E (n=39), F (n=6) and G (n=2). We cannot make this clearer.

reply Yet again De Wet and Bowie have misread their own paper and confused primary and secondary care. There are 47 events in primary care and 11 are the category A-D, therefore possible harm not actual harm. Co-incidentally the number of events in categories E-G is also 47, but 11 of these occurred in secondary care.

De Wet and Bowie state 12. Johnstone states that we make a "throwaway comment" in the opening paragraph because we suggest that the evidence cited is "...likely to be an underestimation". Based on the evidence in hospital care (10), we hypothesised that the rate in primary care is greater than currently suggested, which is standard research practice. Our findings support our hypothesis.

Reply I disagree, De Wet and Bowie's findings show that preventable harm in primary care is 1:125 consultations, almost identical to 1:120 consultations which they thought was an underestimation.

De Wet and Bowie state 14. The issue over the "headline figure" being "inflated" from 9.4% to 9.5% (it was rounded to the nearest 0.5%) is external to the debate on the published paper.

Reply Point taken, I will follow this up separately.

De Wet and Bowie state 17. We did not exclude the records of the 94 patients (18.8%) who had no contact with primary care in the 12-month study period. If we now exclude them (which is reasonable), the actual harm rate is 47/406 (11.6%), strongly suggesting we underestimated the 'true' level.

Reply The rate of preventable harm in 1:125 consultations is unaffected whether the these 94 patients are included or excluded.

De Wet and Bowie state 18. We concede that our Abstract should read "...27/64 (42%) of error and harm events were judged preventable..." However, this issue is reported clearly in the main paper.

De Wet and Bowie state 19. We cannot control how others interpret our findings and are always careful to explain the scientific limitations of our study.

Reply Fair point, but the authors can control how the authors interpret and explain their study. This is not directly related to retracting this paper but on 10th June Bowie gave a power-point presentation titled 'Patient Safety and the GPST' . The presentation states 'The NES Trigger Tool... may enable you to measure the preventable harm rate.. which you can monitor and reduce overtime.' Although they include secondary events, it is hard for primary care clinicians to reduce errors in secondary care, eg pain after a spinal operation. Had De Wet and Bowie applied this sentence to their own paper, they would have reported the preventable harm rate in primary care as 0.8% not 9.4%

De Wet and Bowie state Screening medical records for undetected harm events only provides a single perspective. The process cannot capture 'acts of omission' that lead to harm nor are findings likely to correlate with data from patients, incident reporting systems, significant event analyses, complaints, litigation claims and so on. (11)(12)(13) The patient safety problem is highly likely to be greater than can be deduced from screening medical records, even when we account for source of origin and preventability judgements.

Reply I agree that Patient Safety is a major issue and should be taken immensely seriously. However I do not believe it helps anyone to portray the problem as greater than it is. I believe this paper has inaccuracies and gives the impression of a 9.4% harm rate in primary care when the actual figure could be better described as preventable harm occurs in 1:125 consultations in primary care (0.8%).

Conflict of Interest:

None declared

Dear Editor

Thank you for the opportunity to respond to Dr Chris Johnstone's comments on our 2009 publication.(1)

1. Our paper makes it very clear that this is a study of primary care records and not of consultations, GPs or of primary care per se.

2. There is no "common method of measuring rates of harm" in primary care (2) as Johnstone states. We refer to research on medical error and the consultation (3) in our Introduction to demonstrate evidence of a 'problem', not to indicate our intended data analysis method.

3. The harm rate per number of records and consultations is provided to make the results more meaningful for primary care clinicians who have multiple patient contacts.

4. Johnstone suggests the harm rate should be expressed as '...47 per 2251 consultations (2.1%)...' We provided this measure i.e. 1 per 48 consultations (a ratio that equates to 2.1%). We could calculate at least six different measures involving the following variables: error, harm, preventability, setting, record and consultation. All provide useful but limited information, none is superior. We focused on harm per record because that is what we judged as most relevant to patient safety.

5. We detected harm that originated in secondary care, which concurs with previous research. (4) We discuss the implications of this and that most 'severe' cases originated in this setting. We do not state that the harm rate of 9.4% is "for" primary care but make it clear that it is the harm rate found in the records, regardless of source of origin or preventability judgements.

6. Definitions of error and adverse event are outlined in our Methods section. Careful reading of Tables 3 and 4 shows that matching study data are presented for each category using the NCCMERP Error Index employed (which defines near misses, error and harm). We use harm and adverse event interchangeably, which is accepted practice. (5)(6)(7)

7. The NCCMERP index was selected in the absence of a universally agreed system and because it is used in hospital care. We did not adapt it to reduce bias. We provide a rationale: "In secondary-care studies, only categories E to I have been used, as they correlate with actual patient harm occurring. We included all categories because we assumed that relatively minor events may be more common in primary care". This assumption was proved correct. We also made the following recommendation: "It may be reasonable to use only categories E to I in future to target patient harm rather than just error".

8. Johnstone states that he "almost believes" that we "confuse" these terms "on purpose". This comment is lacking in reason and evidence.

9. Johnstone states that because most 'serious' harm originated in secondary care he believes this is "Hardly a major problem for primary care". There are over 300 million consultations annually in UK primary care. (8) A hypothetical harm rate between 0.1% and 5% (even if only half was preventable and most low impact) is of concern to patients, clinicians and the public.

10. Johnstone queries the category E harm example in Table 2 (refer also to our No. 6 and 7 responses). Some harm events may arise from normal or evidence-based care. There may be no associated error (as in this example) or obvious error. To illustrate: a patient is appropriately prescribed an antibiotic but develops an allergic reaction. Record review reveals a similar prescription and outcome previously, but no allergy code was added. Arguably, this is preventable harm.(9)

11. Johnstone is factually incorrect in stating that of "...the 47 events in primary care, 11 were in the range of possible harm rather than actual harm". 47 records contained evidence of some harm i.e. categories E (n=39), F (n=6) and G (n=2). We cannot make this clearer.

12. Johnstone states that we make a "throwaway comment" in the opening paragraph because we suggest that the evidence cited is "...likely to be an underestimation". Based on the evidence in hospital care (10), we hypothesised that the rate in primary care is greater than currently suggested, which is standard research practice. Our findings support our hypothesis.

13. We have no knowledge of those "who would like to demean general practice for their own ends" or of any current or planned "...unnecessary change on GPs..." that is being "imposed" because of our small preliminary study. It is not an academic matter.

14. The issue over the "headline figure" being "inflated" from 9.4% to 9.5% (it was rounded to the nearest 0.5%) is external to the debate on the published paper.

15. At no point is it inferred, directly or indirectly, "...how awful GPs are..."

16. Johnstone accuses us of having an "agenda" to "...inflate harm in general practice" when we (and the six other clinical reviewers) only ever reported harm in the medical records. No evidence to support this statement is offered.

17. We did not exclude the records of the 94 patients (18.8%) who had no contact with primary care in the 12-month study period. If we now exclude them (which is reasonable), the actual harm rate is 47/406 (11.6%), strongly suggesting we underestimated the 'true' level.

18. We concede that our Abstract should read "...27/64 (42%) of error and harm events were judged preventable..." However, this issue is reported clearly in the main paper.

19. We cannot control how others interpret our findings and are always careful to explain the scientific limitations of our study.

Screening medical records for undetected harm events only provides a single perspective. The process cannot capture 'acts of omission' that lead to harm nor are findings likely to correlate with data from patients, incident reporting systems, significant event analyses, complaints, litigation claims and so on. (11)(12)(13) The patient safety problem is highly likely to be greater than can be deduced from screening medical records, even when we account for source of origin and preventability judgements.

Yours faithfully
Carl de Wet MBChB MRCGP MMed (Fam)
Paul Bowie PhD

References
(1) de Wet C, Bowie P. The preliminary development and testing of a global trigger tool to detect error and patient harm in primary-care records. Postgrad Med J 2009 Apr;85(1002):176-180.
(2) Wetzels R, Wolters R, van Weel C, et al. Mix of methods is needed to identify adverse events in general practice: a prospective observational study. BMC Fam Pract 2008;9:35.
(3) Sandars J, Esmail A. The frequency and nature of medical error in primary care: understanding the diversity across studies. Fam Pract 2003 Jun;20(3):231-236.
(4) Forster AJ, Murff HJ, Peterson JF, et al. The incidence and severity of adverse events affecting patients after discharge from the hospital. Ann Intern Med 2003 Feb 4;138(3):161-167.
(5) Elder NC, Pallerla H, Regan S. What do family physicians consider an error? A comparison of definitions and physician perception. BMC Fam Pract 2006;7:73.
(6) Grober ED, Bohnen JM. Defining medical error. Can J Surg 2005 Feb;48(1):39-44.
(7) Yu KH, Nation RL, Dooley MJ. Multiplicity of medication safety terms, definitions and functional meanings: when is enough enough?. Qual Saf Health Care 2005 Oct;14(5):358-363.
(8) Department of Health. Departmental Report 2008. Department of Health: 2008.
(9) Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA 2003 Mar 5;289(9):1107-1116.
(10) Landrigan CP, Parry GJ, Bones CB, et al. Temporal trends in rates of patient harm resulting from medical care. N Engl J Med 2010 Nov 25;363(22):2124-2134.
(11) Kuzel AJ, Woolf SH, Gilchrist VJ, et al. Patient reports of preventable problems and harms in primary health care. Ann Fam Med 2004 Jul-Aug;2(4):333-340.
(12) Thomas EJ, Petersen LA. Measuring errors and adverse events in health care. J Gen Intern Med 2003 Jan;18(1):61-67.
(13) Field TS, Gurwitz JH, Harrold LR, et al. Strategies for detecting adverse drug events among older persons in the ambulatory setting. J Am Med Inform Assoc 2004 Nov-Dec;11(6):492-498.

Conflict of Interest:

None declared

The Editor
Postgraduate Medical Journal, BMA House, Tavistock House, London WC1H 9JR
30 June 2011

Dear Editor,
A 2009 paper by de Wet and Bowie in the Postgraduate Medical Journal(1) has recently been used by its authors' employer to suggest that, "A recent pilot study reviewing a random selection of primary care electronic medical records found a harm rate of 9.5%".(2) Not only is this interpretation of the paper inaccurate, there are errors contained within the paper which require scrutiny.

The de Wet and Bowie paper was designed to 'develop and test a trigger tool to detect patient harm'. The paper frequently confuses records and consultations. There were 500 records and 2,251 consultations included in the study. The 9.4% (not 9.5% as quotedii) harm rate comes from 47 episodes per record, and not per consultation (the more common method of measuring rates of harm and used by the authors in their opening paragraph). Avoiding this confusion would mean the rate of harm is 47 per 2,251 consultations (2.1%).

The authors appear to have difficulty differentiating between primary and secondary care. The trigger tool is designed to detect errors and harm in primary care records. The authors interpret these as errors and harm resultant from primary care, however the tool actually detects errors in both primary and secondary care. Of the 64 episodes of error, adverse event and/or harm discovered in the 2,251 primary care consultations, 17 occurred in secondary care. Thus, the error, adverse event and/or harm rate would more accurately be reported as 47 per 2,251 primary care consultations (or one episode per 47.8 consultations - 2.1% compared to the headline figure of 9.4%).

The authors also frequently confuse harm, error and adverse event. They use the NCCMERP index for categorizing harm. However the first four categories, A,B,C and D only represent potential harm, not actual harm. This is not made clear in the titles of Tables 3 and 4, both of which mention harm, but not error or adverse event, suggesting all the episodes were of harm, not just errors or adverse events which did not lead to harm. This confusion between error, adverse event and harm is so common in the paper one could almost believe it was on purpose. Incidentally, the paper itself points out that all but 8 episodes of harm were of the lowest level and of this 8, 5 were in secondary care. The only 'severe' episode of harm also occurred in secondary care. Hardly a major problem for primary care, but not clearly explained in the headline figure of 9.4%.

So returning to the 47 events in primary care, 11 were in the range of possible harm rather than actual harm. There were only 36 events which actually led to harm, 36/2,251 consultations, or one per 62.5 consultations in primary care (1.6%). Table 4 points out that 27 episodes of harm were preventable. Eighteen of these occurred in primary care and nine in secondary care. Therefore, of the 36 events in primary care which led to actual harm, only 18 were preventable and cannot be described as an error (e.g. a patient reasonably given a statin for heart disease who develops abnormal LFTs is described as an episode of unavoidable harm). So the error rate in primary care which led to harm is more accurately 18 per 2,251 primary care consultations, or one per 125 primary care consultations (0.8%). Interestingly, this is very similar to the 1 per 120 consultations mentioned in the article's opening paragraph, but which the authors seem to think 'is likely to be an underestimation'. The authors do not give their reasoning behind this throwaway comment and they fail to mention that their very own paper supports the complete opposite view.

In addition to the blatant misrepresentation of the data described above, the paper also contains a mistake in the opening Results section. It says "...an adverse event was found in 47 records... Of these, 27 were judged to be preventable (42%)". In fact 27/47 is 57.4% and not 42%. I think that authors mean 27 of all 64 episodes were preventable but they appear to have confused records with consultations, again.

I think I have shown that the paper has inaccuracies, is confused about records and consultations, primary and secondary care, error, harm and adverse event and the difference between unavoidable and preventable. The headline figure of 9.4% has already been inflated to 9.5% by a respected organisation and the employers of the original paper.(2) This figure is being used and repeated in conversations with politicians, managers and a variety of organisations who would like to demean general practice for their own ends. It is possible, using the same data, to say that avoidable harm only occurs in 1 in 125 consultations. I believe the authors have an agenda to inflate harm in general practice as substantiated by their comment at the end of the first paragraph.

I am a hardworking GP and I do not need to be told how awful GPs are, especially when it is based on erroneous, biased articles. I do not want to go to meetings to hear this 9.4% figure used to impose further unnecessary change on GPs. The paper is not worthy of your publication. Therefore I would be grateful if you would withdraw this paper at the earliest opportunity.

Yours sincerely,
Chris Johnstone
A really pissed off GP
The Barony Practice, Northcroft St, Paisley, PA3 4AD

References
1. de Wet C, Bowie P. The preliminary development and testing of a global trigger tool to detect error and patient harm in primary-care records. Postgraduate.Medical Journal 2009; 85(1002): 176-180.
2. An introduction to Safety Climate [accessed at http://www.knowledge.scot.nhs.uk/media/CLT/ResourceUploads/23626/SC%20overview%20for%20practices%20MASTERCOPY.pdf on 30th June 2011].

Conflict of Interest:

I am a GP in the West of Scotland. I am paid sessional work by NHS Education Scotland. I am not sure if these count.