Who is responsible for investigating suspected ... - Wiley Online Library

Anaesthesia 2012, 67, 455–473

tion rates leave a rich source of data for us to analyse, so that we can learn aberrant patterns and in time, detect much earlier the warning signs. Carlisle is to be congratulated: his is an astonishing, altruistic piece of work that helps expunge the literature of some (at best) highly unusual data. The purpose of experimentation is to learn about nature. If the results of experiments are not genuine, then however prolific, influential or politically powerful their author, the results will not withstand statistical scrutiny, cannot be repeated, or will lead to models for our understanding of nature that are so bizarre as to be proven false. For nature cannot be fooled.

Competing interests No external funding or competing interests declared.

Editorial

J. J. Pandit Professor Nuffield Department of Anaesthetics John Radcliffe Hospital Oxford, UK Email: [email protected]

8.

9.

References 1. Haldane JBS. The faking of genetical results. Eureka 1964; 27: 21–4. 2. Carlisle JB. The analysis of 169 randomised controlled trials to test data integrity. Anaesthesia 2012; doi: 10.1111/j.1365-2044.2012.07128.x. 3. Bellhouse DR. Probability in the sixteenth and seventeenth centuries: an analysis of Puritan casuistry. International Statistical Review 1988; 56: 63–74. 4. Philip U, Haldane JBS. Relative sexuality in unicellular algae. Nature 1939; 143: 334. 5. Grüneberg H, Haldane JBS. Congenital hyperglycaemia in mice. Nature 1940; 145: 704–5. 6. Pandit JJ. ‘Hardy’s Law’ and genomics in Anaesthesia. Anaesthesia 2008; 63: 1284–7. 7. Pandit JJ. The analysis of variance in anaesthetic research: statistics, history

10.

11.

12.

and biography. Anaesthesia 2010; 65: 1212–20. Fisher RA. The correlation between relatives on the supposition of Mendelian inheritance. Transactions of the Royal Society of Edinburgh 1918; 52: 399–433. Janicki PK, Vealey R, Liu J, Escajeda J, Postula M, Welker K. Genome-wide Association study using pooled DNA to identify candidate markers mediating susceptibility to postoperative nausea and vomiting. Anesthesiology 2011; 115: 54–64. Buyse S, George SL, Evans S, et al. The role of biostatistics in the prevention, detection and treatment of fraud in clinical trials. Statistics in Medicine 1999; 18: 3425–51. Feynman RP. Personal reflections on the reliability of the Shuttle. Appendix F, In: Report of the Presidential Commission on the Space Shuttle Challenger Accident, 1986. National Aeronautics and Space Administration, Washington, USA. http://science.ksc.nasa.gov/shuttle/ missions/51-l/docs/rogers-commission/table-of-contents.html (accesssed 07 ⁄ 11 ⁄ 2011). Shafer SL. Shadow of doubt. Anesthesia and Analgesia 2011; 112: 498–500. doi: 10.1111/j.1365-2044.2012.07114.x

Editorial Who is responsible for investigating suspected research misconduct? Serious misconduct in medical research, such as data fabrication, can distort the evidence base and harm patients. Although it is impossible to know exactly how common such misconduct is, it is also impossible to deny that it does occur and that it requires proper investigation. Recent cases include the German anaesthetist, Joachim Boldt, who has had almost 90 articles retracted because 462

of lack of evidence of appropriate ethical approval and is currently being investigated for fraud [1, 2]. An example of flawed work harming patients can be found in the case of the American geneticist Anil Potti. His method for predicting responses to chemotherapy was used to determine which treatment cancer patients received in clinical trials. However, the method was found to

be unreliable and the trials were halted, but only after some patients had, presumably, received suboptimal treatment [3]. Sadly, both the Boldt and Potti cases also highlight problems in investigating suspected misconduct or unreliable work in two countries – Germany and the USA – generally thought to have good national research integrity systems. All German

Anaesthesia ª 2012 The Association of Anaesthetists of Great Britain and Ireland

Editorial

universities are required to have a committee that investigates scientific misconduct and, for federally funded research, can also consult the national ombudsman (actually a panel of three independent scientists) [4]. However, research in German teaching hospitals (where Boldt worked) is less tightly regulated and, according to one commentator ‘‘Physicians performing clinical studies in these environments do so with little research oversight’’ and such hospitals usually have ‘‘very little experience with conducting investigative procedures’’ [4]. In the case of Boldt, a journal editor raised concerns with his institution but it took several months to determine who was responsible for the ethical conduct of the research [2]. Eventually, the investigation was initiated by the state chamber of physicians (the Landesa¨rztekammer), then passed on to the national ombudsman who formed a review committee. This committee could find no evidence of ethical approval for much of Boldt’s research and therefore advised that the publications should be retracted, but it did not address the question of whether the published findings were reliable. Allegations of fraud are therefore being investigated by another committee at Boldt’s institution (the Klinikum Ludwigshafen) and have also been referred for criminal investigation because failure to follow the physicians’s (Bundesa¨rztekammer) code of conduct is a criminal offence in Germany [4]. In the case of Anil Potti, statisticians working at another university raised concerns about the reliability of work he had published in 2007 [3]. This prompted Potti’s institution (Duke University, North Carolina) to

Anaesthesia 2012, 67, 455–473

investigate his work and, in September 2009, suspend the clinical trials (which had been started in 2007). Three months later (in December 2009), the Duke review concluded that Potti’s work was valid and therefore the trials were restarted. However, it transpired that the university had not provided the review committee with the evidence from the statisticians and had also refused to provide the statisticians with a copy of the review team’s report. The whistleblowers were therefore forced to obtain a copy of the report via a Freedom of Information request to the trial funder and, having done so, continued to raise concerns. The trials were stopped again in July 2010 (after a total of 110 patients had been enrolled) after irregularities with Potti’s cv came to light. Potti’s 2007 paper was retracted in November 2010 and he has since retracted or corrected several others. Another common feature of both the Boldt and Potti cases was that concerns were raised by external whistleblowers, in one case a journal editor, in the other a statistician, who were persistent in seeking a proper investigation. In the Boldt case, the institution responded appropriately but there was a delay in determining which body was responsible for the investigation (and, in the end, this case is being investigated by three different groups). In the Potti case, the initial institutional response was clearly inadequate. Such confusion, delay, unwillingness or inability of institutions to investigate suspected misconduct is, sadly, not uncommon. An analysis of cases brought by editors to the Committee on Publication Ethics (COPE) involving con-


tact with institutions between 2007 and 2011 showed that 12 out of 24 received an unsatisfactory response (although this cannot be used to estimate the frequency of such problems since editors probably bring their most troublesome cases to COPE, so the study sample is likely to be skewed) [5]. Another example of a journal editor battling for over 10 years, ultimately unsuccessfully, to get suspected misconduct investigated has been described in an article in the British Medical Journal [6] that notes ‘‘when research misconduct is suspected and the researcher is working outside the jurisdiction of official research bodies, there is nowhere for editors to turn. If they want to investigate their concerns they are invariably forced to go it alone – a lengthy, costly, and difficult process’’. Richard Smith (the editor involved) is quoted as saying ‘‘It is shameful that this case has been left festering so long’’ and that he feels that ‘‘The failure is in part mine’’ [6]. But Smith also notes the difficulties he faced and the fact that he had to do this time-consuming work ‘‘in addition to my day job’’ and that ‘‘the bigger shame lies with the scientific community that lacks means to investigate these international scandals and has to leave it to an individual journal’’. In another article, Smith has made it clear that he thinks that editors should not attempt to investigate misconduct since they cannot provide due process and have no legal standing in such cases [7]. He notes that ‘‘The role of editors is to pass on accusations to the relevant authority … A difficult question is to know how much evidence you need to have. 463

Anaesthesia 2012, 67, 455–473

There is an understandable tendency to think that you need a great deal of evidence to make such a dramatic allegation, but gathering it is often difficult, expensive, and time consuming, and can create many problems’’. The Committee on Publication Ethics (which Richard Smith helped to found) reiterates the principle that editors should not attempt to investigate suspicions or allegations of misconduct but that they do have a duty to alert the relevant authorities and ensure that a proper investigation is conducted. However, the COPE Code of Conduct notes that this can be an onerous duty [8]. Smith has written ‘‘editors … have a duty … to persist in making sure that justice is done’’ and remarks that ‘‘institutions are inclined to let accusations fade away’’ [7]. He describes writing to employers and authorities in 10 countries and often receiving no response. Not surprisingly, editors may become discouraged and feel it is pointless to contact an institution – especially one in another country – because past experience leads them to doubt it will have any effect. Another problem we sometimes observe at COPE is that editors may be reluctant to pass on suspicions of misconduct, especially relating to lesser types of misconduct such as redundant publication, which they may view as potentially less harmful than fabrication, because they are afraid that the accused authors will not receive a fair hearing. Even if editors have confidence in the fairness of the system, they may fear inordinate bureacracy or a disproportionate response that could 464

Editorial

harm an innocent researcher. Robert Daroff (former editor of Neurology) has described a US case involving allegations of data falsification that took nine years to resolve [9]. Spece and Bernstein report another case from the USA which ran from 1997 to 2003 in which the ultimately exonerated researcher ‘‘was left bankrupt, emotionally exhausted, and apparently unable to pursue relief within the federal bureaucracy’’ [10]. The case described by Spece and Bernstein raises another important issue, namely the cost of investigations. The case they describe apparently cost the university ‘‘well over a million dollars’’. More recently, Michalek and colleagues have described a single misconduct investigation carried out at the Rosewell Park Cancer Institute (in the USA) which cost around $525 000 (based on direct costs and time spent by faculty and witnesses) [11]. They note that if these costs are applied to the 217 cases reported to the US Office for Research Integrity (ORI [http:// ori.hhs.gov/]) in the previous year, the direct costs to institutions would be over $110 million. This is in addition to the annual budget of the ORI which is currently almost $10 million [12]. However, while some investigations are necessarily complex and therefore costly, editors sometimes come across clear evidence of wrongdoing, for example inappropriate manipulation or duplication of a digital image revealed by screening, or extensive plagiarism revealed by text-matching software. If such cases are discovered before publication, editors may be tempted simply to

reject the submission, but COPE advises that they have a duty to report misconduct in such cases to prevent the authors from simply submitting their work to a less vigilant journal [8]. Although editors may be reluctant to contact institutions, another reason why COPE recommends this is to prevent authors from removing evidence of problems from their report rather than actually correcting them. If such cases are discovered after publication, editors may feel their primary responsibility is to prevent readers from being misled by retracting the article or publishing a correction, rather than get entangled with institutional investigations. Indeed, in an ideal world, editors’ duties would be limited to alerting institutions or other appropriate bodies of suspected misconduct and then correcting the record if fraud or errors affecting the reliability of published findings are indicated by a proper investigation. As well as the examples of poor practice already given, we can also cite recent cases where institutions responded promptly and effectively. For example, allegations about the work of the Dutch psychologist Diederik Stapel were swiftly and thoroughly investigated by universities at Tilburg, Groningen and Amsterdam, which issued detailed reports that resulted in retractions [13]. Similarly, the University of Connecticut issued a 60 000 page report on the work of Dr Dipak Das and notified all journals that had published affected work (although this investigation did take almost 3 years to complete) [14]. But, sadly, as the other examples have shown, institutions do not always respond appropriately. Perhaps this is not


Editorial

surprising as institutions have a considerable conflict of interest, especially when it comes to publishing or sharing results of investigations, since they will, understandably, wish to protect their reputation [15]. We therefore need a change of culture so that a properly conducted investigation is viewed as a sign of a well-run and trustworthy institution rather than a cause for shame. Rather than avoiding any institution that has published results of misconduct hearings, perhaps researchers should shun those that cannot provide evidence of such investigations, since this suggests they are brushing concerns under the carpet and not taking their responsibilities seriously. Given research institutions’ inherent conflicts of interest, and their poor track record, some form of oversight would seem essential. In the USA, the ORI oversees investigations but only covers research funded by the National Institutes of Health. Similarly, the national German medical funder (the Deutsche Forschungsgemeinschaft) has an ombudsman, but doctors working on non-state funded research are not covered. Australia established its Research Integrity Committee in 2011, which also covers only publicly funded research and will not investigate cases directly, although it will look into whether institutions handled the investigation appropriately (see http://www.arc.gov.au/general/ research_integrity.htm). In the UK there is currently no official, national body since funding for the UK Research Integrity Office (UKRIO) – which was established in 2006 to provide advice and promote good practice – was cut in 2010

Anaesthesia 2012, 67, 455–473

(although UKRIO continues to operate, is seeking alternative funding, and has recently received support from a consensus statement resulting from a meeting organised by COPE and the British Medical Journal [16]). In most countries, there is no authority whom an editor or other whistleblower can contact with concerns, and, even in countries that do have a national body, their remit is only partial so not all researchers are covered. In some cases, medical licensing bodies (such as the General Medical Council in the UK) may have a role, but their track record for efficiency in investigating misconduct is not perfect [17], they will generally investigate only the most serious forms of misconduct, and, of course, they have no jurisdiction over researchers who are not doctors. Some commentators have argued that specialist forensic investigators are needed – but somebody will have to pay for them [15]. Chalmers and Haines have argued that research funders should pay for such services but this seems over-optimistic, at least in the UK, given funders’ apparent unwillingness to fund the less controversial (and less costly) UKRIO. The UK Government also seems unwilling to act, noting in a response to a House of Commons Select Committee inquiry into peer review that it was ‘not minded to set in train … arrangements for an oversight body’ [18]. Research misconduct is hard to prevent because it represents a systems failure, so its causes and solutions are multifactorial. Similarly, systems for investigating misconduct appear to be difficult to establish, fund, and monitor. To be successful,


several different constituencies probably need to be involved. It is not sufficient to leave responsibility for investigations with universities and hospitals; other players such as funders, regulators and editors need to work together. Investigating misconduct is not simple, and the consequences can be serious, therefore expertise and a high level of professionalism are essential. These will be fostered if there are organisations providing advice and training, and promoting good practice. Institutions and individuals will only be able to learn from the experiences of others if there is exchange of information. Transparency should also increase confidence in the system. While it may be helpful to impose confidentiality during a hearing, findings should be published, and interested parties (such as journals that have published affected research) should be properly informed. The Committee on Publication Ethics takes the view that journals and institutions should collaborate closely on cases of suspected misconduct and is currently developing guidance to encourage this (which it plans to publish on the COPE website: http:// www.publicationethics.org). Richard Smith referred to research misconduct as the ‘poisoning of the well’ [7]. Unless we establish better methods for investigating misconduct, researchers and clinicians will be left with a lingering doubt about the purity and therefore trustworthiness of the evidence they rely on to make decisions.

Competing interests I am Chair of COPE – this is an unpaid position. I co-chaired and 465

Anaesthesia 2012, 67, 455–473

planned the British Medical Journal ⁄ COPE meeting which produced the consensus statement (ref [15]).

E. Wager Chair, Committee on Publication Ethics Publications Consultant Sideview Princes Risborough, UK Email: [email protected]

References 1. Editors-in-Chief Statement Regarding IRB Approval of Clinical Trials by Joachim Boldt. http://onlinelibrary.wiley.com/ store/10.1111/%28ISSN%291365-2044 /asset/homepages/Joint_Editorial_ Statement__12_Mar_2011.pdf (accessed 17 ⁄ 2 ⁄ 2012). 2. Shafer SL. Shadow of Doubt. Anesthesia and Analgesia 2011; 112: 498–500. 3. Reich ES. Cancer trial errors revealed. Nature 2011; 469: 139–40. 4. Förstermann U. Research oversight in Germany: safeguards and shortcomings. Anesthesia and Analgesia 2011; 112: 504–6. 5. Wager E. Coping with scientific misconduct. British Medical Journal 2011; 343: d6586.

Editorial 6. White C. Suspected research fraud: difficulties of getting at the truth. British Medical Journal 2005; 331: 281–8. 7. Smith R. Research misconduct: the poisoning of the well. Journal of the Royal Society of Medicine 2006; 99: 232–7. 8. Committee on Publication Ethics. COPE Code of Conduct. http://publicationethics.org/code-conduct (accessed 17 ⁄ 02 ⁄ 2012). 9. Daroff RB. Scientific misconduct and breach of publication ethics: one editor’s experience. Medicine and Law 2007; 26: 527–33. 10. Spece RG, Bernstein C. Scientific misconduct and liability for the acts of others. Medicine and Law 2007; 26: 477–91. 11. Michalek AM, Hutson AD, Wicher CP, Trump D. The costs and underappreciated consequences of research misconduct: a case study. PLoS Medicine 2010; 7: e10000318. 12. Office of Research Integrity Budget. In: Department of Health and Human Services, Justification of Estimates for Appropriations Committees, Fiscal Year 2011; p79. http://www.hhs.gov/asfr/ ob/docbudget/2011cj.pdf (accessed 20/12/2012). 13. Callaway E. Report finds massive fraud at Dutch universities. Nature 2011; 479: 15. 14. Scientific journals notified following research misconduct investigation. UConnToday (blog) http://today.uconn.edu/

15.

16.

17.

18.

blog/2012/01/scientific-journals-notified-following-research-misconductinvestigation/ (accessed 20 ⁄ 02 ⁄ 2012). Chalmers I, Haines A. Skilled forensic capacity needed to investigate allegations of research misconduct. British Medical Journal 2011; 342: d3977. A consensus statement on research misconduct in the UK. British Medical Journal 2012; 344: e1111 (also available at http://publicationethics.org/files/u7140/ A_consensus_statement_on_research_ misconduct_in_the_UK.pdf (accessed 20 ⁄ 02 ⁄ 2012). Hey E, Chalmers I. Mis-investigating alleged research misconduct can cause widespread, unpredictable damage. Journal of the Royal Society of Medicine 2010; 103: 132–7. Peer review in scientific publications: Government and Research Councils UK Responses to the Committee’s Eighth Report of Session 2010-12. 18 October 2011. http://www.publications.parliament.uk/pa/cm201012/cmselect/ cmsctech/1535/1535.pdf (accessed 17 ⁄ 2 ⁄ 2012). doi: 10.1111/j.1365-2044.2012.07132.x

Editorial The Tariff – traps for the unwary Until very recently, clinicians have spent little time considering the financial consequences of their clinical decisions or how their organisations can become more efficient or financially successful. Over the last 20 years, clinicians in a number of countries have paid much more attention to how their organisations actually function. The interest in the financial issues around the provision 466

of medical care, that is evidenced in this [1] and recent [2–9] issues of Anaesthesia, reflects the impact of the growth of payment mechanisms linking activity directly to funding (e.g. ‘Payment by Results’; PbR) that are now common in Europe, North America and Australia. This parallels the development of internal management systems that attempt to provide measures of the economic

success or otherwise of particular hospital services or departments. These changes are often driven by the realisation of many clinicians that they can improve patient care by ensuring that their services are well managed, and by many hospital managers appreciating that clinicians are most likely to have the insight, motivation and ability to make positive changes.
