Here it is. I think it is pretty self-explanatory (nonetheless, I have added some additional context below the fold):
Date: Sep 11, 2019
To: <Fiona Godlee>, Editor in Chief, BMJ
RE: BMJ’s scientifically and ethically indefensible decision about Bristol’s Lightning Process studyDear Dr Godlee,
First of all, thank you for the work that you do as editor in chief of the BMJ. I do not at all mean this as faint praise. While my colleagues and I have expressed our concerns, I believe it is important to note that editing a journal is not always easy. Editors play a crucial part in the production line of science and your own contributions, and those of your editorial colleagues, need to be acknowledged.
Secondly, I feel obliged to add my voice to those others who have written to you about the paper “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial” and, particularly, the associated correction that was published.
You will be familiar with the details of our concerns as outlined in previous emails.
From my own perspective, I have been investigating the issues around controversial therapies and human factors (i.e., error risks) in research for nearly two decades. You will be aware that expectancy effects and confirmation biases are very problematic in clinical therapy research in general. They create particular chaos with therapies that are controversial to begin with.
The so-called ‘Lightning Process’ is undoubtedly one such therapy. In fact, it comprises a number of modalities that are normally classified as pseudosciences. It is based largely on so-called Neuro-Linguistic Programming (NLP), which is a completely discredited practice. The creator of the ‘Lightning Process’ is an advocate and provider of several pseudoscientific therapies (e.g., cranial osteopathy; applied kinesiology; hypnotherapy). All told, there is nothing to suggest that the ‘Lightning Process’ is a promising clinical modality. It has no scientific plausibility; it exists because commercially-minded providers of pseudoscientific treatments have successfully identified a market for it. In that regard, it occupies the same space as, say, crystal therapy.
In offering that description, I do not intend to be pejorative. I merely wish to point out that any hypothesis that the ‘Lightning Process’ is effective for any condition constitutes an “extraordinary claim.” By all means should extraordinary claims be tested. Studies of crystal therapy, for example, should be conducted. However, you will appreciate that in science, “extraordinary claims require extraordinary evidence.” Research purporting to establish the efficacy of the ‘Lightning Process’ needs to exhibit a truly compelling evidentiary standard. The claims made inherently challenge our mainstream understanding of human biology. If the ‘Lightning Process’ is in fact shown to work, we will need to re-write our physiology and neurology textbooks.
On the face of it, the evidence contained in this particular paper is far from extraordinary. In fact, as per your own published correction, the study bypassed the normal requirement for prospective registration. This immediately raises the red flag of reporting bias. The evidence accrued from the paper cannot be relied upon because the authors were able to select to report it, without pre-registering their intentions.
In short, confirmation bias was not prevented. Your request to the authors that they tell you whether or not they were biased is very insufficient.
The data produced in light of this procedural and methodological lapse by no means meet the standard of “extraordinary evidence”.
As I understand it, this paper would not have been published had you known about its irregularity at the time of initial review. Therefore it should not be published now. Therefore it should be retracted.
A serious problem with not retracting this paper is that its existence will strongly encourage other researchers to play fast and loose with pre-registration, in the knowledge that their findings can still be published in a BMJ journal. The paper will stand as a high-profile exemplar of ambiguous editorial commitment to scientific rigour.
Kind regards,
Professor Brian Hughes, PhD, FPsSI
School of Psychology
National University of Ireland, Galway
For context, my email is part of a now lengthy correspondence following an Open Letter to the BMJ arising from the correction of the ‘Lightning Process’ paper. I was one of several international scientists to co-sign the original letter, which called for the paper to be properly retracted. Afterwards, several individual signatories provided their personal perspectives as I have done above.
It is worth noting that I deliberately focused on the matter of confirmation bias, as it reflected a point that I felt would benefit from greater emphasis and about which I feel well qualified to comment.
However, there are a number of other problematic issues relating to this paper. These include the very poor methodology employed in the study, where (again) self-report methods are relied upon in the absence of adequate blinding in a clinical trial. This makes the study wholly unreliable, and raises questions about the quality of peer review used by the journal.
In short, why exactly did the journal fail to pick up on such an elementary methodological weakness?
People will have varying opinions about why this happens so frequently. I have written before about how some fields come to be characterised by mass incestuous deference to eminent researchers and their acolytes, and about how unpoliced pandering frequently diminishes the quality of science by neutering peer review.
I suspect I will need to write about that issue again…
…and again…
and again.
For those interested, you can find out more information about the so-called ‘Lightning Process’ here.
