Category: Scientific literacy

My letter to the BMJ regarding their lax editorial approach to bogus therapy paper

Here it is. I think it is pretty self-explanatory (nonetheless, I have added some additional context below the fold):

Date: Sep 11, 2019
To: <Fiona Godlee>, Editor in Chief, BMJ

RE: BMJ’s scientifically and ethically indefensible decision about Bristol’s Lightning Process study

Dear Dr Godlee,

First of all, thank you for the work that you do as editor in chief of the BMJ. I do not at all mean this as faint praise. While my colleagues and I have expressed our concerns, I believe it is important to note that editing a journal is not always easy. Editors play a crucial part in the production line of science and your own contributions, and those of your editorial colleagues, need to be acknowledged.

Secondly, I feel obliged to add my voice to those others who have written to you about the paper “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial” and, particularly, the associated correction that was published.

You will be familiar with the details of our concerns as outlined in previous emails.

From my own perspective, I have been investigating the issues around controversial therapies and human factors (i.e., error risks) in research for nearly two decades. You will be aware that expectancy effects and confirmation biases are very problematic in clinical therapy research in general. They create particular chaos with therapies that are controversial to begin with.

The so-called ‘Lightning Process’ is undoubtedly one such therapy. In fact, it comprises a number of modalities that are normally classified as pseudosciences. It is based largely on so-called Neuro-Linguistic Programming (NLP), which is a completely discredited practice. The creator of the ‘Lightning Process’ is an advocate and provider of several pseudoscientific therapies (e.g., cranial osteopathy; applied kinesiology; hypnotherapy). All told, there is nothing to suggest that the ‘Lightning Process’ is a promising clinical modality. It has no scientific plausibility; it exists because commercially-minded providers of pseudoscientific treatments have successfully identified a market for it. In that regard, it occupies the same space as, say, crystal therapy.

In offering that description, I do not intend to be pejorative. I merely wish to point out that any hypothesis that the ‘Lightning Process’ is effective for any condition constitutes an “extraordinary claim.” By all means should extraordinary claims be tested. Studies of crystal therapy, for example, should be conducted. However, you will appreciate that in science, “extraordinary claims require extraordinary evidence.” Research purporting to establish the efficacy of the ‘Lightning Process’ needs to exhibit a truly compelling evidentiary standard. The claims made inherently challenge our mainstream understanding of human biology. If the ‘Lightning Process’ is in fact shown to work, we will need to re-write our physiology and neurology textbooks.

On the face of it, the evidence contained in this particular paper is far from extraordinary. In fact, as per your own published correction, the study bypassed the normal requirement for prospective registration. This immediately raises the red flag of reporting bias. The evidence accrued from the paper cannot be relied upon because the authors were able to select to report it, without pre-registering their intentions.

In short, confirmation bias was not prevented. Your request to the authors that they tell you whether or not they were biased is very insufficient.

The data produced in light of this procedural and methodological lapse by no means meet the standard of “extraordinary evidence”.

As I understand it, this paper would not have been published had you known about its irregularity at the time of initial review. Therefore it should not be published now. Therefore it should be retracted.

A serious problem with not retracting this paper is that its existence will strongly encourage other researchers to play fast and loose with pre-registration, in the knowledge that their findings can still be published in a BMJ journal. The paper will stand as a high-profile exemplar of ambiguous editorial commitment to scientific rigour.

Kind regards,

Professor Brian Hughes, PhD, FPsSI
School of Psychology
National University of Ireland, Galway

For context, my email is part of a now lengthy correspondence following an Open Letter to the BMJ arising from the correction of the ‘Lightning Process’ paper. I was one of several international scientists to co-sign the original letter, which called for the paper to be properly retracted. Afterwards, several individual signatories provided their personal perspectives as I have done above.

It is worth noting that I deliberately focused on the matter of confirmation bias, as it reflected a point that I felt would benefit from greater emphasis and about which I feel well qualified to comment. 

However, there are a number of other problematic issues relating to this paper. These include the very poor methodology employed in the study, where (again) self-report methods are relied upon in the absence of adequate blinding in a clinical trial. This makes the study wholly unreliable, and raises questions about the quality of peer review used by the journal.

In short, why exactly did the journal fail to pick up on such an elementary methodological weakness?

People will have varying opinions about why this happens so frequently. I have written before about how some fields come to be characterised by mass incestuous deference to eminent researchers and their acolytes, and about how unpoliced pandering frequently diminishes the quality of science by neutering peer review. 

I suspect I will need to write about that issue again…

…and again…

and again.

For those interested, you can find out more information about the so-called ‘Lightning Process’ here

Psychology in Crisis: My interview with the ‘Medical Error’ podcast

Here I am discussing psychology, the replication crisis, medical error, CFS/ME, the PACE Trial, political collapse, human extinction, and more…

‘Medical Error Interviews’ is a podcast out of Canada, hosted by Scott Simpson. See all the details, including all the episodes of ‘Medical Error Interviews’, on Podbean.

You can also support the podcast on Patreon.

 

‘Psychology in Crisis’ is now available

About the Author

Imprint: 2018
Psychology in Crisis
Author: Brian M. Hughes
Publisher: Palgrave, London

ISBN-10: 1352003007
ISBN-13: 978-1352003000

Click here to view on Palgrave Macmillan
Click here to view on Amazon.co.uk
Click here to view on Amazon.com
Click here to view on Amazon.in
Click here to view on Amazon.co.jp
Click here to view on Barnes & Noble
Click here to view on Book Depository
Click here to view on The Guardian Bookshop
Click here to view on Waterstones
Click here to view on WHSmith

From the cover: Throughout the history of psychology, attempting to objectively measure the highly dynamic phenomenon of human behaviour has given rise to an underappreciated margin of error. Today, as the discipline experiences increasing difficulty in reproducing the results of its own studies, such error not only threatens to undermine psychology’s credibility but also leaves an indelible question: Is psychology actually a field of irreproducible science?

In this thought-provoking new book, author Brian Hughes seeks to answer this very question. In his incisive examination of the various pitfalls that determine ‘good’ or ‘bad’ psychological science – from poor use of statistics to systematic exaggeration of findings – Hughes shows readers how to critique psychology research, enhance its validity and reliability, and understand the strengths and weaknesses of the way psychology research is produced, published, and promulgated in the twenty-first century.

This book is essential reading for students wanting to understand how to better scrutinise psychological research methods and results, as well as practitioners and those concerned with the replication debate.

Psychology in Crisis is an unflinching tour of the challenges of doing psychological science well. Brian Hughes describes six crises facing psychology that could make one think that all is lost. But it is not. At their core, the crises are illustrations of just how hard it is to study human behavior and, simultaneously, why it is worth doing. Hughes closes with a path toward a science that is robust, transparent, and self-skeptical to help accelerate discovery and ensure that psychology meets its potential as a scientific enterprise.” — Professor Brian Nosek, Professor in psychology at the University of Virginia and Executive Director for the Center for Open Science


Contents

Chapter 1 ‘The Same Again, But Different’: Psychology’s Replication Crisis
Chapter 2 ‘Black Is White’: Psychology’s Paradigmatic Crisis
Chapter 3 ‘Never Mind the Quality, Feel the Width’: Psychology’s Measurement Crisis
Chapter 4 ‘That Which Can Be Measured’: Psychology’s Statistical Crisis
Chapter 5 ‘We Are The World’: Psychology’s Sampling Crisis
Chapter 6 ‘Fitter, Happier, More Productive…’: Psychology’s Exaggeration Crisis
Chapter 7 From Crisis to Confidence: Dealing with Psychology’s Self-Inflicted Crises
%d bloggers like this: