My letter to the BMJ regarding their lax editorial approach to bogus therapy paper

Here it is. I think it is pretty self-explanatory (nonetheless, I have added some additional context below the fold):

Date: Sep 11, 2019
To: <Fiona Godlee>, Editor in Chief, BMJ

RE: BMJ’s scientifically and ethically indefensible decision about Bristol’s Lightning Process study

Dear Dr Godlee,

First of all, thank you for the work that you do as editor in chief of the BMJ. I do not at all mean this as faint praise. While my colleagues and I have expressed our concerns, I believe it is important to note that editing a journal is not always easy. Editors play a crucial part in the production line of science and your own contributions, and those of your editorial colleagues, need to be acknowledged.

Secondly, I feel obliged to add my voice to those others who have written to you about the paper “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial” and, particularly, the associated correction that was published.

You will be familiar with the details of our concerns as outlined in previous emails.

From my own perspective, I have been investigating the issues around controversial therapies and human factors (i.e., error risks) in research for nearly two decades. You will be aware that expectancy effects and confirmation biases are very problematic in clinical therapy research in general. They create particular chaos with therapies that are controversial to begin with.

The so-called ‘Lightning Process’ is undoubtedly one such therapy. In fact, it comprises a number of modalities that are normally classified as pseudosciences. It is based largely on so-called Neuro-Linguistic Programming (NLP), which is a completely discredited practice. The creator of the ‘Lightning Process’ is an advocate and provider of several pseudoscientific therapies (e.g., cranial osteopathy; applied kinesiology; hypnotherapy). All told, there is nothing to suggest that the ‘Lightning Process’ is a promising clinical modality. It has no scientific plausibility; it exists because commercially-minded providers of pseudoscientific treatments have successfully identified a market for it. In that regard, it occupies the same space as, say, crystal therapy.

In offering that description, I do not intend to be pejorative. I merely wish to point out that any hypothesis that the ‘Lightning Process’ is effective for any condition constitutes an “extraordinary claim.” By all means should extraordinary claims be tested. Studies of crystal therapy, for example, should be conducted. However, you will appreciate that in science, “extraordinary claims require extraordinary evidence.” Research purporting to establish the efficacy of the ‘Lightning Process’ needs to exhibit a truly compelling evidentiary standard. The claims made inherently challenge our mainstream understanding of human biology. If the ‘Lightning Process’ is in fact shown to work, we will need to re-write our physiology and neurology textbooks.

On the face of it, the evidence contained in this particular paper is far from extraordinary. In fact, as per your own published correction, the study bypassed the normal requirement for prospective registration. This immediately raises the red flag of reporting bias. The evidence accrued from the paper cannot be relied upon because the authors were able to select to report it, without pre-registering their intentions.

In short, confirmation bias was not prevented. Your request to the authors that they tell you whether or not they were biased is very insufficient.

The data produced in light of this procedural and methodological lapse by no means meet the standard of “extraordinary evidence”.

As I understand it, this paper would not have been published had you known about its irregularity at the time of initial review. Therefore it should not be published now. Therefore it should be retracted.

A serious problem with not retracting this paper is that its existence will strongly encourage other researchers to play fast and loose with pre-registration, in the knowledge that their findings can still be published in a BMJ journal. The paper will stand as a high-profile exemplar of ambiguous editorial commitment to scientific rigour.

Kind regards,

Professor Brian Hughes, PhD, FPsSI
School of Psychology
National University of Ireland, Galway

For context, my email is part of a now lengthy correspondence following an Open Letter to the BMJ arising from the correction of the ‘Lightning Process’ paper. I was one of several international scientists to co-sign the original letter, which called for the paper to be properly retracted. Afterwards, several individual signatories provided their personal perspectives as I have done above.

It is worth noting that I deliberately focused on the matter of confirmation bias, as it reflected a point that I felt would benefit from greater emphasis and about which I feel well qualified to comment. 

However, there are a number of other problematic issues relating to this paper. These include the very poor methodology employed in the study, where (again) self-report methods are relied upon in the absence of adequate blinding in a clinical trial. This makes the study wholly unreliable, and raises questions about the quality of peer review used by the journal.

In short, why exactly did the journal fail to pick up on such an elementary methodological weakness?

People will have varying opinions about why this happens so frequently. I have written before about how some fields come to be characterised by mass incestuous deference to eminent researchers and their acolytes, and about how unpoliced pandering frequently diminishes the quality of science by neutering peer review. 

I suspect I will need to write about that issue again…

…and again…

and again.

For those interested, you can find out more information about the so-called ‘Lightning Process’ here

Human Factors in ME/CFS research

I will be speaking in Belfast tonight, at the Hope 4 ME & Fibro NI annual conference, on the topic of human factors in ME/CFS research. Here’s one of my slides…


The full title of my presentation is Off the PACE and not NICE: Challenges with Evidence in ME/CFS.

(I tweaked that subtitle a couple of times. For reasons.)

I plan to look at the nature of research error as it affects medical and healthcare research more broadly, and — of course — research into ME/chronic fatigue syndrome more specifically. Let’s just say that there is plenty of material to discuss.

Other speakers at the event include Caroline Kingdon of the London School of Hygiene and Tropical Medicine, and David Systrom of Brigham & Womens Hospital, Boston, and Harvard Medical School, who is the keynote.

As always, the organisational efforts of Hope 4 ME & Fibro NI, led by the unstoppable Joan McParland, have been hugely impressive.

Proceedings will be recorded so stay tuned for a video in due course…

The Triumph of Eminence-Based Medicine

Lines are drawn

A quarter of a million Britons have been diagnosed with myalgic encephalomyelitis (ME), the condition also known as chronic fatigue syndrome (CFS). ME/CFS is a severe debilitating illness that renders patients either temporarily or perennially immobile. Sufferers will have often led very active lives before being struck down. After onset, they are forced to give up work or education. Many patients become bedridden for decades. People with ME/CFS experience a full syndrome’s worth of symptoms, dominated by ‘crashes’ of extreme unrelievable tiredness (formally referred to as ‘post-exertional malaise’), along with muscle weakness, digestive problems, pain, and hypersensitivity to light or sound.

The World Health Organisation classified ME/CFS as a neurological disease as far back as 1969. Since then, extensive research has shown that ME/CFS involves disruption of several bodily systems, including the nervous, immune, endocrine, cardiovascular, and gastrointestinal systems. Recent studies have revealed extensive cell-functioning impairment arising from irregularities in patients’ DNA.

Despite all the signs that ME/CFS is an organic disease, the precise biomechanics — and therefore, the most appropriate biomedical treatments — are still unclear. Some patients seem to do well with rintatolimod, an immunostimulant, but not enough to suggest it as a universal therapy.

But if you don’t have ME/CFS or know someone who has, chances are you’ve heard of it anyway. Most likely you will know that the condition is controversial. This is because, well, it is. From diagnostics to treatment to research, the world of ME/CFS is mired in never-ending acrimony. Some researchers claim to have been threatened with violence. A number have won humanitarian awards for standing up for their clinical views. Doctors defending medicine — from patients.

healthy clinic doctor health

Photo by Pixabay on Pexels.com

“All in the mind”

A recurring problem relates to how some doctors perceive the nature of ME/CFS, and the treatment decisions that flow from their assumptions. In Britain especially, the dominant clinical view is that ME/CFS is effectively a psychogenic disorder — one that is caused, or at least exacerbated, by patients’ thoughts, attitudes, and beliefs — and not a truly organic illness that involves irregular body chemistry, treatable using pharmacological or surgical intervention.

In other words, in Britain the ME/CFS world is dominated by psychologists, psychiatrists, and behavioural practitioners. Accordingly, the National Institute of Clinical Excellence (NICE) have adopted guidelines stating that ME/CFS should be treated using either cognitive behaviour therapy (CBT), graded exercise therapy (GET), or a combination of the two.

For years, patients have complained that these treatments are of no use. GET is especially disliked. For most people with ME/CFS, exercise leads to extreme pain and debilitation. The very thought of more physical activity is abhorrent. The fact that doctors prescribe exercise as therapy, and dismiss patients as malingerers when they refuse to comply, is endlessly horrifying. In the minds of some, it is little short of cruel and inhumane abuse.

Many UK clinicians are dogged in their defence of what has become known as the biopsychosocial model of ME/CFS. This is despite the physiological science that suggests they are wrong. Such is the root of the controversy: these doctors are sticking to their guns no matter what scientists — or patients — have to say.

British clinicians, you see, have done their own research, duly published in the best medical journals. They say their findings justify the use of CBT and GET — and nothing else — to treat ME/CFS. Unfortunately, their research serves to compound the controversy, rather than to calm it. The very fact these medics refuse to budge in the face of criticism makes the whole situation worse. The atmosphere has become truly toxic. Patients are haughtily dismissed as aggressive and unreasonable know-nothings.

It’s no wonder they are angry.

Keeping PACE

One of the biggest ever psychology studies to be conducted in a healthcare setting in Britain was the PACE Trial, a five-year investigation involving more than six hundred patients that set out to test the effectiveness of CBT and GET as treatments for ME/CFS. Since its inception in 2005, the PACE Trial has been plagued by problems, ranging from disputed diagnostics, to legal proceedings, to claims of dodgy data. In truth, the study’s a right mess.

Despite being extremely well resourced — and its main results appearing in The Lancet, no less — the PACE Trial contained a number of highly odd design features. For example, it relied heavily on self-report measures, but did not employ blinding. Any psychology undergraduate will tell you that self-report studies need to be blinded. Therefore — by design — the PACE Trial stands as a textbook example of bad research. (Literally, this year, it has appeared as such in a textbook.)

As a result of this flaw, we know that the PACE Trial findings must be untrustworthy. This is even before we see any data. The research was holed below the waterline from the very beginning.

white and black desk calculator on white graphing paper

Photo by Pixabay on Pexels.com

As it happens, the data make for gruesome viewing. The biggest defects relate to outcome-switching. After the PACE Trial investigators began their research — presumably, once they crunched some numbers — they decided to change their definition of “recovery”. They did this by reducing the threshold of (self-reported) physical fitness required for patients to be deemed “well”. In short, they moved the goalposts.

This allowed them to report that one-in-five patients were classified as “recovered” when treated with CBT or GET, a lot more than would have been the case had they stuck to the original study protocol. The problem is that they forgot to do anything about the threshold they set for patients to be declared “sick” in the first place. Hilariously (in the ‘gallows humour’ sense), the result of their on-the-hoof adjustments was that their new threshold for “recovery” was now below the threshold for “sickness”.

Patients could be classified as “recovered” even if their scores for physical well-being declined. Somewhere in the region of one-in-eight patients fell into this surreal category. These patients were sick enough to take part. Their conditions deteriorated after therapy. But the investigators still presented their trajectories as evidence that the PACE Trial treatments had worked.

A lay person would be forgiven for calling this method “absurd”.

To say that the PACE Trial is an extremely weak study is to understate an obvious calamity. Such has been the resulting criticism from around the world that last year the Journal of Health Psychology was able to set aside an entire issue just to publish peer-reviewed articles tearing the PACE Trial to smithereens.


In Psychology in Crisis, I use the PACE Trial as an archetypal example of bad psychological science. It is truly emblematic of the chaos currently engulfing psychology. In recent years paranoia about psychology’s poor replication record has become endemic. Studies like PACE make psychology seem a truly dubious endeavour, and worsen the field’s already dodgy reputation. For scientifically-minded psychologists, it is the stuff of nightmares.

But the PACE Trial is not just a case study in flawed research design and iffy methods. It also highlights the way professional politics can intrude on the scientific process. It shows us how professional groups sometimes try to face down objective criticism, look the other way when ineptitude is exposed, and doggedly stick to their guns in order to avoid threatening their own interests.

These days we have a term to describe how repeated mantra-like talking-points can be employed to suppress legitimate and objective rebuttals. We call it “post-truth” argumentation. Such gamesmanship is not confined to traditional politics. Psychologists can do “post-truth” too.

The world of clinical disease management, especially that of ME/CFS, is just as likely to become embroiled in a culture war.

Wagons being circled

Last month, Reuters published an exclusive story reporting how an influential Cochrane review, which had originally concluded that GET is an effective treatment for ME/CFS, is now to be “temporarily withdrawn” in light of “feedback” received regarding its methods and analysis. The review, published in 2017, had attracted scathing criticisms for what were identified as serious methodological flaws.

For example:

  • The sample of studies included was highly selective
  • Positive outcomes were exaggerated
  • An overall lack of significant treatment effects was either overlooked or misreported
  • Studies relying on self-report measures, which are usually considered weak in research terms, were analysed as though they were just as strong as studies that used objective measures
  • The reviewers decided to shift their analytic approach after they had started their work (in psychology, we might refer to this as “a pattern of behaviour”). Their adjustments ultimately enabled them to report maximum effects for GET. Deviating from pre-planned protocols is a huge risk to the validity of any review, and is contrary to Cochrane good practice guidelines.

Many observers also pointed out that the review’s authors included investigators involved in the now beleaguered PACE Trial, meaning that they were effectively involved in a process of marking their own homework.

These criticisms, along with many others, were formally submitted to Cochrane and are currently viewable on their website.

In explaining Cochrane’s decision, Reuters highlighted pressure from patient activists who object to ME/CFS being depicted as a psychological condition. Discernible in the Reuters report was the sound of professional wagons being circled. A former head of Britain’s Medical Research Council was quoted as complaining that withdrawal decisions should not be dictated by patient lobbying.

This week, the Guardian posted a ‘science’ podcast discussing the Cochrane review withdrawal. In the blurb, they perpetuated the (inaccurate) Reuters reporting line, embellishing it with their own reference to “verbal abuse”:

…researchers looking at efficacy of these therapies claim they are verbally abused. But now, it may have gone a step further. Reuters recently reported that ‘a respected science journal is to withdraw a much-cited review of evidence on an illness known as chronic fatigue syndrome amid fierce criticism and pressure from activists and patients.’

More strikingly — perhaps even suspiciously — the producers of the Guardian science podcast decided that this week’s guest should be none other than a colleague of one of the authors of the withdrawn Cochrane review, and — for good measure — himself an author of the maligned PACE trial. Not just marking his own homework anymore, but now — in defiance of expert criticism — defending how well it had been marked.

In post-truth terms, it all felt a bit like an episode of Fox & Friends.

Cochrane doesn’t withdraw reviews lightly. In reality, this review was withdrawn from the public record because for nearly two years independent researchers had highlighted a steady stream of specific scientific criticisms. The Editor in Chief of Cochrane stated clearly that “this is not about patient pressure.”

In other words, the methodological problems proved to be just too glaring. Someone who headed the Medical Research Council should really be able to see that, as should the producers of the Guardian’s science podcast.

Perhaps they are the ones who are succumbing to partisan pressures.

two person doing hand shake

Photo by rawpixel.com on Pexels.com

Friends of friends

It is worrying that review practices intended to be scientifically objective seem so susceptible to distortion by social network effects, cultural etiquette, and a prerogative to preserve colleagues’ professional reputations.

The hegemony of the cognitive approach to ME/CFS is clearly partisan. It is at variance with much of the scientific literature. The very fact that its status is disputed exposes as logically unwise any claim that there is only one side to the ME/CFS story.  Its endurance results from professional politics, not scrupulous science.

It is eminence-based, rather than evidence-based, medicine.

It is also very much a British thing. The corresponding American authority — the National Academy of Medicine — has declared ME/CFS an organic condition requiring biomedical treatment. Its genetic and neurochemical signs are well recognised (although its precise physiology is not yet established), as are the damaging effects of exercise therapies of the kind promoted as standard in the UK. It is the British clinical professions whose establishments believe that ME/CFS is psychogenic, that its main dynamics are “all in the mind”.

Medical conditions are not supposed to vary by international border. An infection contracted in one country should be expressed the same anywhere, and biomedical treatments should have a uniform effect regardless of what nation you live in. That the UK is an outlier when it comes to ME/CFS simply makes no sense.

Except possibly in one respect. We should not see ME/CFS as resulting from mass cognitive hysteria among a quarter of a million UK citizens. Rather, we could see the biopsychosocial theory of ME/CFS as a grand sanctimonious delusion shared by a professional clique who, for circumstantial reasons, find themselves dominant in British behavioural healthcare.

The echo chamber in which reviewers review each other’s work, award each other’s grants, and line up as one other’s acolytes, suggests that little of this will change any time soon.

Bad science is bad enough when it is just science. In the case of ME/CFS, where flawed research materially damages the lives of hundreds of thousands of blameless people, it is nothing short of a scandal, about which the establishment should feel acute embarrassment and, ideally, shame.

* * *


Edited, 3/11/18: Description of podcast rephrased to clarify that its guest was a colleague of one of the Cochrane review authors, as well as an author of PACE trial 

%d bloggers like this: