The BMJ have published a “living systematic review” of interventions for the management of Long Covid. It sets out to gather all relevant studies, and to comb their findings in order to see what works and what doesn’t. Having assessed 24 trials looking at drug and non-drug therapies, they draw this pretty oddly-worded conclusion:

“Moderate certainty evidence suggests that CBT and physical and mental health rehabilitation probably improve symptoms of long covid.”

I have had a look over everything and I have to say, I am far from convinced. I am also quite puzzled by some of the choices made in presenting this review to the public. Something seems very off about how it’s all done.

First thing to say is that with systematic reviews — “living” or otherwise (I don’t know…dead?) — there is always one fundamental rule: namely, Garbage In = Garbage Out. In other words, reviewers can only work with the studies that are in front of them. If the studies are poor, then no review in the world will improve the evidence base.

This is why people who do systematic reviews examine the quality of the studies they are reviewing. Quality judgements are supposed to be incorporated into decision-making. If studies are of poor quality then their findings are considered to stand as poor evidence. If a study is biased, then the findings are also presumed to be biased. Ordinarily, therefore, biased evidence can have no standing. Indeed, you could reasonably argue that biased evidence is worse than no evidence at all; because, biased evidence is systematically misleading.

So, here is my TL;DR on this new BMJ review: the authors are able to produce their findings only by looking at studies they themselves declare are biased. When these biased studies are excluded, the so-called evidence disappears.

The BMJ review reveals that there is, in fact, NO UNBIASED EVIDENCE THAT CBT OR PHYSICAL REHABILITATION IMPROVE SYMPTOMS OF LONG COVID.

* * *

The authors do a very strange thing when reporting their risk-of-bias assessments. They do it very selectively. For example, in a detailed table in their paper, they summarise the risk-of-bias assessment for drug interventions (in Figure 2). That is fine, but it is somewhat strange to go into all that detail because those trials yielded no findings for therapeutic effectiveness. The authors’ supposed main findings were for non-drug interventions (CBT and physical therapy). But, for some reason, the risk-of-bias assessments for CBT and physical therapy trials are not included in the paper at all. Instead they are posted as supplements to the BMJ website, out of sight of the majority of readers who will simply download the actual paper or otherwise receive it second-hand (e.g., by email).

In other words, the risk-of-bias assessments crucial for interpreting the authors’ main findings are, for some reason, hidden away where most people will never see them. This is important because these assessments change everything.

Let’s take a look at why.

* * *

In their online “Supplement 5”, the authors summarise their risk-of-bias analyses for trials looking at physical activity and rehabilitative interventions.

Six trials were included in this risk-of-bias analysis: Alshaimaa (2023); McNarry (2023); Mooren (2023); Nambi (2022); Omarova (2022); and Romanet (2022/RECOVER). Four of these six trials were found to have significant levels of bias, in multiple areas. Alshaimaa (2023), McNarry (2023), Mooren (2023), and Omarova (2022) were found to have “High” or “Probably High” risk of bias in one, two, or three of the diminensions considered. For example, the trial by McNarry (2023) was found to have “High” or “Probably High” risk of bias with regard to (a) missing outcome data, (b) biased outcome measurements, and (c) deviations from the intended study intervention.

In other words, all but two of the trials were found to be biased in ways that render their results automatically misleading.

Of the two remaining trials, Nambi had measures for “Quality of Life” as an outcome.

Only the other trial — Romanet (2022/RECOVER) — had outcomes relating to actual physical health: namely, dyspnea (shortness of breath) and physical function (as measured using the SF-12). This particular trial was very small, but it was also problematic in other ways. The trial compared 27 patients who were given “exercise training” and 33 patients who received “standard physiotherapy”. So there was no actual control group. That is, there was no group of patients who were simply left alone and given no physical intervention. One of the two groups was found to have better dyspnea scores after 90 days, but both groups were treated using “physical activity”. As such, the study outcome does not support the conclusion that “physical activity and rehabilitative interventions” are better than no such treatment. That hypothesis was not even tested.

In summary, therefore, after you remove the biased studies, the accumulated research does not support the use of “physical activity and rehabilitation interventions” for long COVID.

Of course, when you *do* include the biased studies, you can cobble together a finding. But the (known) presence of bias means that that finding will be misleading. And what kind of finding is that?

* * *

In their online “Supplement 6”, the authors summarise the risk-of-bias analyses for trials of behavioural interventions.

These are the trials that looked at CBT. Three trials were included: Kuut (2023/ReCOVer), Samper-Pardo (2023), and Toussaint (2023). All three were found to have “High” risk of bias in one dimension and “Probably High” risk of bias in either one or two additional dimensions. So, in summary, ALL studies — every single one of them — were contaminated by serious levels of methodological bias. By definition, this means that all of the findings are highly likely to be misleading.

Both Kuut (2023/ReCOVer) and Samper-Pardo (2023) were classed as exhibiting “High” risk of bias with regard to the way outcomes were measured. This means that both trials were found to have used misleading, and thus unreliable, outcome scores. Toussaint (2023) was found to have “High” risk of bias because they chose to exclude some of the outcome data. This means that their reported findings were not representative of actual treatment outcomes.

In summary, when you remove the biased studies for CBT….well, you have no studies left!

But again, when you *include* the biased studies, you can produce a finding…which will be just as biased.

* * *

In short, based on the authors own risk-of-bias analyses (which are buried away in an online supplement), the findings of the latest BMJ long Covid review certainly fail to impress.

In fact, when you properly consider risk of bias, we can see the following:

(a) There are plenty of studies: six on physical rehabilitation and three on “behavioural interventions” (CBT)

(b) Virtually all of the studies have “High” or “Probably High” risk of bias, meaning that their findings are very likely to be misleading and so should be considered unreliable

(c) The one unbiased study to look at physical outcomes after physical intervention compared two different types of physical intervention, and had no non-therapy control group. Therefore it cannot be used to recommend physical therapy over no therapy

(d) The only way to derive any “evidence” from these reviewed studies is if you deliberately choose to include studies that you know have “High” or “Probably High” risk of bias

It seems incredibly odd to me that the critical information here is stored in an online supplement that the majority of readers will never see, especially when room was found to include much less important information, in depth, in the actual paper.

But once you see the supplemental tables the state of the evidence becomes pretty clear.

The studies certainly do not support the authors’ conclusions that “Moderate certainty evidence suggests that CBT and physical and mental health rehabilitation probably improve symptoms of long covid.”

There is no certainty, moderate or otherwise. There is no evidence, per se. Any “suggestion” involved would be highly misleading because the studies have been shown to be biased. There is no “probably” about any of this. Their entire sentence is gobbledygook.

Personally, I prefer my evidence un-biased.

===

Addendum: I should have added that the online Supplements document, which runs to some 116 pages, also contained risk-of-bias assessments for one further relevant trial, McGregor (2024/REGAIN). The authors put this trial in a separate table on the basis that it examined both “physical and mental health rehabilitation”. Suffice to say, this trial too was revealed to have “High” risk of bias in multiple dimensions (namely, biased outcome measures and omitted data) as well as “Probably High” risk of bias relating to the way it deviated from its intended intervention.