Skip to content

Skeptics vs. Quacks: Who’s winning?!/bengoldacre/status/60028892009861120

Last week, the British market research firm YouGov published findings from a national UK opinion survey on complementary and alternative medicine (CAM). While the survey focused on CAM in general, YouGov chose to focus on the findings for homeopathy in their publicity materials. In summary, 43% of the sample believed homeopathy to be either “definitely” or “possibly” effective. Given that 20% reported not having a particular view (i.e., “Don’t know”), this meant that only 37% explicitly reported skepticism. In other words, of those adults willing to express an opinion, the majority expressed some level of belief in the effectiveness of homeopathy. This is despite the well established biological implausibility and medical inefficacy of homeopathic treatment.

Similar results emerged for other CAM treatments, suggesting that belief in pseudoscientific medicine continues to prosper in the UK. Noted science writer, blogger, and top skeptic Ben Goldacre even posted a tweet implying that the figures undermined claims that science advocates were “winning” the war on quackery (a point he elaborated in a subsequent blog post). However, should we really be that pessimistic? Because despite these initial impressions, the details within the figures may actually contain some encouraging signs.

Based on a sample of 2,530 men and women, the main survey was conducted in early April 2011. Helpfully, YouGov have posted the overall summary data on their website, and it is certainly worthy of detailed attention. As well as breaking down responses by various categories (including gender, social class, and region), YouGov supply benchmark data from an equivalent sample surveyed last August. Now while our attention can often be grabbed by a once-off headline statistic, usually more reasoned inferences are drawn from social sciences data when trends over time are examined. This is because the context of a once-off statistic can be difficult to interpret, and — presumably — such headline statistics will contain some error. If strong patterns can be found in changes over time, then these errors should cancel each other out, allowing us to draw firm(-ish) conclusions about the general trajectory of shifting events.

In this case, comparing the August 2010 and April 2011 groups should enable some tentative conclusions about whether recent shifts in attitudes encourage optimism or pessimism about the war on pseudoscience. Let’s start by looking at the homeopathy data with this in mind. According to YouGov, rates of agreement with the various statements on homeopathy were as follows:

While YouGov present the data as percentages, it is a simple matter to multiply up the figures to calculate the actual numbers of people in each category. The resulting figures are approximate (note the similarities across columns, which relate to the rounding effects of converting back from percentages), but they give us a better intuitive impression about the dimensions of relative distributions in attitudes across the groups.

The first thing to notice is that positive endorsements of homeopathy (i.e., the numbers in the first two rows) remained static from August 2010 to April 2011. However, the remaining figures suggest that some 150 persons changed their minds from “Don’t know” to either “Is probably not effective” or “Is definitely not effective. This is a pretty sizeable shift, accounting for more than 1 in 5 of last year’s “Don’t know” category. Putting all this together suggests that more people are negative about homeopathy now compared to last year, while the number of people with supportive attitudes has stayed the same. On balance, this looks like positive progress towards winning the war on quackery.

We can actually go a little further with this data and establish whether the changes observed are statistically significant. To do this, it would be helpful to collapse the categories a little, as follows:

We can ignore the “Definitely/possibly effective” category, as it doesn’t change at all from 2010 to 2011. This leaves us with an increase of 152 in the category “Probably/definitely not effective”, and a corresponding decrease in the “Don’t know” category. Statistically, this type of data can be significance-tested pretty easily using a procedure known as the chi-squared test, which shows us that the swing in attitudes  is highly statistically significant. In fact, the relevant p-value comes in at less than 0.0001. This means that, if (hypothetically) the two surveys were conducted again in the future, the chances of failing to show a similar swing would be less than 1 in 10,000.

Using the same logic produces equivalent conclusions about the other CAM therapies covered by the survey. Exactly the same kind of attitude swing is shown for acupuncture, reflexology, traditional Chinese medicine, herbal medicine, chiropractic, and Reiki (where each p is again less than 0.0001), as well as for crystal therapy (where p was 0.0004, which is still highly significant). Actually, for two of these therapies — reflexology and Reiki — the increase in skeptical attitudes was partly drawn from marginal reductions in supportive attitudes. Only one therapy in the survey — osteopathy — bucked the overall trend. In this case, a small reduction in “Don’t know” responses was accounted for by increases in supportive attitudes.

Now, chi-squared tests are definitely simplistic (they are quick-and-dirty and offer descriptions rather than explanations), and so there are plenty of caveats here. For one thing, while strongly significant in the statistical sense, the sizes of the effects were generally very small, especially when judged against the large subset of the population who remain open to claims of CAM efficacy. However, while this might be a limitation in a once-off chi-squared test, there is something uncanny and compelling about these particular findings. This is the fact that, the exception of osteopathy, all of the swings in attitudes were in the same direction. In other words, not only was a swing toward skepticism seen for homeopathy, but the swing toward skepticism was corroborated seven times by equivalent trends for the other therapies. This is strongly suggestive of a robust rise in explicit CAM skepticism, because it reduces the likelihood that the emergence of skepticism was linked to specific cultural events (such as news stories about homeopathy around the time that the survey was conducted).

Of course it has to be mentioned that these inferences about swings in public opinion are based a single shift across two time-points, which, as pictures into attitudinal change go, has pretty low granularity. Maybe attitudes toward CAM are somehow seasonal (with skepticism always elevated in April), or maybe in April 2011 people were particularly skeptical towards most things (because of contemporaneous political events). One way or another, maybe the next survey will simply show a reversal. However, the YouGov data does allow us to consider an alternative version of the possible long-term picture. This is because relatively complex age-breakdowns are presented for attitudes toward each CAM method included in the April 2011 survey. Here are the age breakdowns for homeopathy (excluding “Don’t know”):

These figures are interesting because, at a glance, you may conclude that skepticism is more common in younger adults than in older ones (e.g., while 57% of the oldest group believe in homeopathy, only 44% the youngest group do). In fact, based on an elaboration of the chi-squared test (namely, a statistic known as Cramér’s V), it is possible both to quantify the degree of association that exists between age and attitude and to establish its statistical significance. In this case, the association is small (V = 0.081) but highly significant (p = 0.004). In other words, it really is true that skepticism is inversely associated with age. Once again this pattern is broadly borne out across the other therapies surveyed. In general, skepticism toward CAM is higher in younger adults, suggesting that these more skeptical attitudes will become increasingly prevalent as current the population ages.

In summary, the YouGov data certainly shows us that many adults are willing to endorse statements that specific CAM therapies could be effective. However, the numbers endorsing the opposite view —  that such therapies are probably or definitely not effective — has increased significantly. The pattern among people who had hedged their bets in the past is especially encouraging. On average, one-in-five of the previously undecided now believe that CAM is ineffective. Virtually none have come to the alternative conclusion. And this overall trend toward skepticism is bolstered by the fact that CAM skepticism is much more common in young adults than in older age-groups.

The data show that positive progress is being made, especially in winning over the hearts and minds of people who perhaps had little prior knowledge regarding the efficacy of CAM practices. In all probability, it is these floating voters who will end up tipping the balance. The war on quackery is far from won. But it is vital that skeptics and other science advocates — especially influential and talented communicators like Ben Goldacre — keep fighting to the end. The data suggest that it will be worth the effort.

11 thoughts on “Skeptics vs. Quacks: Who’s winning? Leave a comment

  1. The problem is that we’re not winning over those just inside the belief margins. True Believers are not amenable to evidence, but those who kind of think there must be something in it are certainly reachable and it is these, not the uncertain ones (who are therefore probably at least somewhat skeptically inclined) who are important to real change.

    We need to be a lot more forceful about drawing distinctions between vitalistic bollocks like homeopathy and things which are merely misguided, like naturopathy. There is nothing inherently wrong with the idea that plants can heal, the stupidity there is in believing that an unpredictable dose with unknown impurities can somehow heal /better/ than the same active principle refined purified and measured, because it’s “more natural”. Bubonic plague is natural, living beyond the age of reproductive potency sure as hell is not.

  2. It seems to me problematic to conclude that overall skepticism will increase as the population ages. Might it not be the case that the process of getting older makes people more likely to accept the sort of beliefs that support homeopathy, for instance because they are less inclined to feel comfortable holding a minority view? More research would be needed to clarify this.

  3. I don’t like the wording of the question. I might consider that confidently prescribed placebo could “possibly” be an effective treatment. A nice talk even without a sugar pill might also be an effective treatment.

  4. @Jennie: Could the cause be more to do with the younger generations coming through being more scientifically literate? I think that that skepticism has made its mark in popular culture, with the success of ‘new atheist’ authors and famous skeptics like Richard Dawkins. Because of these authors, and many, many others, skepticism has a presence in mainstream culture that I feel it lacked before.

Leave a Reply to Alan HennessCancel Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: