3
$\begingroup$

In several (political) discussions, I have been told that there are around 200 papers stating that the Hydroxychloroquine is successful in treating COVID-19...


Is this statement true?

How can we estimate coldly and reasonably the absence or not of scientific consensus about Hydroxychloroquine regarding COVID-19?


I'm not able to judge: I am not a virologist myself. But I have a modest statistical background that made me have extreme doubts about the behaviour of Dr Raoult: Answers to my statistical question in April 2020 https://stats.stackexchange.com/q/459501/2910 highlight those doubts.

Looking back in time, there are things that does not help the seriousness of RAOULT statements: During January February and early March 2020;

  • in those times Dr. Raoult was mocking Coronavirus repeatedly , saying pandemics only exists in video games (I'm gonna a add a link to this one, it's just mind-blowing to see his carelessness)
  • Dr. Raoult was also joyfully repeating that Dr FAUCI was senile.

But:

  • there may be personal conflicts between FAUCI and RAOULT
  • there may be pharmaceutical conflicting interests. That already happened in the past in other economical fields... Yes that can exist sometimes, so that remains possible. In France, the Raoultmania and friends transforms this argument into: "Big Pharma" wants to bring down Dr. Raoult at all costs, he is the best virologist in the world, he is "the only one" real specialist.

So what to say to people saying that there are 200 scientific papers concluding that Hydroxychloroquine is effective: maybe there's a possibility to know how many papers are stating the contrary?

Any pointer is welcome. Even in my family it has sometimes become a political drama. I would like facts, or factual trends, or constructive conclusions based on clearly stated hypothesis.

I don't know where to start and it is important for me. And I clearly don't want to fall in the opinion-based trap.

$\endgroup$
20
  • 5
    $\begingroup$ I think Skeptics.SE cover questions of "how much support is there really for this in the scientific literature". $\endgroup$
    – A. B.
    Commented Jun 5, 2021 at 18:17
  • 2
    $\begingroup$ This question is off-topic in my opinion in that it is not a question about biology or even medicine (that would be "is hydroxychoroquinine effective against covid?") but about reaching decisions and arguing with others. Both of these are in the realms of sociology and psychology. In life we all make decisions on incomplete data. I trust the scientific and medical institutions in advanced Western societies (Britain, the US, W. Europe) and so if they haven't rejected this I trust there is a good reason for it. I don't need to read 200 papers to make my decisions in life. cont. $\endgroup$
    – David
    Commented Jun 5, 2021 at 20:43
  • 2
    $\begingroup$ cont. If you haven't come to the same conclusion of me I might have a low opinion of your decision making, but I need to ask myself why. What social influences have prevented you coming to what I consider the obvious conclusion. I am not going to argue with you because I realize it is a waste of time. Likewise, should you decide that hydroxychloroquine is snake oil, I don't reckon much for your chances of convincing your family unless they regard you as some sort of a guru on the subject. $\endgroup$
    – David
    Commented Jun 5, 2021 at 20:47
  • 3
    $\begingroup$ Skeptics.SE has been mentioned; note that there are already several questions there on this topic. This answer might be the most relevant. $\endgroup$
    – Ben Bolker
    Commented Jun 6, 2021 at 18:25
  • 2
    $\begingroup$ This site deals with specific problems in biology. One such problem related to your concern might be "Is Hydroxychloroquine an effective treatment for Covid 19?", although I would think it more appropriate for SE Medical Sciences. But your question is really about the philosophy of science or of life — How do we distinguish between what is true and what is false, as individuals, as scientists — and hence is not on topic here. And of course, human judgement involves a heuristic approach, which doesn't always produce the right answer. There is no perfect approach, but there are sane ones. $\endgroup$
    – David
    Commented Jun 22, 2021 at 21:13

1 Answer 1

11
$\begingroup$

The difficulty here is that these sorts of questions are hard to understand for non-scientists and the results difficult to interpret for non-scientists.

This will be long, and I will likely miss things while writing this, so anyone please feel free to weigh in and/or edit if you feel the need.

First some background about treatments and how we (scientists and medics) know they work or don't work:

In general for a drug to be assessed for its efficacy it needs to undergo something known as a clinical trial. These are studies that involve proper experimental design and have at least one testable hypothesis with results that can be interpreted statistically. Proper experimental design for these sorts of things is difficult and aims to ensure that any there are no biases in the data that may invalidate the results!

First off the trial should be controlled in some manner. Control in this context means a comparison group who receive something other than the treatment in question. This might be that the patients receive a different drug (of known efficacy or not) or a placebo. You can even compare different doses of a known efficacious drug (say you know it works a dose X, but you want to see if it works at 10-times that dose). You can also compare things like surgery with no surgery - sometimes by doing mock surgery as a placebo.

The treatment also has to be tested in a randomized manner (assigned to some participants on a random basis), you also need to make sure that neither the participants nor the people giving the drugs know which people are receiving the placebo and which the real drug(s) until after the results are in and have been fully analysed. This is a process known as blinding. There are quite a few different sorts of blinding. The one I described above is called double-blinding.

If you put all these things together you have what is known as a controlled randomized double-blinded study. These are often considered the most effective at determining the efficacy of treatments. Not all trials will meet these conditions - for example it would be very hard to double-blind a surgical trial as it would be hard to blind a surgeon as to whether they have done real surgery or not.

Next, clinical trials should also ideally involve lots of random participants from random populations so as to not introduce bias(es) (for example: don't sample just a few patients from your local hospital when you live in a high-wealth area and assume that your results are able to be generalized to the whole population). This means you should sample young, old, rich, poor, body types etc). There are a few exceptions to this rule - perhaps you want to examine the effect of a particular treatment on obesity (so you need obese people to start with) or perhaps a particular age-group, but that can be part of your experimental design so is less of a problem, but you still need to sample randomly within that group

However,in addition to all of the above, there are a few more conditions that should be met:

The next rule is that you need to sample large numbers of people - often in the several thousands. Smaller sample sizes may have larger (relative) biases (e.g. maybe, just by accident, 50% of the 60 patients in your trial were from a single family with a genetic mutation who all caught the bacterial infection you are studying at a family gathering - which could bias your results towards how that genetic mutation affects your treatment). Another issue is that small samples may not have enough statistical power to show minor effects. Proper experimental design will have determined the statistical power needed and how large the minimum sample size should be before the study is conducted. Large sample sizes also allow you to look at different groups within the study during analysis - perhaps your drug isn't effective in the population as a whole, but it is in those over age 70, or those with kidney disease or a missing left leg (very unlikely - but you wouldn't see this last unless you had a large enough sample size!).

A clinical trial should also be registered with a clinical trial register (e.g. The USA one) with open methodolgy and aims etc. outlined. As I write this, there are 5874 clinical trials on COVID-19 registered on the Clinicaltrials.gov website from all sorts of places around the world and looking at things from known treatments (e.g. remdesivir) to wearable smart devices.

Another requirement is that the results are reported in the literature (we are getting closer to your question here...). If they are a well designed study that meets all the above criteria and show a result of interest to the medical community, then it should have no problem with peer-review (where experts working in the field examine all aspects of the trial before the results get published) and will likely get published in a well established and respected scientific or medical journal.

Thanks to @BenBolker for the following idea and paper:

To show you just how hard all of this is, there's a review paper in the British Medical Journal (BMJ, one of the top-ranked medical journals) with the title Drug treatments for covid-19: living systematic review and network meta-analysis. Link to article: here. What this paper aims to do is to look at many of the drug treatments for SARS-CoV-2 infection and aggregate data from all the published trials from the start of the pandemic through to February 2021. The abstract provides a summary of the data starting out with inclusion:

196 trials enrolling 76 767 patients were included; 111 (56.6%) trials and 35 098 (45.72%) patients are new from the previous iteration; 113 (57.7%) trials evaluating treatments with at least 100 patients or 20 events met the threshold for inclusion in the analyses.

This means out of the ~200 studies, just over half met the criteria of significance (for this analysis) - they excluded vaccination, nutrition, antibody-based therapies, supportive therapy and traditional therapies that included more than 1 active molecule.

As further evidence of how hard it is: Figure 1 from the BMJ paper shows the triage process for the articles they looked at. It shows that they started out scanning 31,752 articles on COVID-19 and trimmed that down to 632 randomized trials, which were then examined closer and found that 343 of these weren't actually randomized or were excluded for other methodological problems, leaving 289 trials. These were then pruned to 206 for a bunch of reasons (duplicates, pre-print (so not verified), corrections, retractions etc.) leaving 196. A further 83 were excluded for not meeting criteria on size, lack of data, no outcomes reported or same-drug comparison (not controlled properly). Leaving 113 in total.

So if you take the 632 trials and are left with 113... that's only about 17%: less than 1 in 5 get it right!

Now, this is where it gets murky! What is a "good" journal? Well, that's a tale for another day, but there are many "scientific" journals that are neither scientific nor good. Many of them are listed on Beall's list, which is a list of publishers and journals known or suspected of being "pay for display" - essentially pay them money and they will publish your paper without doing the proper peer-review process, which science uses to determine if a study meets scientific standards and rejects those that don't.

Unfortunately, proper peer-review takes time - the journal gets a paper submitted to them. An editor reads and then rejects or sends out for peer-review. The reviewers then read the paper, write a report suggesting problems, fixes, questions etc - these then get communicated to the editor, who either rejects the paper or sends it back to the authors for amendments (may be needs more experiments, more statistics etc). The authors fix and then re-submit to the editor, who again reads and sees if the review has been addressed. The paper may then be sent out to the reviewers again, rejected (if concerns not addressed) or published. Generally this takes between 3 and 12 months.

With the current pandemic there has been a problem with this system of peer-review. Because of the urgency of the pandemic and the strain on resources, people have been submitting papers to journals for peer-review and posting them on "pre-print" servers (not peer-reviewed), where they are read and used, based on the information presented. Many journals also suspended peer-review because of the need to get information, especially about therapeutics, out to their audiences as quickly as possible. This resulted in some of the checks-and-balances that normally go on in the scientific field to be suspended and papers published that would not normally have passed peer-review.

We are now finally at the point where I can discuss Hydroxychloroquine (HCQ):

Hydroxychloroquine has been around for a long time and touted as a potential therapeutic for a number of diseases. It works well for rheumatoid arthritis, systemic lupus erythrematosis and as an anti-malaria drug. However, it has never been shown to be an antiviral in any form outside of the laboratory.

Initially reports of HCQ as a treatment for COVID came out in May 2020 (e.g. this British Medical Journal paper (an excellent journal). These were small trials which were some times randomized and generally non-blinded (AKA open-label), and showed variable effects. 6 out of 8 papers mentioned in the FDA revocation of emergency use for HCQ letter (see Table 1 on page 5. PDF!) show no effect, while 2 do, with the largest in terms of numbers of people (the BMJ one linked above) showing no effect. The ones that do show an effect are very small sample size

Based on the evidence presented in some of these studies and (I think, can't find publications) some laboratory data indicating that HCQ might be effective against SARS-CoV-2, the FDA issued an emergency use authorization (I also don't know if there was some political interference in this). At the same time the University of Oxford (UK) started recruiting for a large clinical trial of HCQ. This trial is known as the RECOVERY trial and is not just for HCQ, but a bunch of other drugs too. Currently they have recruited nearly 40,000 patients for their trials across multiple countries.

The results from the RECOVERY trial into HCQ were released in June 2020 (this is very very quick for a clinical trial, but then massive numbers of patients) with 11,000 patients tested in a randomized, controlled, double-blinded manner. The results were then published in New England Journal of Medicine (another excellent journal) and showed no effect of HCQ on any of the metrics recorded. Since then other trials, mostly smaller ones (~1000 patients) have been run by a bunch of people, none of which (to my knowledge) have found any effect of HCQ on SARS-CoV-2 infection.

Taken together, but heavily weighted towards the results of the RECOVERY trial, the data indicate that HCQ is not effective at all against SARS-CoV-2.

Edited to add: @Chris has commented with a link to a site that aims to look into the science behind medicine and some of the claims made around therapeutics and make it available for the layman. The link for hydroxychloroquine is here

$\endgroup$
7
  • 4
    $\begingroup$ You are right (as far as I followed this up): There is no proper trial (number of patients, controls etc.) which has found an effect of HCQ. Only the very early, methodically heavily flawed "studies" saw small effects. This should have been a massive warning sign on its own. $\endgroup$
    – Chris
    Commented Jun 6, 2021 at 12:21
  • 5
    $\begingroup$ Wish I could upvote this twice! It sounds as if the answer to the "200 papers" thing is that if there really were 200 papers in favour of HCQ, they were rubbish ones. All scientific papers are not created equal and, sadly, there are a lot of rubbish ones about - just because something is stated in a "published paper" does not by itself mean it's reliable. $\endgroup$
    – A. B.
    Commented Jun 6, 2021 at 13:31
  • 2
    $\begingroup$ bmj.com/content/370/bmj.m2980 ("Drug treatments for covid-19: living systematic review and network meta-analysis") is a very good reference. Someone who felt like it could build an answer based on that resource ... $\endgroup$
    – Ben Bolker
    Commented Jun 6, 2021 at 18:30
  • $\begingroup$ @BenBolker thanks. I've edited that in with some explanation about difficulty of doing trials correctly. $\endgroup$
    – bob1
    Commented Jun 6, 2021 at 22:21
  • 1
    $\begingroup$ To add to your answer: The blog "Science based medicine" has a number of articles on this topic: sciencebasedmedicine.org/… $\endgroup$
    – Chris
    Commented Jun 22, 2021 at 17:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .