Study comparing side effects of Covid-19 and vaccines nothing to do with WHO

An article published by the Exposé blog claims that a “World Health Organization [WHO] Study concludes risk of suffering Serious Injury due to COVID Vaccination is 339% higher than risk of being hospitalised with COVID-19” [sic]. 

The description of the paper as a “World Health Organization Study” is inaccurate, according to both the WHO and an author of the paper. A number of experts have also told Full Fact they have questions about its findings.

Study isn’t endorsed by the WHO

Beyond the headline describing the study as a WHO study, the Exposé article goes on to describe it as being “endorsed” by the organisation multiple times. 

Its rationale for doing so appears to be the fact that the study uses an adapted version of a list of what it calls “serious adverse events [SAEs]” compiled by a group of experts on vaccine safety called the Brighton Collaboration. The original version of this list has been endorsed by the WHO. 

This doesn’t mean however that the WHO had any involvement with the study itself. When asked if it had any connection, a WHO spokesperson said: “WHO has no link with this.” 

The study itself received no funding from the WHO or any other body, and only mentions the organisation twice. Dr Peter Doshi, one of the study’s authors, told Full Fact: “We share concerns about erroneous claims made about our article, such as that it was a World Health Organisation study.”

What did the study actually look at? 

The study claims to have found that the Pfizer and Moderna Covid-19 vaccines, which are both mRNA vaccines, were associated with an increased risk of certain serious side effects (described as “serious adverse events of special interest” by researchers). 

The study compared this to the risk of hospitalisation with Covid-19, and claimed that the risk of side effects from the vaccines outweighs the risk of hospitalisation for Covid-19.

To do this, researchers analysed the 2020 trial data from the Pfizer and Moderna vaccine trials and compared the number of serious adverse events (SAEs) in the group who received the vaccine against the number of people who suffered severe Covid-19 in the control group. 

Several experts in statistics have questioned the findings of the paper—pointing out potential issues with the period of time the study covers, the way in which adverse effects of the vaccine were counted and what actually counted as a serious adverse effect. 

The study hasn’t been peer-reviewed yet, and Dr Doshi himself told Full Fact that any claims the “results are definitive in any way” are “erroneous”.

The abstract of the article has been viewed more than 500,000 times, and the full paper more than 100,000 times. 

We passed all of the criticisms of the paper to Dr Doshi. You can read his full response, from which we have quoted throughout this article. We’ve explored two of the key questions in detail below. 

The study only compared harms over a short period of time 

One of the main criticisms of the paper is that it is based on data from the phase three randomised control trials for the Covid-19 vaccines, which were cut off in November 2020. This meant that in both the Pfizer and Moderna trials, participants were monitored for a median of two months. 

This data was the same as the data used by the US Food and Drug Administration when they approved the use of the Pfizer and Moderna vaccines under emergency use authorisation (EUA) licensing. 

But adverse effects from vaccines usually appear within the first six weeks, while real-world data shows the protection against hospitalisation from two doses of the Pfizer and Moderna vaccines lasts significantly longer than this. This means that the analysis in the study is likely to capture the majority of specified adverse effects from the vaccine, but will miss much of the protection from the vaccines against severe Covid-19, which will last much longer than two months. 

Professor Sir David Spiegelhalter, chair of the Winton Centre for Risk and Evidence Communication at the University of Cambridge, told Full Fact: “The ‘benefit’ of the vaccines used in this study only considers Covid hospitalisation during the trials themselves, which covered only around two months at a time of low Covid.  

“The true benefit of vaccines extend far beyond this period, so the harm/benefit comparison used in this study seems entirely inappropriate—the trials were not designed to estimate the reduction in severe Covid.”

The fact that the vaccines were trialled during a period of low Covid-19 prevalence is important, when assessing whether vaccines may result in a higher increased risk from SAEs than they reduce in the form of fewer Covid-19 hospitalisations. As the blog Science-Based Medicine says: “In these two trials, there were 74,000 participants, 36,930 of them received a vaccine, while only 366 of them had Covid. The vaccine had many more opportunities than the virus to make people feel rotten during the RCT [randomised controlled trial].” 

Professor Kevin McConway, Emeritus Professor of Applied Statistics at the Open University, described the study’s comparison as “apples and pears”, telling Full Fact: “It makes no sense to me to compare the rate of serious adverse effects from the vaccines, which will generally make themselves known in a short time after vaccination, with hospitalisation rates, where the vaccines can reduce the risk for a much longer period of time.”

Dr Doshi’s response to these criticisms included an acknowledgement that the trial data analysed by the study authors ended when the EUA (Emergency Use Authorisation) was given, as after this point the participants were unblinded—meaning they found out if they were given the real vaccine or a placebo, and the placebo group was offered the vaccine. Dr Doshi told us this rendered “the post-EUA dataset less reliable due to breaking of randomization”. 

He added: “There are hypothetical reasons such as those given by the experts you spoke with […] that could alter the harm-benefit balance as they apply to populations and time periods different from those studied in the trial. But it is critical to realise that the hypotheticals do not all point in one direction. Some hypotheticals improve the anticipated harm-benefit, while others worsen it. A key message from our paper is that it is unlikely there is a single harm-benefit ratio for all populations at all time periods.” 

Paper counts adverse events—not how many people had them

The paper has also been criticised for equating the number of SAEs reported following the vaccine with the number of people who suffered an SAE—because some people who experienced SAEs may have reported more than one. 

Dr Adam Jacobs, senior director of biostatistics at Premier Research, told Full Fact: “I’m not sure why the authors chose to do the analyses they did, such as… analysing the number of events rather than the number of patients with events.  

“It is far more conventional in adverse event analysis to analyse the number of patients with events, as it is not always possible to know whether multiple events in the same patient are really separate events or just a single pathological process with repeated flare-ups.”

For example, one person could report both abdominal pain and diarrhoea after receiving a dose of the Covid-19 vaccine, but be counted in the paper twice.

By comparison, someone who had experienced a Covid-19 hospitalisation would only have been counted once, even if they too had experienced a number of symptoms and diagnoses.

Dr Doshi acknowledged this, saying “there is merit to both approaches”, namely looking at the number of SAEs and the number of people experiencing any SAE. He added: “We could not complete the analysis of the number of participants with any SAE because individual participant level datasets are not available.” 

“If we gain access to these data, we will run both analyses. But the present lack of individual patient data forced us to perform the analysis only on the number of serious adverse events of special interest (AESI), rather than on participants.”

He clarified that the paper didn’t double-count repeated flare ups of the same SAE “as this would have been reported as a single event”. He added: “It is plausible that individuals susceptible to one vaccine induced SAE may be susceptible to multiple SAEs. We did identify a signal that this may—we stress may—be occurring with the mRNA vaccines; as stated in the paper ‘approximately twice as many individuals in the vaccine group experienced multiple SAEs than the placebo group’.”

Full Fact contacted the Exposé for comment but had not received a response at the time of publication.

Photo courtesy of Braňo