Fact Check: COVID Vaccine Data Do NOT Show Increased Risk Of Serious Adverse Events Of Special Interest In Randomized Trials

Risk Inflated

Does a reanalysis of Pfizer and Moderna mRNA COVID-19 vaccine data show an increased risk of serious adverse events of special interest? No, that’s not true: The authors of this study created their own subset of serious adverse events of special interest (AESI), which give the appearance of there being a strong correlation between them and the vaccines when, in fact, there was very little. Also, both the Centers for Disease Control and Prevention (CDC) and the Food and Drug Administration (FDA) say the coronavirus shots are safe and effective.

The claim appeared in a YouTube video (archived here) published by John Campbell on December 30, 2022, under the title “Reanalysis of mRNA trial data.” The description opened:

Swine flu vaccine (1976), 1 serious event per 100,000 vaccinees, Vaccine withdrawn

Rotavirus vaccine Rotashield, (1999),1 to 2 serious events per 10,000 vaccinees, Vaccine withdrawn

Covid mRNA vaccines, 1 serious event per 800 vaccinees, Vaccine officially promoted

This is what the video looked like on YouTube at the time of writing:

(Source: YouTube video screenshot taken on Tue Jan 3 19:07:17 2023 UTC)

The swine flu and rotavirus numbers come from the CDC website on “Historical Vaccine Safety Concerns.” The numbers in the video are accurate but are not specific about the adverse event, as the CDC documented. For swine flu, the increased risk was for one additional case of Guillain-Barré Syndrome (GBS) following vaccination with a swine flu vaccine. For rotavirus, the CDC estimated that one or two additional cases of intussusception would be caused among every 10,000 infants vaccinated with the RotaShield® vaccine.

John Campbell is a popular YouTube host with more than 2.6 million subscribers to his channel. He is not a physician but a retired emergency department nurse. According to the “About” page on his YouTube channel, his Ph.D. “focused on the development of open learning resources for nurses nationally and internationally.” Since starting his channel in 2007, he has posted thousands of videos on a variety of medical topics but has focused primarily on COVID since the start of the pandemic, promoting unproven treatments like Ivermectin (see here, here, and here), spreading misleading death statistics minimizing the number of victims and misinterpreting data about vaccine safety

Campbell’s video details a peer-reviewed article (archived here) published September 22, 2022, in the online publication Vaccine, titled “Serious adverse events of special interest following mRNA COVID-19 vaccination in randomized trials in adults.”

The article looks at data that was included in the FDA clinical review memo, statistical safety review memo and other public documents used in the authorization of the Moderna and Pfizer (Pfizer partnered with BioNTech) vaccines. The “harm-benefit analyses” called for in the study have already been performed by the FDA. This type of analysis can’t provide new information about product safety beyond what is already documented.

Additionally, the original clinical trials by Pfizer and Moderna are now nearly 3 years old and COVID variants have changed the playing field. New trials would have to be conducted to determine if there’s a cause and effect between the vaccines and the serious adverse events of special interest, which are pre-specified potential side effects.

In its discussion section, the peer-reviewed version of the article comes to this conclusion:

The excess risk of serious adverse events found in our study points to the need for formal harm-benefit analyses, particularly those that are stratified according to risk of serious COVID-19 outcomes. These analyses will require public release of participant level datasets.

In Dr. David Gorski‘s September 5, 2022, article “Peer review fail: Vaccine publishes antivax propaganda disguised as ‘reanalyses’ of Pfizer and Moderna COVID-19 vaccine clinical trial data,” he challenged the conclusion the study’s authors came to, singling out Peter Doshi, the senior editor at the BMJ (published by the British Medical Association) and a faculty member at the University of Maryland School of Pharmacy:

If you really want to know how safe and effective the vaccines are now (and have been over time), the original clinical trials are far from the best dataset to use, and this ‘reanalysis’ failed to find even a signal that there might have been a severe design defect or fraud in the original clinical trials. I’m all for data transparency and have no objections to releasing patient level data, but Doshi knows that this is not a trivial thing to do, as all the data have to be carefully de-identified and curated.

In a January 3, 2023, email to Lead Stories, the CDC Media Office said the public health agency stands by its data, which it says are under constant review:

Statements that imply that reports of deaths following vaccination equate to deaths caused by vaccination are scientifically inaccurate, misleading, and irresponsible. COVID-19 vaccines are undergoing the most intense safety monitoring in U.S. history. To date, CDC has not detected any unusual or unexpected patterns for deaths following immunization that would indicate that COVID vaccines are causing or contributing to deaths. CDC recommends that everyone who is eligible should get vaccinated.

COVID vaccines are recommended for everyone ages 6 months and older, according to the CDC website.

The FDA also responded to a Lead Stories query on the video and the study it was based on. In a January 3, 2023, email, it provided this statement:

FDA disagrees with the conclusions in the paper from Doshi et.al. Based on the agency’s thorough evaluation of the safety and effectiveness data for the mRNA COVID-19 vaccines, as well as the ongoing safety surveillance of the vaccines, we continue to find their benefits far outweigh their risks in preventing COVID-19, including its most serious outcomes of hospitalization and death.

VAERS data

The ongoing safety surveillance and data on adverse events come from the Vaccine Adverse Event Reporting System (VAERS). It is co-managed by the FDA and the CDC as a crude early warning system and not as a database for the quantification of specific outcomes.

Anyone with internet access can add a report to the VAERS list of reports. The public access link to it expressly warns against unwarranted conclusions based on VAERS material because the list only provides a tally of unverified notes about any health event people experience after they are vaccinated.

The list itself cannot be used to prove or quantify, since all it shows is a chronological correlation, not the causal link that would be more difficult to establish. It’s the equivalent of a police precinct’s running “blotter” reports that may serve as a starting point for police work, not an endpoint.

Playing with numbers

Abram L. Wagner, a research assistant professor of epidemiology at the University of Michigan, provided this interpretation of the article for Lead Stories in a September 12, 2022, email for a different story that looked at the same study:

The [study] compares the number of adverse events in those vaccinated compared to those not vaccinated. Even taking the results at face value, there needs to be a cautious interpretation of them. The data they analyzed included almost 40,000 individuals enrolled in clinical trials, and yet their results include wide confidence (or ‘compatibility’) intervals. This speaks to the very low number of adverse events as well as uncertainty in their analysis.

I have the following issues with the study. For one, they do not have access to individual-level data, and so have to compute statistics based on aggregate data. This limits a comparison of their data with what vaccine companies previously published. So in this [article], one individual reporting multiple adverse events over time is counted as multiple events, whereas in previous reports, this would just be one case of an adverse event. Their analysis therefore could result in double counting of events if they derive from a single adverse reaction to a vaccine.

Additionally, they modified the adverse events included in the original Bright Collaboration list. They do include the original list as a sensitivity analysis, and this sensitivity analysis shows a much broader, and therefore higher level of uncertainty in their data.

The authors of the study effectively created their own made-to-order list of AESI. Adverse events can be either serious or non-serious but the study only looks at what it calls the serious ones. It’s a category defined by the authors of the study and wasn’t used by government agencies when putting vaccines through the authorization process.

Wagner’s email points to issues known in statistical analysis as p-hacking/selective reporting and data dredging/data fishing. Basically, it’s a way of manipulating a data pool. Doing so can unearth something that got lost in the broader context, but it can also be used to show results that really weren’t backed by the data.

Susan Oliver, a scientist who studies nanomedicine at The University of New South Wales in Australia, made a video called “Antivaxxers fooled by P-Hacking and apples to oranges comparisons.” It exposes the methods Doshi used to reach his conclusions. The video is posted below:

Additional fact checks related to COVID-19 vaccines can be found here.