Florida’s COVID-19 Vaccination Analysis Is Flawed, Experts Say
The state of Florida recently announced that it was no longer recommending that younger males receive mRNA COVID-19 vaccines, based on an unpublished analysis that purportedly found an increased risk of cardiac-related death following vaccination. But experts who specialize in the unique method used in the analysis say it was not properly done.
Even if it had been, the findings would not mean that individuals should not get vaccinated. The analysis includes no risk-to-benefit comparison, and by its own admission is “preliminary” and “should be interpreted with caution.”
On Oct. 7, the Florida Department of Health said in a press release that the state’s surgeon general, Dr. Joseph A. Ladapo, was issuing new guidance about COVID-19 vaccination after the department conducted an analysis that found “an 84% increase in the relative incidence of cardiac-related death among males 18-39 years old within 28 days following mRNA vaccination.”
“With a high level of global immunity to COVID-19, the benefit of vaccination is likely outweighed by this abnormally high risk of cardiac-related death among men in this age group,” the press release continued. “As such, the State Surgeon General recommends against males aged 18 to 39 from receiving mRNA COVID-19 vaccines.”
No other state has made such a recommendation, and it’s counter to guidance from the Centers for Disease Control and Prevention and other medical authorities. The mRNA vaccines, which include the Pfizer/BioNTech and Moderna vaccines, have been shown to be remarkably safe in clinical trials and in follow-up safety studies. There is a heightened risk of inflammation of the heart muscle or surrounding tissue, known as myocarditis or pericarditis, particularly in younger males after a second dose. The condition, however, is rare and most patients recover within a few months.
The same day of the press release, Ladapo, who is known for questioning the safety of COVID-19 vaccines and has promoted unproven COVID-19 treatments, shared the analysis on Twitter. The social media company initially removed his tweet for violating its misinformation policies, but restored it two days later.
Ladapo was appointed to his post by Republican Gov. Ron DeSantis in September 2021 after the previous surgeon general resigned.
During Ladapo’s tenure, the state of Florida has also recommended against vaccinating babies and children below the age of 5 and against vaccinating healthy children between the ages of 5 and 17. This advice is also at odds with that of the American Academy of Pediatrics, the CDC and numerous medical experts.
Florida’s Analysis
The Florida Department of Health’s analysis is a self-controlled case series, which is an unusual method that by design only includes people who experienced a possible side effect after vaccination — in this case, death.
Instead of comparing deaths among vaccinated or unvaccinated people, as a more standard study might do, the comparison in the Florida study is between the number of deaths during a defined “risk” period versus a control, or “baseline,” period, for each person.
“The key benefit of SCCS is that each individual serves as a self-control, so in principle this design adjusts for confounding that is typically a major issue when comparing separate ‘vaccinated’ and ‘unvaccinated’ groups,” University of Pennsylvania biostatistician Jeffrey S. Morris told us in an email, referring to other factors that could affect an outcome.
Self-controlled case series can be powerful tools, but they can also be tricky to do, particularly when dealing with a terminal event such as death — and as we’ll explain, experts in the technique said they found the department’s analysis to be flawed.
But first, a quick review of what the analysis, which is unpublished and does not even list any authors, claimed to find.
According to the methods section, the analysis included adults living in Florida who died within 25 weeks of a COVID-19 vaccination, but excluded those who received boosters or who had a documented coronavirus infection or a COVID-19-associated death.
It found that in the first 28 days after vaccination, there was no increase in all-cause deaths, relative to the remaining 25-week period — and for those 60 years and older, vaccination reduced the risk of death slightly. But there was a small, statistically significant increase in cardiac-related deaths for the entire group, which were defined by certain codes written on the death certificates.
Broken down by age, only two groups — 25- to 39-year-olds and those 60 and older — had statistically significant increases in cardiac-related deaths, with the strongest effect in the younger group. The statistically significant effects were limited to males. When just looking at the mRNA vaccines, there was an 84% increased risk of cardiac-related death among males 18 to 39, based on 20 deaths in the “risk” period and 52 in the “baseline” period.
A key problem, however, is that the authors did not account for multiple vaccine doses, which can bias the result, experts told us.
“The flaw in the calculation is that the timing of the death determines the observed exposure pattern, and that this is not taken into account in this method of analysis,” Paddy Farrington, a professor emeritus of statistics at the Open University in the U.K., told us in an email.
Farrington specializes in SCCS and is an author on three of the six papers the analysis cited, including a review paper covering “recommendations for best practice” using the method in vaccine safety studies.
He has helped conduct a couple SCCS studies of COVID-19 vaccine safety, which have not turned up any concerns for the mRNA vaccines.
To understand the issue in the Florida analysis, Farrington said to consider a hypothetical scenario in which there are two doses given four weeks apart and in which deaths occur at a constant rate of one per week, unrelated to vaccination. For a 29-week period, there would be 29 total deaths: four in the first four weeks after dose one, four in the first four weeks after dose two, and 21 in the 21 subsequent weeks.
But if one were to do the analysis that Florida’s health department did, comparing the rates of death in the four weeks after the last dose given, Farrington said, there would be eight deaths within four weeks of a vaccine dose. The apparent rate ratio, he said, would be 2, suggesting a doubling of the risk. But this would be an error.
“This is biased upwards: the true rate ratio is 1, since deaths arise at a constant rate of 1 per week irrespective of vaccination,” he said.
The correct way to do the analysis, Farrington said, is to use a specific methodology he published in January in the journal Statistics in Medicine, which solves these issues. “The method is difficult to explain,” he said, “but it takes into account all the doses given.”
The authors of the Florida analysis appear to have modeled their approach on an unpublished paper posted to a preprint server medRxiv in March, which they cited. The preprint, by scientists in the U.K., also evaluated the risk of all-cause and cardiac death following COVID-19 vaccination by doing an SCCS analysis on the last vaccine dose given.
Unlike the Florida analysis, however, it did not identify any increased risk of death after vaccination. It also performed a similar analysis after a positive COVID-19 test — which the Florida analysis did not do — and found a large increase in the frequency of both all-cause and cardiac-related deaths after infection in people who were unvaccinated.
The lead author of the preprint, Vahé Nafilyan, a principal statistician for the U.K.’s Office for National Statistics, told us in an email that the preprint was “conducted very quickly to respond to rising concerns about the safety of vaccine in young people,” and his group had rerun the analysis using the methodology Farrington recommended. The new paper is currently under review, he said, and the preprint is being released this week.
But Nafilyan also said his team had a work-around to avoid the concern Farrington had with analyzing the last vaccine doses.
“In the initial analysis, we purposely restricted the follow-up period to 12 weeks – the minimum separation between doses,” he said in an email, referring to the longer interval used in the U.K. when the vaccines were first rolled out. “This was a quick way to circumvent the issue. The Florida study does not apply such a restriction and that is the issue.”
“For the Florida study, it would be far more tricky since the minimum gap between doses in the US was 3 weeks,” he added, “making it impossible to use the same ‘trick’ as in our initial analysis.”
Farrington agreed that this difference between the studies would mean the U.K. paper “is probably OK, whereas the Florida one is not!”
Additional Concerns
Other critics have raised additional concerns and caveats about the Florida analysis. For example, cardiac deaths were not checked by reviewing medical records, but instead were defined by the presence of certain codes on death certificates, which aren’t a guarantee that a person died from a cardiac problem.
As Dr. Kristen Panthagani, an emergency medicine resident at Yale New Haven Hospital, wrote in a blog post, the codes included cardiac arrest, “which simply means ‘the heart stopped’ and can be the terminal event for many different diseases, not just cardiac issues.”
The authors say as much in the limitations section, noting the study “cannot determine the causative nature of a participant’s death” and the “underlying cause of death may not be cardiac-related.”
The limitations section, notably, also mentions that the increased risk in cardiac deaths the analysis observed in the overall population and the 60 and over group could be due to confounding by age in the older group. Because so many more deaths occurred in older people, when the 60-plus group was removed, there was no longer any statistically significant result for cardiac-related deaths after vaccination, for mRNA vaccines, or males who received mRNA vaccines.
Panthagani and others have said that given the relatively small number of deaths in the “risk” period for younger males receiving mRNA vaccines — 20 — a change of just a few because of inaccurate coding could render the finding insignificant.
Another issue is how exactly the authors dealt with COVID-19. People with COVID-19 listed as a cause or contributing factor to death were correctly excluded from the analysis, but it’s not entirely clear if the same was done for people who tested positive for COVID-19. This could mean that some of the cardiac deaths being attributed to the vaccine are actually from the disease, Panthagani noted.
In one part of the write-up, the analysis says people with positive COVID-19 tests were excluded, but in another, “COVID testing status was unknown for those who did not die of/with COVID.”
On Oct. 10, Ladapo responded to some of the criticisms on Twitter, arguing that it didn’t matter if the diagnosis codes for cardiac deaths were “imperfect,” since they were the same for all groups. He also said the state “used all of our data resources,” including test results, to exclude people with documented coronavirus infection.
But if that’s the case, then what is written in the limitations section is incorrect. We reached out to the Florida Department of Health to clarify this issue, among others, but did not hear back.
Morris, the University of Pennsylvania biostatistician, told us that the analysis might be fine “if properly executed,” but he couldn’t verify if that was the case given the lack of detail in the write-up — and he had “several unresolved questions that make me wonder.”
Along with the confusion over the COVID-19 testing and “potential bias from considering the last dose only,” Morris said that he did not understand why some of the average follow-up times in some cases were longer than the 25-week period set for the entire analysis. He said he had asked for more information from the state health department on this and other issues, but had not received a reply.
Morris and Farrington also wondered why the analysis excluded booster doses. And critically, despite a mention of “infection” in the title and in another sentence of the discussion — which could be a typo — Morris did not understand why the analysis did not also evaluate the risk of cardiac deaths following coronavirus infection. Not only is this what the U.K. study did, but this would be “a relevant factor to weight against any vaccine-related risks.”
Analysis Poor Basis for Changing Vaccine Recommendations
Indeed, several critics have pointed out that even if the analysis is taken at face value, it doesn’t include a comparison of the risks of the vaccine with its benefits, so the analysis itself fails to show that the risks outweigh the benefits. And as a single unpublished, preliminary analysis that contradicts other published papers, it’s hardly enough to justify modifying public health guidance.
“There is a large literature on the safety of mRNA vaccines, and recommendations should be based on a comprehensive overview of the available evidence, not any one single study (even less so if it is unpublished),” Farrington said. “What I find extraordinary in the Florida saga,” he added, “is that recommendations on vaccination can be made based on a single (in this case, flawed) study, when there is ample other evidence to the contrary.”
As we said, myocarditis and pericarditis have been identified as rare side effects of the mRNA vaccines. Studies have consistently shown that for the overall population, the risk of myocarditis is much greater after a coronavirus infection than after vaccination. For some groups, such as men under 40, the risk of myocarditis may be higher after vaccination than after an infection, as a large U.K. study recently found for the second dose of the Moderna vaccine. But considering all the other possible bad outcomes from COVID-19, numerous teams of experts have concluded that the benefits of the vaccine outweigh the risks, even for younger males.
“Florida’s public health recommendation is specifically for men age 18-39. Despite the fact that this age group is at lower risk of severe complications of COVID-19 than older Americans, CDC, FDA, and other federal agencies continuously have found that the evidence is clear: the benefits of vaccines clearly outweigh any risks,” Sarah Lovenheim, assistant secretary for public affairs at the department of Health & Human Services, said in a statement. “This is why FDA authorized, and CDC recommended, that all individuals in this age group get vaccinated against COVID-19.”
Lovenheim called Florida’s decision “flawed and a far cry from the science.”
According to an expert consensus paper published in March by the American College of Cardiology, “a very favorable benefit-to-risk ratio exists with the COVID-19 vaccine for all age and sex groups evaluated thus far.”
While it’s conceivable that some deaths have occurred because of the vaccines, this risk is exceedingly small. During a CDC vaccine advisory committee meeting in February, the agency presented data showing that 13 deaths involving myocarditis had been reported among people 30 years and younger who had received a first or second mRNA dose. However, none of these was thought to be vaccine-related. We asked the CDC for an update on the numbers, but didn’t receive a response.
Studies suggest that post-vaccination myocarditis is milder than typical viral myocarditis and that most people recover within three months.
In interviews with Politico and the Washington Post, Ladapo defended the analysis and his decision to stop recommending the mRNA vaccines for young males, given high levels of immunity in the population now.
Ladapo told the Post that he hoped his former mentor, Harvard economist David Cutler, would endorse the methods used in the analysis. But according to the Post, Cutler said the analysis was flawed and should not serve as the basis for state vaccine policy.
“If I was a reviewer at a journal, I would recommend rejecting it,” Cutler said.
Editor’s note: SciCheck’s COVID-19/Vaccination Project is made possible by a grant from the Robert Wood Johnson Foundation. The foundation has no control over FactCheck.org’s editorial decisions, and the views expressed in our articles do not necessarily reflect the views of the foundation. The goal of the project is to increase exposure to accurate information about COVID-19 and vaccines, while decreasing the impact of misinformation.
This article has been archived for your research. The original version from FactCheck.org can be found here.
