Flawed Flu Vaccine Study Sparks Misinformation Storm

In a video originally published on Inside Medicine, MedPage Today editor-in-chief Jeremy Faust takes aim at a widely circulated Cleveland Clinic preprint on flu vaccine effectiveness — one that was touted on Fox News as proof that this year’s flu shot made people more likely to get sick. Faust explains why the study is methodologically unsound, highlights overlooked biases like susceptible depletion, and makes the case for better public understanding of basic epidemiology.

The following is a partial transcript of his remarks:

Hey everybody. Jeremy Faust here with Inside Medicine, and people have been talking a lot about this really interesting preprint, which is being misunderstood to be good science. It’s not good science, but it’s really a good opportunity to talk about how mistakes are made in science.

It’s a big paper about influenza vaccines out of an ostensibly good place — Cleveland Clinic — but the authors forgot how to do science and we don’t know what the results are, but they should also say that they didn’t know because they ought to know better.

So what I wanted to do is just to talk you through something, just go through the science by talking about what it is actually that they did. Let me actually just show you a little bit of this slide deck actually, and this is the Fox News headline: “[Flu] Vaccine Linked to Higher Infections.” That’s scary, “but doctors note limitations.” It’s not even limitations. This is not even the beginning of the answers, and it’s just a good teaching point. So let’s do it.

Of course, people who are anti-vax, love to jump all over this. It’s a “stunning Cleveland Clinic … negative effectiveness study.” What’s stunning about it is how bad it is. So if you don’t know that, you shouldn’t be in this space. Go back to writing fiction.

OK, here’s the preprint. I’m interested. I actually think that there could be a world in which vaccines are overused. So I was looking into this. Is it? Here’s the real figure that everyone’s freaked out about. And it shows vaccinated versus unvaccinated cumulative infection rate with flu, with the blue line being the vaccinated cohort and the red line being the not vaccinated. And by the end of the flu season this year, there was more flu illnesses in the vaccine group.

Now, a lot of people have talked about some things that could explain why this is, but I’m going to talk about why it’s wrong in one big way, and then I’ll mention two other ways at the end. First is — and this is the one that I think is not getting discussed — is something called susceptibility or susceptible depletion bias. Susceptible depletion bias, susceptible. Are you susceptible? And it’s such a big important thing that it’s just not discussed in this paper, and actually it’s not discussed in a lot of vaccine papers and it’s a real big problem. OK? Here’s how it goes.

First of all, you need to know something, and that is if you get vaccinated against flu, your protection probably lasts for infection somewhere between 3, 6, 9, 12 months, certainly wanes quickly, but by next year, your infection protection is probably gone. Now, there might be some protection against severe disease, but that’s a different story.

But if you’re vaccinated, that protection is really short-lived for infection, whereas we think that people who get the flu actually have protection against flu or at least related strains for longer, could be several years. And so what happens here — this is what you have to understand is — if you get infected, we’re going to assume for this model that you cannot get reinfected for a few seasons. And that is where these folks fail to think.

OK, so look, let’s look at this. Let’s take a cohort of a hundred vaccinated people in blue and a hundred unvaccinated people in red, and we’re going to imagine a flu virus. It is super contagious, and in which the vaccine group, 20% of the cohort got infected in the season, in season one, and 90% of the unvaccinated people in the unvaccinated group got infected. That’s not real, but I’m using these numbers to make a point about this kind of bias that the studies have to take into account.

And vaccine effectiveness in this season would be really clear. You would just take one minus the rate of infection in one group divided by the rate of infection in the other, and get a vaccine effectiveness of 78%. Everyone’s happy the flu vaccine worked. OK, great.

Here’s the problem. Next season, season two. Well, let’s say they detected 16 people in the vaccinated group. That’s just what they found. So 16 out of a hundred people in season two got infected, and they also noticed that just nine people in the unvaccinated group got infected that year. OK, that’s just what they found. They haven’t thought about susceptibles or not. That’s what they measured out of these hundred people.

OK, the problem is, if you don’t think, that you get a VE, vaccine effectiveness, that’s negative because you have more infections out of a hundred in the vaccinated group than the unvaccinated group. But that’s because they have not remembered that if you got flu, which 90% did in the unvaccinated group, you’re not going to get it again for a few years. You’re protected. Now, that protection came at a cost. Some of these people got really sick, you don’t want to do that.

What happens when we do season two with thinking? That is, taking into account the depletion of a number of people who are susceptible. Well, now we have 80 people left in the vaccinated group, right? Because last year, 20 got infected. That means only 80 could really get it this year. So 16 out of 80 is 20%, right? So again, 20% got infected here. In the unvaccinated group, we know there’s only 10 left at risk in that group, 90% again, right?

So now if you do the vaccine effectiveness on season two with thinking, with taking into account the depletion of the susceptible population, you’re back to the same number. In other words, among people who remain equally susceptible, the vaccine is still 78% just like it was before. And that is so important.

For more, please watch the video above.

Please enable JavaScript to view the comments powered by Disqus.