A: No. This metric sounds useful, but it can’t tell us anything about vaccine effectiveness.
Since the roll-out of COVID vaccinations, many headlines have reported the percent of COVID hospitalizations and deaths who were vaccinated, trying to convey how well the vaccines are working.
‘Only 1% of COVID hospitalizations are vaccinated people’ sounds encouraging, doesn’t it? Likewise, doesn’t ‘50% of COVID hospitalizations were fully vaccinated’ make it sound like the vaccines aren’t working well?
In reality, if you’re trying to figure out how effective the COVID vaccines are, these particular statistics are useless. It is impossible to use this statistic alone to figure out how well the vaccines are working. By itself, this number provides incomplete information and is highly misleading. Here’s the Nerdy reason why….
💥The Base Rate Fallacy💥
This statistic confuses people because it invites them to commit a type of logical fallacy called the “base rate fallacy”. Logical fallacies are errors in reasoning that occur when our brains try to oversimplify things to make them easier to understand, but that oversimplification causes us to ignore key details, which often leads to inaccurate conclusions. The base rate fallacy is a type of logical fallacy that occurs when people try to estimate the chance of something happening based only on specific examples in front of them, ignoring the background levels of those events in the population.
➡️ An example: suppose two COVID patients were admitted to the hospital, one vaccinated and one not. The headline might (accurately) report that ‘50% of hospitalized COVID patients are vaccinated!’ Mathematically, this statement is true. And it sounds high, right? If we only focus on this percentage, we might conclude that vaccination makes no difference in the rate of hospitalization for COVID.
But what if I told you this hospital is located in a town of 100 people, where 98 people are vaccinated and only 2 are not. This dramatically changes our interpretation of what’s happening in the hospital — now we know only *1 in 98* vaccinated people are hospitalized for COVID, while *1 in 2* unvaccinated people are hospitalized. Rather than the % of the hospitalized who are vaccinated, we care most about rate of hospitalization in the vaccinated vs. the unvaccinated.
Clearly, the risk of hospitalization for vaccinated people is much lower than it is for unvaccinated people. But our headline provided only partial information, so it missed this conclusion entirely. Without the background rates of vaccination, it’s impossible to interpret the statistic provided in the headline. We simply don’t have enough information.
🔢 If this were a math test and we were asked to figure out how well the vaccines are working, but the only information we were given was ‘50% of hospitalized COVID patients are vaccinated,’ we couldn’t answer the test question. We’d have to walk up to the teacher and tell them that more information is needed.
The ‘headline statistic’ (the percent of COVID hospitalizations who were vaccinated) is highly dependent on the vaccination rate in the community. Below is an animated visual that helps make this counterintuitive concept a little clearer. ⬇️
As the vaccination rate increases in population, the percent of hospitalized patients who are vaccinated also increases, even when there’s NO CHANGE in how well the vaccines are working. As more people are vaccinated (and all else is equal), total hospitalizations will decrease but the *percent* of vaccinated hospitalizations will increase.
💥To make this point really obvious, look at the scenario where 100% of the population is vaccinated (the very end of the visualization) — there are way fewer hospitalizations overall, but 100% of those few hospitalizations are vaccinated people.
❓What about age?
To add to the confusion, we have to remember that both the risk of being hospitalized for COVID and the chances that someone is vaccinated for COVID are linked to another important factor: age.
On average, older people are more likely to be hospitalized for COVID compared to younger people. And due to a combination of factors (availability of vaccines, personal assessment of risk, etc.), in most populations, older people are also more likely to be vaccinated for COVID than younger people. Because of this, we also need to look at the vaccination rates of people of similar ages to compare like with like. (The same concept applies for other demographic variables like sex, socioeconomic status, etc.)
Last summer when Delta was surging, a table from a Public Health England report on COVID in the UK was circulated widely online: the data was being used to argue that those who were vaccinated were at higher risk of dying from COVID compared to the unvaccinated. What was going on here?
It turns out those spreading this inaccurate rumor had committed the base rate fallacy. If you look only at the data in the table that doesn’t account for age, then yes, a higher percent of those vaccinated died compared to those who were unvaccinated. But their conclusion was inaccurate because the data in the table was highly skewed by age. Younger people in this data set were far less likely to be vaccinated than the elderly, and younger people are also at lower risk of dying from COVID (or dying from anything) compared to older people. Once the data was broken down by age group (which was provided in the same table), those who were vaccinated clearly had a *lower* risk of death compared to those who were not vaccinated of similar age.
When we’re studying how diseases impact populations, taking into account background rates in the population and controlling for “confounding” variables such as age is essential. This is Epidemiology 101.
➡️ So the next time you see one of those headlines reporting “X percent of COVID hospitalizations are vaccinated!” your first thought should be: I need more information, please.
🙏🏽Thanks to Nerdy Guest Dr. Kristen Panthagani, MD PhD for this awesome explainer AND sensational data visualization.
You can read more about Dr. Panthangani on FB You Can Know Things and her website.
Links:
This post was in part inspired by these twitter threads:
More details on data in Public Health England Report
COVID vaccination rates by age in US
Why are 40% of COVID hospitalizations in the UK among the vaccinated?
