The Differences Between the Vaccines Matter

By Hilda Bastian, The Atlantic

08 March 21

Yes, all of the COVID-19 vaccines are very good. No, they’re not all the same.

ublic-health officials are enthusiastic about the new, single-shot COVID-19 vaccine from Johnson & Johnson, despite its having a somewhat lower efficacy at preventing symptomatic illness than other available options. Although clinical-trial data peg that rate at 72 percent in the United States, compared with 94 and 95 percent for the Moderna and Pfizer-BioNTech vaccines, many experts say we shouldn’t fixate on those numbers. Much more germane, they say, is the fact that the Johnson & Johnson shot, like the other two, is essentially perfect when it comes to preventing the gravest outcomes. “I’m super-pumped about this,” Virginia’s vaccine coordinator told The New York Times last weekend. “A hundred percent efficacy against deaths and hospitalizations? That’s all I need to hear.”

The same glowing message—that the COVID-19 vaccines are all equivalent, at least where it really counts—has been getting public-health officials and pundits super-pumped for weeks now. Its potential value for promoting vaccination couldn’t be more clear: We’ll all be better off, and this nightmare will be over sooner, if people know that the best vaccine of all is whichever one they can get the soonest. With that in mind, Vox has urged its readers to attend to “the most important vaccine statistic”—the fact that “there have been zero cases of hospitalization or death in clinical trials for all of these vaccines.” The physician and CNN medical analyst Leana Wen also made a point of noting that “all of the vaccines are essentially a hundred percent” in this regard. And half a dozen former members of President Joe Biden’s COVID-19 Advisory Board wrote in USA Today, “Varying ‘effectiveness’ rates miss the most important point: The vaccines were all 100% effective in the vaccine trials in stopping hospitalizations and death.”

There’s a problem here. It’s certainly true that all three of the FDA-authorized vaccines are very good—amazing, even—at protecting people’s health. No one should refrain from seeking vaccination on the theory that any might be second-rate. But it’s also true that the COVID-19 vaccines aren’t all the same: Some are more effective than others at preventing illness, for example; some cause fewer adverse reactions; some are more convenient; some were made using more familiar methods and technologies. As for the claim that the vaccines have proved perfectly and equally effective at preventing hospitalization and death? It’s just not right.

These differences among the options could matter quite a bit, in different ways to different people, and they should not be minimized or covered over. Especially not now: Vaccine supplies in the U.S. will soon surpass demand, even as more contagious viral variants spread throughout the country. In the meantime, governors are revoking their rules on face masks, or taking other steps to loosen their restrictions. It’s tempting to believe that a simple, decisive message—even one that verges on hype—is what’s most needed at this crucial moment. But if the message could be wrong, that has consequences.

The idea that all of the vaccines are pretty much the same, in that they’re perfect at preventing COVID-19 hospitalizations and death, got its legs on social media. The USA Today op-ed by the former members of the Biden team illustrated this by linking to a data table found on Twitter. Created by the infectious-disease doctor Monica Gandhi, it showed a variety of trial results for six different vaccines. One column was rendered in canary yellow—“Protection from hospitalizations/death”—and every cell read “100%.” A similar table, tweeted out a few days earlier by the dean of Brown University’s School of Public Health, Ashish Jha, conveyed the same idea through a grid of zeros—as in, zero people hospitalized, zero people dead. The prominent physician and researcher Eric Topol followed with his own clinical-trial data summary featuring a column of 100 percents. “That is impressive!” he wrote across the top. All told, their posts would be retweeted about 15,000 times.

The data were indeed suggestive of an encouraging idea. Based on the numbers so far, we can expect the vaccines to provide extremely high levels of protection against the most dire outcomes. Still, we don’t know how high—and it’s clear they won’t uniformly cause hospitalizations and deaths from COVID-19 to disappear in vaccinated people.

The experts understand this, of course. Gandhi has been updating her table as more data come in, and now pegs Moderna’s efficacy on that front at 97 percent; Jha has since tweeted that “nothing is 100 percent … But these vaccines sure are close”; and Topol told The Atlantic that the numbers in his tweet are not a sufficient basis from which to draw “any determination of magnitude of effect,” though the fact that they all point in the same direction is “very encouraging.” Still, the message of perfection that their initial tables and tweets spawned—the gist, for many readers, of all those 100s and zeros—has since been picked up far and wide, and misinterpreted along the way.

To grasp the shaky nature of these particular data, it’s important to remember how the vaccine-development process began. Last April, not long after the pandemic began, the World Health Organization set out a target efficacy for vaccines of 50 percent, with options for how that value should be measured. A vaccine could be shown to reduce the risk of symptomatic disease, severe disease, or transmission of the coronavirus. The FDA offered similar guidance in June, and other regulatory agencies also followed the WHO’s lead. Among these choices, symptomatic disease was the most feasible, because it’s both a common outcome and one that’s easier to confirm in a large-scale trial. An outcome that included asymptomatic infections would have been even more common, but screening for all infections would have been prohibitive, especially early in the pandemic. So that’s how the vaccine trials were designed: Each would try to demonstrate at least 50 percent efficacy with respect to symptomatic disease as its “primary outcome.”

The trials could have used severe disease, hospitalization, or death as primary outcomes, but that would have slowed things down. These events are far more infrequent—there could have been 200 infections for each COVID-19 death in the U.S.—and that means it would have taken more time, and larger numbers of trial participants, to generate enough data to be sure of a 50 percent efficacy. Developers did include “severe COVID-19” as a secondary outcome—that is, one that would be measured and analyzed, but for which the trial might not have been designed to provide a definitive answer. Efficacy against hospitalization and against death, however, were not included as secondary outcomes for every trial.

Given that fact, the data can’t support a claim that the vaccines are 100 percent effective at preventing these serious outcomes. (Topol highlighted this very issue in an op-ed last fall for The New York Times.) Out of the six vaccines included in the dramatic data tables that made the rounds on Twitter, the clinical trials for only two of them—Oxford-AstraZeneca’s and Johnson & Johnson’s—included hospitalization for COVID-19 as a secondary outcome, and reported that efficacy rate. The clinical research for one other vaccine, made by Novavax, had hospitalization as a secondary outcome, but that trial hasn’t been reported in full yet. (On my website, I’ve provided more detailed information and analysis of the relevant data.)

Now, a casual reader of clinical-trial reports—or their summaries on social media—might take the fact that no hospitalizations of vaccinated people are mentioned to mean that none occurred. That’s risky, given that pieces of the data have been published across various medical journals and via several different regulatory agencies rather than in full in one place; that the plans for some trials did not specify ahead of time that the vaccine’s efficacy at preventing hospitalizations would be calculated; and that we’ve seen only minimal early data (via a press release from Novavax) from one of them. It would be just as risky to assume that all hospitalizations would be included in the analyses of people who developed severe COVID-19. Hospitalization and severe disease are not synonymous—people could be coping at home even though COVID-19 has caused their oxygen levels to drop severely, and moderately ill people might be hospitalized out of an abundance of caution when they are at high risk of getting worse.

The two vaccine trials that did explicitly report hospitalizations as an efficacy outcome make this latter issue very clear. For the AstraZeneca vaccine, one person in the control group had severe COVID-19, but eight people were hospitalized; for Johnson & Johnson, 34 people in the placebo group had severe COVID-19, but only five people were hospitalized. It’s true that zero vaccinated people were hospitalized in either study after the vaccines took effect. But with numbers that small, you can’t draw a reliable conclusion about how high efficacy may be for these outcomes. As Diana Zuckerman of the National Center for Health Research pointed out about the Johnson & Johnson trial, “It’s misleading to tell the public that nobody who was vaccinated was hospitalized unless you also tell them that only 5 people in the placebo group were hospitalized.” She’s right. And you can’t be confident about predicting effectiveness precisely in a wider population outside the trial, either. For example, some of the vaccine trials included relatively few people older than 60 as participants.

You can see how fragile these numbers are by looking at those compiled for severe disease. In the Pfizer trial, for example, just one vaccinated person developed severe COVID-19 versus three in the placebo group—which meant that a single bout of disease made the difference between a calculated efficacy rate of 66 percent and one of 100 percent. For the Novavax and Oxford-AstraZeneca trials, there were zero people with severe disease in the vaccinated group versus only one in the control group, so adding or subtracting one would have been even more dramatic. The problem is even greater for deaths. For that efficacy analysis, only two of the vaccine trials—for Moderna’s and Johnson & Johnson’s—reported any COVID-19 deaths at all in the control groups.

It’s also important to remember that these are early results: Some people who enrolled very late in the trials aren’t yet included in reported data, and analysis is still under way. Indeed, the FDA pointed out in December that one vaccinated person in the Moderna trial had been hospitalized with apparently severe COVID-19 two months after receiving a second dose. That person was in a group still awaiting final assessment by the researchers, and was not mentioned in Moderna’s formal readout of results.

We’ve learned a little more from the ongoing public vaccination programs. Four important reports have come in the past two weeks. In one, researchers compared about 600,000 people who had had a full course of the Pfizer vaccine in Israel with 600,000 people matched in age and other demographics who had not been vaccinated. The shots’ effectiveness at preventing hospitalization was measured at 87 percent. (“This vaccine is fabulous in a real world setting,” Jha tweeted in response.) A preprint from Scotland reported an efficacy rate against hospitalization of about 80 percent among people 80 or older, almost all of whom had received only one dose of either the Pfizer or the AstraZeneca vaccine. Two reports from Public Health England estimated a reduction of hospitalization of about 50 percent and 43 percent for the same age group, again almost all after just one dose of the Pfizer vaccine. These are exciting outcomes—those vaccines really, really worked! But they oughtn’t lead anyone to think that the vaccines are all the same, and that protection will be perfect.

Where does that leave us for making decisions? As Anthony Fauci told The New York Times last weekend, “Now you have three highly effective vaccines. Period.” Again, you will get a lot of benefit from any of them, and your risk will shrink even more as those around you get vaccinated too. Whichever one you start with, a booster may be coming in the not-so-distant future, of the same vaccine or perhaps a different one. By taking the first vaccine you can get, you’ll also avoid the risk of finding yourself without protection if infection rates surge where you live.

Efficacy is merely one layer, though. The Pfizer and Moderna vaccines have an edge at preventing symptomatic illness, but the Johnson & Johnson vaccine brings its own advantages. It has no demanding freezer requirements, which means it’s easier to distribute and more accessible to many communities. It’s more affordable than the other two—the company is providing it at cost around the world. Then there’s the fact that resources can be stretched a lot further when only a single dose has to be administered.

For individuals, too, the Johnson & Johnson vaccine has benefits. As a one-and-done injection, it’s more convenient. It also has a lower rate of adverse events than Moderna’s. You can’t compare results of these trials too precisely, but there are indications of a striking difference. About 2 percent of those who got the Johnson & Johnson vaccine recorded having reactions, such as fatigue, muscle aches, and fever, that were severe enough to interfere with daily activities. For those getting their second injection of Moderna, that rate was higher than 15 percent. People who are on the fence about getting vaccinated may find that this difference tips the scales in favor of getting a shot. Others who have doubts about the newness of the mRNA technology in the Pfizer and Moderna vaccines may appreciate the fact that Johnson & Johnson’s approach has already been deployed in the company’s Ebola vaccine, which got full drug approval in Europe last year.

Given these concerns, there’s some danger in the message—however well intentioned—that the COVID-19 vaccines are all the same by any measure, or that they’re perfect wards against severe disease. Vaccination is a public-health imperative, and going full tilt to promote uptake supports the common good. But it’s a personal health decision too. People want to protect themselves and those close to them, and they are likely to care about outcomes other than hospitalization and death, no matter what anyone says now.

Still, raising these concerns in public can be fraught. In response to an inquiry about her data table, Gandhi affirmed the importance of looking at severe-disease outcomes and noted that “careful, collegial and collaborative scientific discourse on the vaccines is imperative moving forward to help us get through the pandemic.” Topol pointed out that he has emphasized the vaccines’ measured efficacy against symptomatic disease many times before, so any isolated reference to his table “takes that particular post out of context.” Jha wrote in an email that he stands by the message of his original tweet, and notes that COVID-19 hospitalizations and deaths are so rare among the people vaccinated in these trials, to quibble over differences is akin to “counting how many angels are dancing on the head of the pin.”

I can see why this might seem like quibbling, but I just don’t think it’s a trivial matter. It would be different if I thought the effectiveness of every one of those six vaccines against hospitalizations and death would really end up being close to 100 percent—or if I bought into the idea, now widespread, that they have already been shown to “nearly” or “effectively” eliminate these outcomes. There is very good reason to be encouraged by the data, but to say right now that people who have been vaccinated face zero risk of serious outcomes—that, for them, COVID-19 is no more dangerous than the common cold—is sure to influence behavior. Imagine how people in high-risk groups would feel about going to the movies, or how their employers would feel about putting resources into workplace safety, if we all assumed that vaccines confer perfect protection against hospitalization or death. Now imagine how the same people and employers would feel knowing they were 85 percent protected.

Nor is there any reason to believe that the public or the personal interest will be served by hype. People who think the vaccines provide ironclad protection may lose trust in experts if reality falls short. Trust in coronavirus-vaccine information is already a problem, and could sink even lower. Activists who are opposed to vaccination may end up turning experts’ “super-pumped” promises against them.

“The idea that people can’t handle nuance,” Jha tweeted at the end of February, “it’s paternalistic. And untrue.” I couldn’t agree more. The principle of treating people like adults is fundamental. We don’t need to exaggerate. Talking about the trade-offs between different medicines and vaccines is often complicated, but we do it all the time—and we can do it with COVID-19 vaccines too.