UPDATE: Thanks to a commenter for finding the study protocol for the Pfizer study. That helps a lot.
1. If you assume that everyone is identical, and if you want to try to eradicate the virus by giving everyone the vaccine, then you don’t need to show that it is 90 percent effective. My guess is that 20 percent effectiveness will do the trick. The problem you face is a PR problem. Most people hear the word “vaccine” and think “super-power that keeps me from getting sick.” But to be honest you would have to say “You might still get sick, but if everyone takes the vaccine and also exercises some degree of precaution, the virus will die out and eventually you won’t have to worry about it.”
2. But if your goal with the virus is to target high-risk populations or people whose working conditions might expose them to high viral loads and make them safe, then you want the super-power story to be true. I would want to see that high viral load does not degrade the effectiveness of the vaccine, and I would want to see that being in a high-risk category does not degrade the effectiveness of the vaccine.
3. The most I can find out about the Pfizer study is from this press release.
The first primary objective analysis is based on 170 cases of COVID-19, as specified in the study protocol, of which 162 cases of COVID-19 were observed in the placebo group versus 8 cases in the BNT162b2 group. Efficacy was consistent across age, gender, race and ethnicity demographics. The observed efficacy in adults over 65 years of age was over 94%.
There were 10 severe cases of COVID-19 observed in the trial, with nine of the cases occurring in the placebo group and one in the BNT162b2 vaccinated group.
There are a number of comparisons you could make between the placebo group and the treatment group.
–number of people who tested positive for the virus
–number of people who tested positive and showed symptoms (this appears to be what they used)
–number of “severe cases” (this was reported in the press release)
–number of hospitalizations (is this the same as “severe cases”?)
–number of deaths
Suppose that nobody in either group died from the virus. A headline that says “Vaccine prevents zero deaths” would not be very inspiring, would it?
When a study can look at many possible outcome measures and chooses to report only those that favor the drug, this is known as p-hacking. I don’t know that Pfizer was p-hacking, but I don’t know that they weren’t.
UPDATE: the protocol specified the outcome measures in advance, so no p-hacking.
The context here is one in which people who spread the virus differ greatly in their ability to spread it (at least, that seems like a good guess), and people who come in contact with the virus differ greatly in the extent to which they get sick. In that context, a ratio of 162/22000 compared to 8/22000 is promising but not definitive. I would be much more impressed if it were 1620/22000 and 80/22000. With the smaller numbers, I can think of a dozen ways to get those results without the vaccine actually being effective.
UPDATE: Now that I have seen the protocol, I can go back to the first two points. For the purpose of trying to eradicate the virus with universal vaccination, you don’t need much efficacy. But I think you want to be really, really, confident that there is some efficacy, because otherwise you will have blown it with the public if your vaccine campaign does not eradicate the virus. If I were the public official in charge of making the decision, I would want to see a larger number of cases in the sample before I would make this kind of a bet.
For the purpose of protecting vulnerable populations, you need to conduct a different protocol, in my opinion. Again, you want to know how the vaccine does as viral load goes up and as vulnerability of the individual goes up. I think that would argue for a different study protocol altogether.
Presumably, Phase 3 trial statistics line up with eventual efficacy of drugs/vaccines etc. I’d think if there was a significant error there, phase 3 criteria and procedures would be amended to address that. I can’t find a study that shows what typically happens when you go from phase 3 to general population, but I’d assume if the numbers didn’t line up, something would have be done about it. I mean, the entire point of the whole approval process is to come up with a measure that correlates well with the eventual efficacy.
There are all sorts of mechanisms that can lead to phase 3 numbers not being representative of the eventual results. If we don’t know what those mechanisms are, then the next best thing is to see statistically, how often those mechanisms make phase 3 numbers irrelevant, and by how much.
I’d say FDA people should have those numbers, otherwise, how would they know if their design of experiments is correct.
A problem is that it’s hard to even put a confidence interval around the efficacy, because the number of people actually exposed to the virus in each group is a latent variable, and not necessarily even between groups (although not gamed).
And as you suggest binomial observational data can be quite noisy.
Arnold, the study protocols are available online. This appears to be the Pfizer one: https://pfe-pfizercom-d8-prod.s3.amazonaws.com/2020-09/C4591001_Clinical_Protocol.pdf
The pre-registered phase 2/3 endpoints start on page 30. If you haven’t read it, perhaps you should before intimating that they’re p-hacking.
Having 10 times as many cases would of course make us more certain, but if it took 10 times as long to get those results at a time when the country is averaging 1600 deaths per day, the certainty might not be worth the wait.
The pre-specified endpoints, including the ones designated “exploratory”, do not include breakdowns by demographic group. The press release statement…
Efficacy was consistent across age, gender, race and ethnicity demographics. The observed efficacy in adults over 65 years of age was over 94%.
…evidently goes beyond the protocol-specified analyses.
I don’t think they’re cherry-picking favorable results in reporting the after-the fact demographic breakdowns, just responding to the intense demand for same. But with even smaller N’s in the subgroups, making statements about “consistent” efficacy across the groups is questionable.
So, are the libertarians starting to sound like the silly right and their accusations of voter fraud conspiracies? P-hacking accusations are just another variation of this…very interesting to bear witness to this.
“The problem you face is a PR problem. Most people hear the word ‘vaccine’ and think ‘super-power that keeps me from getting sick.’”
Yeah, I dunno. The annual flu vaccine has an efficacy of like 40%. Yet, every year the health officials, media and my local CVS do massive marketing campaigns and everyone lines up like sheep to get the shot (myself included). Yet, everyone intuitively knows that it’s not 100% effective, even if they cannot pinpoint the precise efficacy.
So, for this particular virus, this doesn’t seem like an insurmountable problem to me, especially since the strange orange man will be gone.
You can lead a libertarian horse to water, but will he drink?
Hans makes a good point. People understand how this works with the flu vaccine. It improves your odds. It’s not a guarantee. And those who really take an interest will know that there are versions of the common cold caused by other coronaviruses and no one has succeeded with any vaccine against them. That’s why the story here is that the initial results exceeded expectations by a lot.
>—“I think you want to be really, really, confident that there is some efficacy, because otherwise you will have blown it with the public if your vaccine campaign does not eradicate the virus.”
Disagree. How many vaccines other than Smallpox have fully eradicated any disease? Nothing you do will satisfy hard core anti-vaccers and most of the rest of the public will be happy to have their odds substantially improved.
Thanks for you additional insights! I actually feel really sorry for us as a country right now. This is the low point. The HEEs just gifted us two viable vaccines with a third not far behind. And yet, we are left with this overly skeptical nonsense. What the f*ck is wrong with us?
Data so far seems to show that two shots of the Oxford/AZ vaccine gives antibody levels equivalent to the average infection. One shot of Moderna/Pfizer gives gives the same result, a booster 2-3x antibody levels of a recovered patient. So, while not empirically tested, you can infer protection against higher initial viral dose.
Also, keeping in mind secondary (post-vaccine) immune responses get going about 12 hours post infection. Contrast this with a primary response that typically peaks ~14 days post infection.
If we do vaccinate even 60% of our populations, likely also, the number of super spreaders shedding large amount of virus will decrease.
In any event…. what’s the alternative here? The option is a vaccine that seems likely to give high levels of protection or nothing. Where I live in Australia, with very little covid, perhaps we can wait and be picky for the best product. For everyone in Europe and the USA, getting covid seems highly likely in the near future, seems best to take whatever vaccine is on offer.
I think Arnold is worried that if we say the vaccine is the cure, and its not the cure, whatever trust is left in the public is done for good.
That doesn’t mean we shouldn’t do the vaccine, or that it won’t be good enough.
Added to this is Arnold’s worry that public health officials have shredded their credibility this whole time because they are just terrible.
So, the guy that brought us silly voter fraud conspiracies over the last few weeks is left to defend the anti-vaxxer arguments. Love it! God bless you.
The Oxford/AZ results are looking less robust now:
https://www.wired.com/story/the-astrazeneca-covid-vaccine-data-isnt-up-to-snuff/
The Pfizer and Moderna studies appear to have been conducted far more rigorously.
It’s premature to conclude that either of the Pfizer and Moderna vaccines will deliver 90+% efficacy and long-term safety in broad use. There’s too much we don’t know about them yet. But would anyone rather that the initial read-outs had shown poorer results? Take the initial reports as the good news that they are, but keep the unresolved questions in mind and continue to pay close attention to what is learned from ongoing follow-up in the trials and as field experience with these vaccines expands.
I agree we should have more data before making an all-out bet on the vaccine, but it looks like the point is moot: Cold storage, double dose, production, etc. mean that we cannot place the “big bet” anytime soon. By the time will are able to consider a “big bet,” we will have evidence of efficacy among first responders, etc. in specific locations so information about high viral loads. When we face the choice, we will have much better information from which to make an informed decision — and perhaps some alternatives available as well.
Arnold, yes we want to know where we are today on the virus, the disease, and the pandemic. I don’t know where I’m but more importantly, I’m sure that nobody knows. We can fantasize and speculate about where we are today and where we will be a year from now, but to know what happened the past 12 months and where we today require hard, serious work, well beyond what any of us individually can do in less than 12 months. I shared your early disappointment on our poor understanding of what was going on, but your new three posts show that we have learned little since then (our disguise your ignorance by discussing the vaccine but what you say applies to all other specific issues). Too much serious research has been done. A few days ago I received an email from SSRN saying that now I can access more than 21,000 medical preprints, but it looks like entering into an old library: like Tyler Cowen, I can say a few words about the collection and each book, but I know nothing about them.
We need scientists’ knowledge for politicians to make decisions today. Last March, we knew that the Administrative State, both at the world level (WHO and other international organizations) and the country level (all countries, except a few with different histories and state capacities) had failed despite the huge resources that had been assigned to them for the specific purpose of protecting us. The best we can say is that the new virus was so different from previous experiences that scientists treated it as if it were a new species —the differences were critical. At the national level, most politicians, however, opted for treating it as an extreme case of previous coronaviruses because they needed to announce a plan to recognize the pandemic (otherwise they’d have been accused of shouting fire). Yes, that is the best we can say about what happened last March.
My impression is that today most scientists don’t believe Covid-19 is a new species. Most politicians, however, want us to believe it is and that extreme forms of the initial plan (flattening the curve) are still required. For most scientists, the end of the pandemic is a question of a few months regardless of what politicians do (yes, I may be relying too much on scientists’ “herd” instinct as promoted by perverse incentives). For most politicians, the end of the pandemic is something they must attempt to control for their own personal benefit.