When Will We Really Know if Moderna’s Vaccine Works?

Moderna today released a long document detailing the protocol for its COVID-19 vaccine trial. Although there has been much speculation that the results of that trial would be available relatively soon, leading to an early authorization to use the vaccine, this release appears to douse that expectation. According to a New York Times article summarizing the information in the document, the soonest they expect to analyze the vaccine’s efficacy is in late December.

But a detailed analysis of the trial’s protocol gives reason to believe the first interim analysis will be much sooner, perhaps as early as mid-October. The protocol is specific that the first analysis will be done when they reach a total of 53 postive cases of COVID-19 combined between the test group and the control group. The late December target is merely a guess as to when they’ll reach that number.

Here are the factors that will determine how quickly they actually hit 53 cases:

• How fast they sign up volunteers. Their goal is 30,000. The plan was to add those volunteers equally over 13 weeks. So far they seem to be ahead of schedule, as they are about ten weeks in, but have already had 25,000 people begin the trial.

• How well the vaccine works. The better it works, the longer it will take to get to 53 cases, as most cases of COVID-19 will have to come from the control group. Of course, that would make it more likely that the results are statistically significant, so would increase the odds that the first analysis is sufficient for authorizing the vaccine’s use.

• How many people in the control group test positive for COVID-19. This is where the Moderna document is ultra-conservative. It assumes that, over a six month period, 0.75% of the study participants in the control group will get COVID-19. That breaks down to 29 people per 100,000 per week. However, in the seven days ending September 16, approximately 82 people per 100,000 tested positive in the U.S. That’s in the general population. There is good reason to expect that the people in the Moderna study will be more likely to test positive, for two reasons: one, Moderna has focused its study in areas with high rates of incidence, and two, people in the study are closely monitored and frequently tested, so there should be a lower proportion of undiagnosed cases than there are in the general population.

Can we estimate when they’ll actually hit 53 cases? Per the assumptions in the model, it will occur around the last week of December. If participants test positive at the current rate of Americans in general, they’ll hit that target by mid to late November. If they’ve done a good job enrolling high-risk candidates and after accounting for the use of consistent testing, I expect they’ll be ready to do the initial analysis in the middle of October.

If The Vaccine Is Analyzed So Soon, Will We Know if it Works?

Suppose that, of the 53 people in the study with positive tests, 38 are from the control group and 15 are from the vaccine group. That result would be consistent with Moderna’s goal of 60% efficacy. And it would be statistically significant, in the sense that, it would be certain that the vaccine had some level of efficacy, though perhaps less than 60%. The numbers would still be too small to rule out chance in making the vaccine appear more effective than it really is.

According to the protocol document, they will “declare victory” at the point the numbers demonstrate a 95% level of confidence that the vaccine is no less than 30% effective. If they hold to that standard, it could force them to wait for the second initial analysis, likely to be a month or so later.

It’s not hard to imagine this target being softened. After all, there is nothing magic about it. If the vaccine appears to be 60% effective, and there are no documented safety concerns, and if we’re 99% sure it’s at least somewhat effective, is there a good reason to continue to wait? I think the pressure to authorize its use will be quite strong at that point, and those pushing for it will have a pretty compelling case.