How We Decide if COVID Boosters Work: The Test-Negative Design

If you watched, or read about, last week’s ACIP meeting, you might have heard CDC scientists talking about the “test-negative design” as a tool for evaluating COVID boosters. But if you’re not a (respiratory) infectious disease epidemiologist, there’s a very good chance you’ve never encountered this niche concept before. Which is a shame, because it’s important and a cool idea. So, let’s talk about what a test-negative design is, when it works, and when it doesn’t.

What is the test-negative design?

The test-negative design is a type of observational study design used for many years in influenza epidemiology and more recently in COVID epidemiology. It is a subtype of a more widely known study type called a the case-control study but with a couple unique features that help to address some of the specific ways in which studying the flu and COVID is hard.

In the simplest terms, the test-negative design recruits participants who have received a specific type of diagnostic test. In the case of COVID, that might be a lab-based PCR test. Often, there are a fixed subset of labs which participate in these studies over time and which provide information about all tests of this type conducted during the study time frame. Tests identified by the study team are then categorized into two groups: positive tests and negative tests. These two groups define the groups for analysis. If you’re familiar with the idea of case-control study the positive tests are the “cases”, and the negative tests are the “controls”.

Now that we have two groups of people, we can compare them and (hopefully) learn something from the differences between them. This is where things get a bit complicated, because ideally, we would like to know what proportion of people who got vaccinated have a positive versus negative test compared to the proportion among people who didn’t get vaccinated. But instead the data we have are the proportions of people who did versus didn’t get vaccinated among those with a positive test and among those with a negative test.

That is, we don’t actually care what the probability that someone had the vaccine is among those with positive test (i.e., Pr(V+|T+)), we care what the probability of having a positive test is among those who got vaccinated (i.e., Pr(T+|V+)). Unfortunately, we don’t know how many people total in the population got vaccinated so we can’t measure the probability we actually care about.

This is where the magic of statistics comes in.

It turns out that if we could compute the odds of testing positive vs negative among the vaccinated and compare that to the odds of testing positive vs negative among the unvaccinated (a quantity which we call the odds ratio), this will be mathematically equivalent to comparing the odds of being vaccinated vs not among those with a positive test compared to the odds of being vaccinated vs not among those with a negative test. And even though we can’t do the first thing, we can do the second thing.

So, like magic, we can use the data we have to answer the same question we would have answered with better data!

For those of you who like to see math as equations rather than text, here’s what that means (if you hate equations, feel free to scroll on past these. no judgement here!):

\(\begin{array} {l|l|l} & \text{vaccinated} & \text{unvaccinated} \\ \hline \text{test positive} & a & c \\ \text{test negative} & b & d \\ \end{array}\)

\(\begin{align} \text{odds of test+ among vaccinated} &= \frac{\text{test +, vaccine}}{\text{test-, vaccine}}\\ &=\frac{a}{b}\\ \\ \text{odds of test+ among unvaccinated} &= \frac{\text{test +, no vaccine}}{\text{test-, no vaccine}}\\ &=\frac{c}{d}\\ \text{odds ratio for test by vaccine status} &= \frac{\text{odds of test+ among vaccinated}}{\text{odds of test+ among unvaccinated} } \\ &=\frac{\frac{a}{b}}{\frac{c}{d}} = \frac{ad}{bc} \end{align}\)

\(\begin{align} \text{odds of vaccine among test+} &= \frac{\text{vaccine, test+}}{\text{no vaccine, test+}}\\ &=\frac{a}{c}\\ \\ \text{odds of vaccine among test -} &= \frac{\text{vaccine, test-}}{\text{no vaccine, test -}}\\ &=\frac{b}{d}\\ \text{odds ratio for vaccine by testing} &= \frac{\text{odds of vaccine among test+}}{\text{odds of vaccine among test-} } \\ &=\frac{\frac{a}{c}}{\frac{b}{d}} = \frac{ad}{bc} \end{align}\)

For those of you who hate equations, the bottom line is that both of these odds ratios can be measured with the same exact calculation: (ad)/(bc). That means that even though we don’t know how many people in the population received the vaccine, we can still calculate the relative proportions of testing positive versus negative by vaccine status!

Summary: for the test-negative design, we collect information on everyone with a positive test result and a negative test result, find out whether they were vaccinated, and then calculate the odds ratio. This tells us about the relative frequency of a positive compared to negative test is among the vaccinated versus the unvaccinated.

When does the test-negative design work?

The test-negative design is a type of case-control study, which means that the number one requirement for this study design to give a reliable answer is that the selection of people into the study must not depend on the likelihood that an individual has been vaccinated. Specifically, in the case of a test-negative design this means we need the following things to be true:

1. There must exist other diseases with symptoms mimic the disease we are vaccinating against and that can trigger testing for this disease.

2. Relative severity of symptoms with those other diseases must be similar to that of the disease of interest, so that diagnosis-seeking behavior is independent of likely test status.

3. The chance of being infected with one of those other diseases must be entirely independent of the disease and vaccine we are studying.

4. The decision to get vaccinated or not must be unrelated to risk of infection with the disease of interest.

5. The vaccine must offer “all-or-nothing” protection (either it works completely or it doesn’t work at all, for a given individual)

If these five things are true, then we don’t need to worry about errors in our analyses caused by differences between who does and doesn’t have access to diagnostic testing or other medical care. That’s a very useful feature!

When does the test-negative design not work?

If any of the five criteria above are not true, we can’t necessarily expect the answer from a test-negative design study to be accurate. This is particularly true if:

1. The vaccine is “leaky” meaning that some people can get partially protected from infection by the vaccine.

2. People who are at higher risk of exposure to infection are more likely to get vaccinated.

3. People alter their health-care seeking behaviors based on vaccination status (for example, if people who are vaccinated are more likely to seek care for milder symptoms than people who are unvaccinated).

4. The disease of interest has easily identifiable symptoms, so that people with other diseases are unlikely to be tested for the disease of interest.

In these circumstances, the basic requirements for the test-negative design won’t be met. However, there may still be ways we can use the test-negative design data if we apply more complicated analytic tools to correct for these problems.

Does the test-negative design tell us about the “general population”?

One of the questions at last week’s ACIP meeting was to what extent the test-negative design can tell us about the effect of a vaccine in the “general population”, as opposed to some specific population of high-risk people who might get vaccinated.

The answer is that if the criteria for the test-negative design hold, the results will tell us about all people who are eligible to receive the vaccine. If the vaccine is available to everyone, then a valid test-negative design will tell us how well the vaccine works in everyone.

If the vaccine is not available to everyone, then we need to modify our design to make sure we are only including positive and negative test results among the types of people eligible for the vaccine.

Further reading

If this post got you all excited about the test-negative design, and if you want to really dig into the technical details, I recommend Lewnard et al 2018, which is available at PubMed Central here: https://pmc.ncbi.nlm.nih.gov/articles/PMC6269249/

---

As an independent epidemiologist, I make my living entirely through reader support. Readers like you make E is for Epi possible. If you find value in this work, consider becoming a paid subscriber.