Why is it so hard to review the Johnson & Johnson vaccine? Data.

The future of the single-shot Johnson & Johnson covid vaccine remains in limbo after an advisory panel recommended taking a deeper look into reports of rare-and sometimes fatal-side effects.

The US Centers for Disease Control and the Food and Drug Administration advised suspension of the Johnson & Johnson vaccine on Tuesday, after reports that six people who had received a dose developed rare blood clots in the brain, combined with another disorder that actually inhibits clotting. One patient died.

After discussing the situation, the CDC's advisors said that the pause would continue for at least a week while information was gathered and reviewed.

We'll never have perfect data, and there will always be uncertainty," said Grace Lee, a professor at Stanford University and chair of the advisory panel's Covid-19 Vaccine Safety Technical Subgroup, when the group met on Wednesday. It's really, for me, about getting better risk estimates."

Committee members agreed to reconvene once they've had more time to gather and assess data about who might be most at risk of complications, and how that compares with the risk of catching and spreading covid.

All six of the cases reported after the vaccine became widely available occurred in women; one additional case-a man-was reported during clinical trials. All patients were between 18 and 48, and several were treated with the blood thinner heparin, which is typically used for clots but worsened the condition of these patients. The symptoms appear very similar to ones associated with AstraZeneca's covid vaccine, which many European countries have limited or even stopped using. The active components of both are delivered to cells by adenoviruses that have been modified so they can't replicate.

But because there are other treatments available that use totally different methods, experts say it is sensible to hold off to see if more information becomes available. The Johnson & Johnson vaccine counts for only 7.5 million of America's 195 million shots delivered; the Pfizer-BioNTech and Moderna vaccines, which use mRNA rather than adenoviruses, are responsible for the rest.

We'll never have perfect data, and there will always be uncertainty. It's really, for me, about getting better risk estimates."

The risks and benefits of continuing to administer the J&J vaccine can't be looked at in isolation," says Seema Shah, a bioethicist at Lurie Children's Hospital in Chicago. If people have alternatives, at least while the FDA is figuring things out, it makes sense to steer people in the direction of those alternatives."

Resumption of Johnson & Johnson shots may not mean that it becomes available to everybody, however. Ensuring the safety of vaccines is especially important because they're given to healthy people, rather than treating people who are already sick, and successfully figuring out which groups might see the most benefit-or most harm-could lead to tiered recommendations. Several EU countries, for instance, have said the AstraZeneca vaccine should be given to older people at higher risk of complications from covid, rather than younger people who might be at higher risk of vaccine complications.

At the end of the day, the critical issue is if I'm a 30-year-old woman and I get this vaccine, how much will that increase my risk of this bad thing?" says Arthur Reingold, chair of California's Covid-19 Scientific Safety Review Workgroup and a former member of the CDC's vaccine advisory panel.

A more complicated question is what data the committee will review to make a final decision.

No comprehensive data

Information may be limited because the issue was caught quickly, and because the Johnson & Johnson vaccine is so far being deployed only in the US (the company said it was delaying delivery to European Union countries). But making a determination may also prove difficult because America's medical data is highly fragmented.

Without a national health-care system, there's no comprehensive way to assess risks and benefits for different groups that have received the vaccine. There is no routine federal capability to connect patient data with vaccine records. Instead, regulators hope clinicians will hear about the pause and proactively report cases they hadn't previously connected to vaccinations.

It might stimulate some clinician to say, Oh my God, Mrs. Jones had that three weeks ago,'" says Reingold. In addition, he says, there's still quite a few people who have gotten a dose within the last two weeks, and some of them could develop this rare side effect."

The voluntary system may seem archaic, but that is how the six cases under review came to the attention of the authorities. They were reported to the CDC through an online database called the Vaccine Adverse Events Reporting System, or VAERS. It is an open website that medical staff, patients, and caregivers can use to notify the government about potential vaccine side effects.

Because the system is so open, and requires opt-in participation, it's impossible to calculate exact risks using VAERS data. Epidemiologists generally think of it as a place to look for hypotheses that tie vaccines to side effects, rather than a source that can be used to confirm their suspicions.

It's a messy system. Anyone can report anything, whether it's biologically plausible that it's related to the vaccine or not," says Mark Sawyer, a member of the FDA's Vaccines and Related Biological Products Advisory Committee, which reviewed the covid-19 vaccines for public use. Then the job is to sort through and figure out, is there really a signal here?"

The next best thing to a national health-care system is the CDC's Vaccine Safety Datalink, a consortium of American health insurers that provide medical care to patients in-house. The system includes records of about 10 million patients. Unfortunately, only 113,000 Johnson & Johnson vaccines have been captured by that system so far.

There just haven't been enough Johnson & Johnson vaccine doses given in the context of these other monitoring systems to detect a problem. This is just too rare an event," says Reingold. If all you've got left is VAERS or another passive reporting system, then you do the best you can."

There are many ethical and practical concerns around this pause, as discussed at Wednesday's committee meeting. Because the Johnson & Johnson vaccine requires just one shot and doesn't need to be frozen, it may be easier to distribute to people who have less access to health clinics. Other members of the panel, meanwhile, expressed concern that a continued pause would stoke vaccine hesitancy.

Many speakers wrestled with the limitations of available data, particularly regarding the breakdown of risks for different groups of people.

It's possible we may not have any more information, at which point we're still going to have to make a decision," said Stanford's Lee during the discussion. But my hope is that in the next week or two, we'll be able to capture [risk] in a more robust way."

Additional reporting by Mia Sato

This story is part of the Pandemic Technology Project, supported by the Rockefeller Foundation.