Be wary of press releases reporting Covid-19 treatment breakthroughs. Most have not undergone the scrutiny of independent review.

But there are also growing concerns — on the part of scientists and journalists — that the studies being offered up and showcased are not ready for “prime time.” In fact, many are not studies at all, but subjective conclusions based on data, and methods that remain hidden and thus difficult to validate. Never before has full and immediate transparency been so important, and never before has the scientific picture around Covid-19 been so opaque.

What difference does the source make?

Press releases, pre-print papers and published papers all serve different purposes, and carry different weight for both scientists and journalists.

A study published in a credible scientific journal is — in theory — the final, complete version. To get published here, a study has to undergo a process called peer review. Kate Grabowski, an assistant professor in the department of pathology at Johns Hopkins University, calls the peer-review process “multiple, independent sets of eyes” on a paper. While peer review is by no means fool-proof, it typically reflects the expertise of many people in a particular field who don’t necessarily have a “dog in the race.”

“I think it’s just so valuable to picking up potential errors that are largely unintentional, and also just making the science better. Usually when we submit papers, they’re like rough drafts, and then may get refined [several times] until they’re much better,” Grabowski said. She described the process, to us, as “iterative.”

But the past few months have highlighted that the road to solid science can be full of potholes, speed bumps, blind spots and hairpin turns. If you are not careful, sometimes that road can lead you straight off a cliff.

Here are several recent examples of the story getting ahead of the science:

In fact, when it comes to the vaccine race, despite all the talk you may have heard from Moderna or Oxford University or “Operation Warp Speed,” or having billions of doses by the end of the year, remember: We’ve only seen that one study published in The Lancet. That’s it. We need to remember to temper our hopes and enthusiasm with the facts we have.

I include these examples because they show how different parts of the process can break down.

“Trying to do science by press release, without backing it up, either with a traditional journal or a preprint … has universally led to misunderstanding and has no place in science. The biotechs are doing it because they’re writing for their shareholders, they’re writing for their investors, but it’s being done in a way that’s oblivious to its public health impact and needs to stop,” Hotez said.

Whiplash for consumers of health news

When this happens — and especially when errors are revealed or papers are retracted — public trust is eroded and people begin to doubt science.

“It’s like whiplash,” Oransky said. “I’d be really confused.”

But nowadays that whiplash has become almost inevitable because that back and forth is how science moves forward in this Covid way of life.

Adding to the whiplash and confusion is just the sheer number of studies coming out. According to Grabowski, who based her estimate on the NIH’s iSearch COVID-19 Portfolio, approximately 35,000 articles have been amassed to date on the topic — and they keep coming.

To get a handle on all of them, Grabowski heads up a team of about 50 Johns Hopkins University researchers at the Novel Coronavirus Research Compendium, which curates and reviews emerging research. She estimates they screen between 1,500 and 2,000 articles per week, and have looked at more than 10,000 of them total. As you might expect, some are garbage and some are gems.

“I would say there’s definitely some excellent studies that are being conducted under a really rapid timeframe,” Grabowski said. “It’s really amazing to see science moving at this pace. I don’t think we’ve ever seen anything like this before.”

Buyer and seller beware

All the experts said it is great news that so much research is being done, despite the fact that so much of the work is emerging through press releases and pre-prints.

“The fact that scientists are getting work out there earlier so that other scientists can pore over it and we can maybe learn things more quickly — that’s a good thing,” Oransky said. “The fact that we’re treating it all equally as if it’s all been … subject to the same level of scrutiny — that’s the problem.”

Schwitzer cautions journalists and those who disseminate information to take the time to do it right. “Just reminding people to slow down. So much of what we’re doing, reporting breathlessly at breakneck speed, doesn’t need to be reported restlessly at breakneck speed,” he said, adding, “Words matter and the data matter.”

So, what does this all mean for you? “I think that someone reading, viewing, watching, listening should never make any decisions based on a single report they read, whether it’s a study or a news report on a study,” Oransky said. “Particularly if that news report doesn’t put into context everything that’s come before and doesn’t explain what we still don’t know.”