Define your terms. It’s one of the oldest rules of writing. Yet when it comes to defining the exact resources used to conduct their research, many scientists fail to do exactly that. At least that’s the conclusion of a new study, published on Thursday in the journal PeerJ.
Looking at 238 recently published papers, pulled from five fields of biomedicine, a team of scientists found that they could uniquely identify only 54 percent of the research materials, from lab mice to antibodies, used in the work. The rest disappeared into the terse fuzz and clipped descriptions of the methods section, the journal standard that ostensibly allows any scientist to reproduce a study.
“Our hope would be that 100 percent of materials would be identifiable,” said Nicole A. Vasilevsky, a project manager at Oregon Health & Science University, who led the investigation.
The group quantified a finding already well known to scientists: No one seems to know how to write a proper methods section, especially when different journals have such varied requirements. Those flaws, by extension, may make reproducing a study more difficult, a problem that has prompted, most recently, the journal Nature to impose more rigorous standards for reporting research.
“As researchers, we don’t entirely know what to put into our methods section,” said Shreejoy J. Tripathy, a doctoral student in neurobiology at Carnegie Mellon University, whose laboratory served as a case study for the research team. “You’re supposed to write down everything you need to do. But it’s not exactly clear what we need to write down.”
Ms. Vasilevsky’s study offers no grand solution. Indeed, despite its rhetoric, which centers on the hot topic of reproducibility, it provides no direct evidence that poorly labeled materials have hindered reproduction. That finding tends to rest on anecdote. Stories abound of dissertations diverted for years as students struggled to find the genetic strain or antibody used in a study they were recreating.
A Red Herring?
Here’s what the study does show: In neuroscience, in immunology, and in developmental, molecular, and general biology, catalog codes exist to uniquely identify research materials, and they are often not used. (The team studied five biomedical resources in all: antibody proteins, model organisms, cell lines, DNA constructs, and gene-silencing chemicals.) Without such specificity, it can be difficult, for example, to distinguish multiple antibodies from the same vendor. That finding held true across the journals, publishers, and reporting methods surveyed—including, surprisingly, the few journals considered to have strict reporting requirements.
This goes back to anecdote, but the interior rigor of the lab also wasn’t reflected in its published results. Ms. Vasilevsky found that she could identify about half of the antibodies and organisms used by the Nathan N. Urban lab at Carnegie Mellon, where Mr. Tripathy works. The lab’s interior Excel spreadsheets were meticulous, but somewhere along the route to publication, that information disappeared.
How deep and broad a problem is this? It’s difficult to say. Ms. Vasilevsky wouldn’t be surprised to see a similar trend in other sciences. But for every graduate student reluctant to ask professors about their methods, for fear of sounding critical, other scientists will give them a ring straightaway. Given the shoddy state of the methods section, such calls will remain a staple even if 100 percent of materials are perfectly labeled, Ms. Vasilevsky added. And that’s not necessarily a problem.
“It’s not bad for promoting scientific collaboration,” she said.
Indeed, it’s possible to see the study’s focus on reproducibility as a red herring, ultimately. There is another motive in play. Several of its authors are members of Force11, a collaboration that arose in 2011 to modernize scientific publishing for the digital age. One particular concern of the group is that computers, rather than human beings, will ultimately digest the studies for research databases. Proper labeling will make those algorithms much more effective.
So, researchers, give the team’s recommended guidelines a chance.
Don’t do it just for yourself. Do it for the machines.