Science often sounds simple. Mix two substances, and the test tube explodes. Give a sailor vitamin C, and he avoids scurvy. No matter how many times you try, the outcome is always the same.
But that’s not always how it works. Scientific experiments sometimes produce different results when repeated—or can’t even be repeated at all. Who checks our scientific knowledge, and more importantly, is it even checkable?
In 2023, a network was launched in the Netherlands focused on the reproducibility of scientific research. It started in Groningen, and now nearly every Dutch university is involved, including TU/e (see box).
Michiel de Boer, a professor of epidemiology at the University of Groningen, and coordinator Daniela Gawehns, a data science PhD candidate in Leiden, hope this marks the beginning of a cultural shift.
Why is scientific research often difficult to replicate?
De Boer: “As a scientist, you should describe your approach as clearly as possible. That’s how we were all taught: in principle, someone else should be able to repeat your research. But in practice, that happens less often than you might expect.”
The problem sometimes starts with the description of the method, he explains. If you don’t know exactly how the research was conducted, you can’t replicate it properly. Often, information is missing.
De Boer: “Journals often impose word limits on publications, so you can never fully explain what you’ve done. And sometimes researchers simply don’t make their data or questionnaires available.”
Some parts of research can be replicated. Do you get the same results then?
Not always, says De Boer: “Replication studies show that less than half of repeated studies produce the same results.” But usually, this is based on a limited number of replications in a specific discipline. He can’t speak for science as a whole.
De Boer: “The fact that it went so poorly was an eye-opener for me.” The observed effects are often weaker than in the original study. Replication is so difficult that people talk about a replication crisis.
Should every study be replicable?
“It’s at least important to be transparent. Share not only your methods and data, but also, for example, your analysis techniques. Replicability will never be 100 percent, but it should be possible to get higher than 50 percent. Let’s say 80 or 90 percent. It may also depend on the discipline and the context of the research.”
Gawehns: “It also partly depends on your definition of successful replication. You’ll never get exactly the same outcome because there’s always noise in your research. Is replication only successful if you reach the same p-value, or is it enough that the effect points in the same direction? The latter is easier. In some disciplines, you can say: for replication, the graph should look roughly the same.”
What’s the danger if studies really turn out to be non-replicable?
Gawehns: “For example, there was promising research on dementia drugs. Millions of euros went into it. But the research turned out to be fraudulent, so it couldn’t be replicated. That’s a real waste of money.”
De Boer: “Science should be cumulative: you build on each other’s insights. But if the foundation is wrong, you have a problem. Sometimes it takes a long time before it becomes clear that we’re heading in the wrong direction. And a lot of research is funded with taxpayer money.”
When does this affect everyday life?
De Boer: “Especially in medicine, it can really be harmful if the effects of new drugs are not reproducible or even have negative effects. During the COVID-19 crisis, hydroxychloroquine seemed like a good drug for a while, but the research was shaky. It caused more harm than good. Details were left out, and it wasn’t clear where the data came from.”
And that’s why there should be more attention to replication?
De Boer: “It’s a piece of the puzzle. Of course, it’s not the only thing that matters in science. Sometimes you could replicate results from poor research, but then you just end up with bad research twice.”
“Collaborating on reliable science is essential”
Daniël Lakens, Associate Professor at TU/e in the Human-Technology Interaction group and member of the advisory board of the NL Reproducibility Network, echoes the network’s message: “Collaborating on reliable science across universities is essential.”
According to Lakens, a large part of the problem lies in human bias. “Scientists often report selectively what works and ignore results that don’t,” he says. This happens partly because exciting discoveries look better on a career track, but it is also deeply ingrained in us.
“Bacon already noted in 1620 that we have confirmation bias. We like to show that we are right,” Lakens explains. This tendency can lead to a body of literature full of unreliable results: something may appear to have an effect, while in reality it is just a statistical fluke.
Registered Reports
To counter this, Lakens says concrete steps have already been taken. “I proposed introducing replication grants at NWO, and those now exist. This allows researchers to repeat existing studies to check and validate the results.”
He also sees methods like Registered Reports as a way to structurally improve reliability. “With Registered Reports, the study design and analysis plan are reviewed before any data are collected,” he explains. “Based on this, submitters receive an acceptance letter – this guarantees that their paper will be published as long as they follow the plan correctly, regardless of the outcome. This increases objectivity and really helps determine what works and what doesn’t.”
In practice, this is already being done, but according to Lakens, not often enough. It is now up to the journals themselves to decide how to proceed. “If I were a research funder, I would require everyone who receives funding from me to use this format,” he says. But that doesn’t happen. “Research funders are often too lenient with us; they just let us do our own thing. That freedom is important, but sometimes they should take more responsibility,” he believes.
A complete shift
Such a new approach requires a complete shift, and that takes time. According to Lakens, universities should provide more support to researchers to implement good practices in their work.
“Science is very different than it was twenty years ago. There are so many new developments, and everyone has to learn to deal with them. For open science and data management, we now have data stewards who support researchers, but in the field of meta-science, there is still little support. Often we see that changes only happen once a law is passed, making it mandatory. That’s a shame. If the university can contribute to making science more transparent and reliable, why would you let that opportunity pass?”
Are you noticing more attention to the topic?
Gawehns: “On paper, everyone supports us. And we also see initiatives within institutes and research groups. Think of PhD students replicating each other’s research. But how do we make sure that all these small projects and good intentions actually lead to reproducibility in our daily work?”
Does it, for example, play a role when applying for a research grant?
De Boer: “There is some attention, but the system is far from perfect. When you submit a proposal to the research funder NWO, you often have to say something about open science: how will other researchers access your data? That ties into reproducibility. But you get ten or fifteen lines to explain it, and that’s usually enough. The reviewers mainly consider whether the proposal is interesting. And this all happens only upfront. There’s little monitoring once the project is underway. That’s problematic.”
Is extra oversight the solution?
De Boer: “Universities could certainly do more. Policy documents often say the right things, but in practice it’s hardly enforced. Then you rely on the intrinsic motivation of researchers to do it properly. We do see pockets of excellence, as we call them, but outside those departments, nothing happens.”
Doesn’t it also depend on the discipline? I can imagine this topic is more relevant to quantitative research than, say, historical literature.
De Boer: “Still, it’s one of our priorities: reproducibility for the non-quantitative research disciplines. I know two historians who do replication studies with their students. They examine exactly how someone approached their work. Are the footnotes sufficient to trace how people carried out their research? Is their methodology transparent enough? That sparks really good discussions.”
A new government is coming. What do you expect from politics?
De Boer: “Work pressure is an important issue for us. One reason researchers avoid working transparently is that it takes time. At least that’s the perception. Of course, it helps if there’s some breathing room in the system. That said, there are also things we can do ourselves in science.”
What can science do itself?
De Boer: “We could perhaps publish a bit less. Scientists are often judged on their publication list, but some articles simply aren’t that valuable. We could be more selective and free up time to work more carefully. That insight is slowly spreading.”
The NL Reproducibility Network was founded in 2023. Almost all universities are now involved, including TU/e. Last month, the third symposium took place.
This article was translated using AI-assisted tools and reviewed by an editor
Discussion