Onderzoek 2. Najaar 2020.

A single replication is not a replication

Replication studies must contribute to finding the truth. Yet, instead, replication research often seems to create confusion. Does a failed replication mean that the original conclusions were incorrect? ‘Often, several replications are needed, and the outcome turns out to be more nuanced.’

Text: Gaby van Caulil, image: Vincent van Gurp

For his finding that a pen in your mouth causes you to smile and for his discovery that this this was not the case after all, Fritz Strack received the Ig Nobel Prize – a parody on the Nobel Prize – for Psychology this year. The German psychologist responded sportively (‘The pen procedure may not be effective to alter moods.’) when his research from 1998 proved not to be replicable in 2017. Usually, replication research does not lead to a clear confirmation or rejection of the original conclusion. According to a Science article from 2008, conservative people are supposed to be be more afraid. Following a replication study at the University of Amsterdam, that proved to be a very far-reaching conclusion. That reading literature makes you more empathic – published in 2013 in Science  – was found five years later to be far more nuanced. And it is very questionable whether you really do walk more slowly after reading words that are associated with old age.­­­­­­­

Register in advance and stricter statistics

That replications fail to stand up is not due to deliberate deception in the original research, according to a range of replication experts. They consider publication bias to be the most important cause. As a result of that, far too many studies with positive results are published in the scientific literature.

If a researcher wants a result to be positive, then he or she is on dangerous ground

Joost de Winter

According to Joost de Winter, biomechanical engineer at TU Delft, and a replication expert, that can be solved. ‘By registering research in advance and applying stricter statistics. But the researcher's critical attitude is crucial. If a researcher wants a result to be positive, for example because he or she has an interest in that, then he or she is on dangerous ground.’­­­­­­

Replication crisis in psychology

The scientific attention for replication can be traced back to the so-called replication crisis: in 2015, one hundred psychology experiments were repeated, and in more than half of the studies the original effect was not demonstrated. In 2016, seven of the eighteen economic studies tested, proved not to be replicable. Replication is therefore important – falsification is a basic principle in science. Nevertheless, at NWO some replication proposals were not funded because innovation is an important criterion too. Furthermore, few researchers find it inspiring to repeat the work of others. Therefore, psychologist Daniel Lakens suggested simulating replication with separate funds. In 2016, NWO was the first research funding body in the world to make grants solely available for replication research. Seventeen projects are currently underway, and two studies have been completed.­­­­­­­

Male nude: pupils dilate

Joost de WInter knows exactly how difficult it is to repeat a study and why it is important to do several replications. In 2017, he had the ambition to realise an exact replication. ‘This immediately gave rise to a dilemma because we knew that the original method had limitations.’ In a classical experiment from 1960, the American psychologist Eckhard Hess concluded that you could infer somebody’s emotions from the size of that person’s pupil. Hess showed study subjects various photos, and he filmed the pupil. For female study subjects, the pupil was up to twenty percent more dilated it if they saw a baby and male nudity. A landscape caused the pupils of women to constrict by several percent. The pupils of men were almost twenty percent bigger when they saw female nudity. In brief, if you see something that you find interesting, the pupil becomes more dilated. According to Hess, a repulsive image makes the pupil smaller. Science published the conclusion and the article was subsequently cited 790 times. Years later, Hess wrote that he had a vested interest in the study: he was a consultant for an advertising agency that wanted to determine whether advertisements for Coca-Cola, for example, were found to be attractive.­­­­­­­

Pupil constriction never measured

De Winter's team analysed the work of Hess (48 boxes with notes and tables), made copies of the photos used, ordered the same projector on eBay as was used at the time and reconstructed the wooden setup. The results were only partially confirmed. The effects on the pupil diameter could largely be attributed to differences in the clarity of the image. Therefore a methodological error. De Winter repeated the experiment with a revised design. ‘We showed images with just contours so that the light intensity remained the same. We did not measure the pupil constriction described by Hess a single time. Other results from Hess were replicated. Moreover: merely showing a stimulating word such as nude already led to pupil dilation.’ Other aspects to take into account are the so-called hidden moderators that can hinder a replication. De Winter: ‘For example, in our case, it cannot be excluded that cultural context plays a role. Photos that people found interesting in the 1960s are possibly less exciting in 2019. Also, I cannot rule out that the setting had an effect, for example, if participants suspect that the studies are about sexual attraction. The exclusion of such hidden moderators again requires new replications.’­­­­­­­­

Reproducing from the article

Replication is a relatively young discipline. However, according to the most experienced replication expert in the Netherlands, cognitive psychologist Rolf Zwaan, it is nevertheless clear that a failed replication does not yet mean that the conclusion of the original research was incorrect. Zwaan is a member of the NWO programme committee Replication Studies and author of “Making replication mainstream”. ‘Often, several replications are needed, and the outcome turns out to be more nuanced. It is also difficult to reproduce a study based on information in an article. Therefore it is vital that researchers make data available for others.’ Replication specialists still find it too early to conclude that replication studies lead to more robust science. More replication studies need to be completed before such a statement can be made. Zwaan does, however, think that science has become more reliable because replication exposes the hidden moderators. ‘When the pen study of Fritz Strack was replicated, the effect was not found in experiments where the study subjects were filmed. It might be that people's behaviour is different if they know that they are being filmed. You can test that with an experiment in which you compare the conditions with and without a camera. I would call that progress. Such an experiment has already been done, but it was not realised well. Just like other scientists, replication scientists make mistakes too.

Onderzoek 2019_2_herhaalonderzoek-1.jpg