Smart statistics: lots of information from little data

Case

Smart statistics: lots of information from little data

How small samples lead to a robust outcome

Drawing meaningful conclusions from a small quantity of data is statistically speaking exceptionally difficult. How can you reliably compare small groups with each other? Rens van de Schoot and his colleagues from Utrecht University have specialised in that field. They're coming up with new methods for the social sciences. ‘The challenge is posing the right questions and then developing reliable tools with which others can set to work.’

Rens van de SchootRens van de Schoot

How can you determine whether the differences you find are down to chance or not? Statisticians prefer to work with thousands of study subjects or with datasets containing thousands of measurement points. Sometimes there are simply too few study subjects or limited data. That is the case for rare physical disorders. How can you still count on such data and make meaningful statements about these? That is a specialisation of statistician Rens van de Schoot, Professor of Solutions for Small Datasets at Utrecht University and member of the Young Academy of the KNAW. ‘The basis for my research is a Vidi grant’, says Van de Schoot, ‘supplemented with grants for my PhDs, for example from the NWO programme Research Talent.

Early recognition

We urgently need this type of research, emphasises Van de Schoot. ‘There are numerous questions that we would like to have answers to as a society, but these are difficult to investigate because there are so few data. Take, for example, research into post-traumatic stress among parents of young children with severe burns. Fortunately, we know very few such cases! Nevertheless, there are important things that we would like to know. For example, which type of parents are most vulnerable to post-traumatic stress? As soon as you know that, then you can recognise those parents at an early stage and provide them with more targeted support.’

we obtain that information from experts: people who are daily confronted on the work floor with the phenomenon we are investigating
- Rens van de Schoot

Can researchers pool data from different countries to obtain a larger research group? ‘That is possible, as long as you have comparable countries in terms of language, for example the Netherlands and Belgium. However, for many studies that causes problems because we study concepts that can have a different meaning in different countries, the so-called measurement invariance. You cannot simply combine datasets of things that are interpreted slightly differently.’ That is very often the case in the social sciences emphasises Van de Schoot – unlike raw biomedical data such as blood values or carbonate.

Gut feeling

Van de Schoot works with a special branch of statistics, known as Bayesian statistics. This is particularly suitable for small datasets. Bayesian statistics uses observations and combines these with so-called priors: expectations based on existing knowledge, experiences or publications. ‘In our research, we obtain that information from experts’, says Van de Schoot. ‘Not from researchers, but from people who are daily confronted on the work floor with the phenomenon we are investigating. We try to pour their experiences, their gut feelings, in a reliable manner into a statistical model; that is called expert elicitation.’

The challenge is not just to choose the right experts, but also ask them the right questions. ‘There is a risk that, as a researcher, you are biased and therefore only obtain the answers that fit with what you already expect. You can solve the first challenge by involving as many experts as possible in the research, and you can solve the second challenge by posing smart questions. For example, in the case of the children with burns, we ask the nurses: how do you think the stress levels of these children have developed over the course of time? And how certain or uncertain are you of that?

Robust

You can fit both of those measures in the model, explains Van de Schoot. ‘And then you check in retrospect how robust your outcomes are without adjusting the original settings’, he says. ‘We refer to that as sensitivity analysis. You run your model several times, and then you fine tune it. What happens if I vary this factor? If a minimal change has a large effect on the final outcome, then the model is not that robust and perhaps we should not attach too much value to the outcome.’
With this approach, Van de Schoot and his team have been able to develop a model for post-traumatic stress among parents of children with severe burns. ‘We have developed a wide range of statistical measures and tricks’, he says, ‘based on relatively large number of experts and lots of information. We can now indeed argue that the model is robust.’

Quiet square with few peopleFew test subjects, robust conclusions. Photo Shutterstock

Smart disclosure

Based on the type of analysis can you also select which experts to include in your research and which not? ‘No, because that is, in fact, cherry-picking. And you run the risk of only involving the experts who confirm your hypothesis. However, a realistic option is to calibrate experts by testing in advance how reliable they are, for example based on standard questions to which an expert in that discipline should really know the answer. That already happens in the natural sciences, but how do you do that among psychologists? Which questions should you pose them? That is very difficult and something we would like to work on. That should be fun!’

Did you know? With 'machine learning' you can learn a computer program to recognize relevant information, so that the researcher does not have to read through thousands of texts.

An entirely different approach is to analyse what has already been written about a certain subject, for example in scientific publications or in court reports. ‘We are working on methods to smartly unlock that information by using machine learning techniques. Then you teach a computer program to recognise and select relevant information. That means you no longer need to read through thousands of texts as a researcher.’

Predicting

Recently, the statisticians from Utrecht were in the Dutch news due to the secondary school advice issued to primary school pupils. Each pupil receives two recommendations: one based on the result of a central final test, such as the Cito test, and one based on the opinion of their teacher. Since 2015, the final test has been guiding for the secondary school advice. Van de Schoot and his colleagues advise something else, however. ‘We want to know which advice best predicts the school level of the pupil three years later’, says Van de Schoot. ‘Our research revealed that both recommendations give a slightly different picture. The central final test best predicts which pupils will achieve pre-university (vwo) education level and the teachers have the best view on which pupils will drop to the preparatory vocational education (vmbo) level. We think that you can best merge the two sources into a single reliable advice. We are now busy working on that together with Cito.’
The next step is to analyse which teachers with their prediction come closest to the data.

Never finished

Using statistics to solve societal questions is Van de Schoot’s ambition. ‘We do not work on the actual societal questions, but on the building blocks needed to solve these’, he emphasises. ‘We use real-life cases, but our product is the methodology. The toolkit with which researchers in a wide range of other disciplines will soon be able to analyse their own data.’

The Vidi research of Van de Schoot is nearing its completion; it has about one more year funding. Is his research almost finished? The statistician laughs. ‘It is never finished’, he answers. ‘The next phase is that we will participate in grant applications of other researchers to deploy our toolkit in far more types of research. Then we can see: does it actually work in practice? Or have we failed to see something? Which adaptations are needed in which situations? That work will always carry on.’

More information

Photo in the banner: Shutterstock