Understanding Statistical Biases in Peer Review

Summary

Peer review plays a central role in science, and so its proper, unbiased operation is of the highest importance. Reviewers, editors, and others involved in peer review may be biased against scientists because of their social identity (e.g., scientists’ gender or race), or against projects because of their methodological approach. A lot of research has been devoted to measuring such “individual biases” and to potential remedies.
However, my work on probabilistic models of peer review reveals that there is another source of bias that has not yet received significant attention. Statistical biases arise when the uncertainty involved in peer review affects groups at differential rates, regardless of whether or not reviewers or editors are (individually) biased. This may happen when this group’s work is systematically harder to assess. Remedies that are effective against individual biases may be ineffective or counterproductive against statistical biases. Simultaneously, statistical biases are most likely to negatively affect those groups already disadvantaged through individual biases and related social dynamics.
My project has two aims. First, it will establish the concept of statistical biases and identify the underlying mechanisms. This is done by representing peer review in a probabilistic model, and showing how peer review can be biased even when the individuals involved are unbiased. Second, it will measure statistical biases. This is done through statistical analysis of existing data on peer review. In doing so, the project brings together techniques from philosophy (especially formal and social epistemology), statistics, sociology, and economics.
At the conclusion of this project, specific statistical mechanisms will have been identified as potential sources of biases in peer review. This complements our understanding of the sources of bias, thus making possible more targeted interventions aimed at allowing peer review to perform its role in the social structure of science more effectively.

Details

Project number

016.Veni.195.141

Main applicant

Dr. R. Heesen

Affiliated with

Rijksuniversiteit Groningen

Team members

Dr. R. Heesen

Duration

01/02/2019 to 31/12/2021