Fiona Fidler – Can forecasts of replicability improve peer review? – PoSE 2024

Fiona Fidler – Can forecasts of replicability improve peer review? – Perspectives on Scientific Error 2024 https://osf.io/ayfek/ Now in its sixth year, the repliCATS project (Collaborative Assessment for Trustworthy Science) has evaluated over 4,000 published social science articles across 8 disciplines, including psychology, economics, and education, as well as many preprints. For each paper, a diverse group of experts forecasts the likely replicability of the research findings and makes a variety of other judgements about the credibility of the evidence presented using a structured deliberation protocol. This talk will present our approach to evaluating research, and for cases where we have the outcome of actual replication studies, data about the accuracy of our forecasts. I will also discuss how structured expert elicitation, deliberation, and decision protocols like those used in repliCATS might improve peer review more generally.