A new study has found that research conclusions can vary dramatically depending on who is doing the analysing. The researchers from around the world observed significant variations in the findings of almost 500 independent analyses in relation to the same question using the same data in 100 studies. Although most of the reanalyses largely confirmed the main theses of the original studies, the effect sizes, statistical estimates and degrees of uncertainty often differed considerably. The study ‘Estimating the Analytic Robustness of Social and Behavioural Sciences (SCORE)’ was published in Nature.
In around a third of cases, all analysts came to the same conclusion as the original authors. The project delivers a clear message: Scientific objectivity is not about identifying a single ‘true’ analysis, but about making the space of plausible alternatives transparent – both in research reports and in communication with the wider scientific community. University of Cologne researchers from the Faculty of Management, Economics and Social Sciences and the Faculty of Human Sciences were involved, including Professor Dr Markus Weinmann from the Cologne Institute for Information Systems (CIIS) and Professor Dr Angela R. Dorrough from the Social Cognition Center Cologne (SoCCCo). Other researchers from the University of Cologne who were involved in the work are Dr Alexander Trinidad from the Department of Sociology and Social Psychology at the University of Cologne as well as Daria Lisovoj, Dr Marc Jekel and Carolin Häffner from SoCCCo.
The discrepancies do not result from a lack of expertise. Experienced researchers with sound statistical knowledge came to different conclusions just as often as others without this kind of experience. At the same time, observational studies proved to be less robust than experimental studies, suggesting that more complex data structures allow for greater analytical flexibility – and thus greater uncertainty.
Balázs Aczél, professor at Eötvös Loránd University, came to this conclusion: “These results do not call into question the credibility of previous research. Rather, they point out that the presentation of a single analysis often does not reflect the true extent of empirical uncertainty and that ignoring analytical variability can lead to unwarranted confidence in research conclusions.”
Barnabás Szászi, assistant professor at Eötvös Loránd University and Corvinus University, added: “We are in favour of a broader application of multi-analyst and ‘multiverse’ approaches, especially for issues of high scientific or societal importance. Instead of looking for a single true answer, these approaches reveal how stable – or fragile – research conclusions actually are.”
Professor Markus Weinmann from the Cologne Institute for Information Systems was one of the reanalysts and independently reanalysed several studies. For each assigned study, he analysed the original data using his own methods, estimated an effect size, and drew a conclusion. His findings were then compared with those of the other teams and the original effects. Weinmann explains: “The study shows that empirical findings in the social sciences are heavily dependent on analytical decisions made by researchers. Only 34 per cent of the independent reanalyses came to the same conclusion as the original study. 74 per cent drew the same qualitative conclusion. This does not mean that previous research is wrong, but it does show that the common practice of reporting only a single path of analysis underestimates the uncertainty in empirical findings. Multi-analyst designs and multi-verse analyses can provide more transparency here.”
The findings derive from a large-scale international collaboration led by Balázs Aczél and Barnabás Szászi (Eötvös Loránd University and Corvinus University), which was carried out as part of the ‘Systematizing Confidence in Open Research and Evidence’ (SCORE) programme. A team of 457 independent analysts from institutions around the world conducted 504 reanalyses of data from 100 previously published studies in the social and behavioural sciences. All analysts received the same data set and the same central research question, but were free to conduct the analysis as they saw fit.
In standard scientific practice, a dataset is usually analysed by a single researcher or a research team, and the resulting publication presents the outcome of a particular analysis pathway. While peer reviews assess methodological acceptability, they rarely show which results could have been achieved under alternative but equally defensible statistical choices. However, empirical research involves numerous decision-making points: how data is cleansed, how variables are defined, which statistical models or software are used and how results are interpreted.
Media Contact:
Professor Markus Weinmann
Cologne Institute for Information Systems (CIIS)
+49 221 470 89981
weinmann@wiso.uni-koeln.de
Press and Communications Team:
Robert Hahn
+49 221 470 2396
r.hahn(at)verw.uni-koeln(dot)de
Publication:
https://www.nature.com/articles/s41586-025-09844-9