Multiverse Analysis

‘Science progresses by reducing uncertainty. We assume that most of that uncertainty is from the world - the samples and circumstances we study. But, some of that uncertainty is from us - our decisions about how to analyze and draw inferences from data. Multiverse Analysis exposes the unrecognized uncertainty from analytic decisions and provides a systematic approach to incorporating it into the process of investigation and discovery. With richly described case examples, Young and Cumberworth provide a comprehensive philosophical and practical guide to understanding and using multiverse analysis. After reading this book, you will be much more expert in what we don't know, and what to do about it.’ Brian Nosek, Executive Director, Center for Open Science, Professor, University of Virginia

‘There is no deeper problem in empirical social science than establishing credible quantitative claims in light of their potential sensitivity to the various theoretical and statistical assumptions made by an analyst. In Multiverse Analysis, brilliant methodologists Cristobal Young and Erin Cumberworth develop a systematic methodology for exploring how empirical claims vary or remain robust across alternative assumptions. Every quantitative social scientist should study this important book.’ Steven Durlauf, Frank P. Hixon Distinguished Service Professor, University of Chicago and Director, Stone Center for Research on Wealth Inequality and Mobility

‘Young and Cumberworth blaze the trail to a future of more logical, transparent, and objective social science in this book. Multiverse Analysis gives us the modeling distribution - the variation in estimates across alternative modeling choices. The modeling distribution quantifies the uncertainty modeling choices add to results and identifies the choices with most leverage over a conclusion. This book will change how you think about statistical models and what they tell us about the social world.’ Mike Hout, NYU

‘‘The multiverse’ is less of a method than a way of thinking about choices in coding, analysis, and reporting. This new book works through a range of social-science examples to demonstrate how to use the multiverse to be open about uncertainty as a way to guide research and understanding, instead of the traditional ‘robustness study’ whose goal is to shield fragile results from criticism.’ Andrew Gelman, Department of Statistics and Department of Computer Science, Columbia University