Manzi accomplished something remarkable with Uncontrolled. He put together an intellectually deep, lucid and constructive manifesto for how social sciences should help find "useful, reliable and non-obvious" solutions by drawing on epistemology, history, genetics, statistics, economics and business strategy - without ever losing its way in digressions, or forgoing clarity. Carefully and comprehensively, Manzi builds his argument into a powerful dictum, close to the traditions of Adam Smith and Friedrich Hayek: "The foundation of social progress is the unconscious advance of trial and error through social evolution." (p. 242). Unpacking this insight is what Uncontrolled is all about.
Manzi starts with the philosophical roots of induction in the writings of Francis Bacon and David Hume - and how they fared when Popper and Kuhn entered the picture. He describes the significant leap brought by three ideas. First, as the complexity of nature combines with humanity's pattern-seeking dispositions, experience and observation, the light in empiricism, become unreliable because we cast a shadow over them. Here is Manzi, quoting Bacon, on this: “the human understanding is of its own nature prone to suppose the existence of more order and regularity in the world than it finds.” (p. 6). Second, a departure from seeking ultimate causes towards searching practical, useful, knowledge fundamentally changed how progress in science is attained. Manzi summarizes this key point as a change in perspective: "The ultimate goal of Baconian science is not philosophical truth; it is improved engineering." (p. 8). And third, the mother of all induction problems: generalizability. As Hume noted, the reliability of inductions must necessarily be contingent, makeshift, as there will always be a chance that an undiscovered causal rule lurks behind whatever we gauged with our experience. These three ideas posit a challenge: "[…] without rules that generalize from experience, we have nothing more than a catalog of data, but inductive evidence can never tell us with certainty that our generalizations are correct." (p. 15). Enter experimental science.
Experimentation cuts through the three features that make our observations of nature so difficult: high causal density, that is, multiple causal relationships interacting in non-linear ways; holistic integration, how complex phenomena only arises, emerges, along the web of the interaction of a systems' components - but not in isolation; and motivated reasoning, as Bacon already foresaw. Statistician George Box put it sardonically: "To find out what happens when you change something, it is necessary to change it", and experimentation does precisely that. Since James Lind's 1747 allocation of treatments for scurvy in the British ship Salisbury, Pierce's 1884 randomized experiment on the perception of weights, and Neyman & Fisher's field randomization of fertilizer, experimentation has fundamentally changed how we accumulate knowledge. By accounting for white-coat biases that can inadvertently assign treatments to non-comparable groups, and holding approximately equal the unobserved differences in the members of control and treatment groups, randomization opens a window to causality - despite the complexity of the environment and the motivated reasoning of experimenters. Randomization, therefore, "[…] is another statement of epistemic humility." (p. 76).
The promise of experimentation, however, is not without its caveats. Immediately after tracing the history and importance of randomization, Manzi jumps to the criticisms and limitations underlying experiments - drawing on Heckman's 1991 paper "Randomization and social policy evaluation" and other sources. The problems abound. Most notably, the generalizability from experimentation just doesn’t do away with Hume's problem of induction: a single, well-run randomized experiment cannot convincingly show all hidden conditionals. Worse: the problem remains even after an experiment's replications. As Manzi asserts: "There is never an absolutely reliable probability distribution for external validity developed through replication." (p. 86). Even worse still: some phenomena, like macroeconomic shocks or epidemiological contagions, are not susceptible to experimentation and remain the arena of stories built around observational data. Experimentation can only tackle fundamentally local phenomena; econometrician Charles Manski (2011) puts it starkly: "A randomized experiment has no predictive power when interactions are global". So in areas like economics, as Manzi himself notes, "the maze of causation is now far beyond anything that physicists or biologists typically have had to address." (p. 102).
Yet there is room for experimentation somewhere else, somewhere where it can actually yield practical, tangible, benefits: business. Manzi is the founder of Applied Predictive Technologies (APT), a firm that streamlines experimentation using software in the retail sector. He describes how a culture of experimentation rose in banking, and later moved to other sectors, changing the landscape of business strategy: "Though individual companies will surely come and go, the experimental revolution is likely to become a permanent feature of the business landscape. " (p. 148), and it has, as companies like Google or Netflix run thousands of experiments to improve their services using basically the same methods Neyman and Fisher devised decades ago. Manzi's practitioner's experience hits home here, showing what has come out of years of the experimental work in business: "First, innovative ideas rarely work. Second, those that do work typically create improvements that are small compared to the size of the strategic issues they are intended to address, or as compared to the size of the dreams of those who invent them." (p. 166). If experimentation becomes more widespread, this lesson will become even more valuable in the areas where randomization strategies are incipient, and even more where they have been already oversold.
The fertile ground for experimentation in business is less so in the areas of public policy. Manzi notably summarizes the available randomized control trial (RCT) evidence in areas like crime, education, welfare, political science and economics in the U.S. as of 2011. From here he sets out a course of reform that puts his evolutionary epistemology to practice along existing U.S. institutions, suggesting how to experiment more by facilitating state waivers on federal policies; to introduce an agency similar to the FDA that would promote and standardize experimentation at different levels of government; using immigration as recruitment of human capital; unbundling welfare to distinguish the effects of education, healthcare, and other policies; among other prescriptions.
"We need freedom because we are ignorant" proclaims Manzi in the introduction of the book. And right he is.
Now, there is one aspect of the book that was disappointing. Uncontrolled misses by an inch the 5th star for one simple reason: the unbalanced coverage in discussing experiments in business vis-a-vis experiments in science and policy. The sections of the book on business experimentation cannot help but sound like a sales pitch, as the rigorous tone disappears when talking about business experimentation. This seems like a missed opportunity to show the results of business experimentation, and it’s a shame that the reader walks out from reading the book without a balanced grasp of experimentation in science and business alike.
Besides this, after 3 readings of the book I still find Uncontrolled inspired, erudite, and relevant. An impressive accomplishment.
Coda:
Uncontrolled deserves a follow-up or a second edition. In recent years, two important things happened that deserve to be seen through the lens of this book. First, the conversation on experimental social science has changed a lot since the book was written, as RCTs have become more and more frequent and influential, especially in development economics. Subjects like microfinance microfinance (see American Economic Journal: Applied Economics, Vol. 7, No. 1 January 2015) or how the poor allocate cash transfers (see Banerjee et. al. (2017) "Debunking the Stereotype of the Lazy Welfare Recipient: Evidence from Cash Transfer Programs Worldwide") have been redefined by RCTs. Quasi experimental evidence has also become more widespread, and natural experiments could also help bridge the observational/experimental divide - something the book doesn’t touch upon. Second, the replication crisis in some subfields of experimental psychology, mostly (and fortunately) irrelevant outside of academia and the pop-psychology book market, has exposed big methodological and sociological problems in experimentation, and I for one would like to see these problems filtered through Manzi's clarifying lens.
- Amazon Business : For business-only pricing, quantity discounts and FREE Shipping. Register a free business account
