Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society 1st Edition

This (another jewel from the Basic Books publishing house) is written by an actual businessman/ engineer type and is three-in-one:

1. Science;
2. Social Science;
3. Political Action

And even within the minibooks, each chapter reads a bit more like a freestanding essay. They are of such a quality that the mediocre final third of the book does not detract that much from the book's overall rating.

I can imagine this book from the perspective of three people:

1. A Scientist-- For him, it will be an easy read and a lot of things that we all know in one place. He will take a few new things from the book, but most of what he enjoys will be seeing the author lay out the definition and history of science in one place;
2. Intelligent Public-- For a person who is reasonably bright, this book might give an idea of why social sciences are not in the same category as physical sciences. And exactly why complex, multivariable, nonlinear systems (environmental/ political/ social) do not lend themselves to prediction. The third chapter (Implicit and Explicit Knowledge) will be the hardest chapter of the book to read. The book overall is something that will need to be gone over line by line;
3. Not-so-intelligent Public-- This will be something that they will never understand. So, a religious environmentalist will not be able to make the connection between the difficulty in predicting the behavior of the environment and being very cautious in making social policy about something that he does not understand.

There are a lot of themes that come up in this book that come up in other popular books;

1. Epistemic humility (p. 194) and "the model is not the system" (p. 127) are discussed by Nassim Taleb in many of his books (one of which is quoted in this book);
2. Policy makers' ignorance (p. 195) was discussed beautifully in Easterley's The Elusive Quest for Growth: Economists' Adventures and Misadventures in the Tropics
3. Federalism vs. Subsidiarity (p. 213). (The Death of Money: The Coming Collapse of the International Monetary System)
4. Liberartarianism does not mean unlimited immigration and being a dumping ground for low skilled people. (p.
5. Economists don't know much of anything that is experimentally valid because something like 95% of their papers are NON experimental. In order to find something that works, you have to roll up your sleeves and get about the business of finding what DOESN'T work. (Adapt: Why Success Always Starts with Failure).
6. The incremental/ experimental nature of knowledge. The fact that the overwhelming majority of experiments will fail ("knowledge is rare," from the very first page of the Sowell book). And knowledge as a social technology (Knowledge And Decisions).

The theme that the author hammers home is that: Given that we don't know what works, it is best to have a preference for the status quo. And that is because the VAST MAJORITY of ideas will fail. Given that the author has actual experience doing actual projects and trying to make them work, his experience is something that I can appreciate and believe.

Verdict: One cannot fail to find at least something good from this book. Worth the secondhand price. Worth the time. Recommended.

Manzi accomplished something remarkable with Uncontrolled. He put together an intellectually deep, lucid and constructive manifesto for how social sciences should help find "useful, reliable and non-obvious" solutions by drawing on epistemology, history, genetics, statistics, economics and business strategy - without ever losing its way in digressions, or forgoing clarity. Carefully and comprehensively, Manzi builds his argument into a powerful dictum, close to the traditions of Adam Smith and Friedrich Hayek: "The foundation of social progress is the unconscious advance of trial and error through social evolution." (p. 242). Unpacking this insight is what Uncontrolled is all about.

Manzi starts with the philosophical roots of induction in the writings of Francis Bacon and David Hume - and how they fared when Popper and Kuhn entered the picture. He describes the significant leap brought by three ideas. First, as the complexity of nature combines with humanity's pattern-seeking dispositions, experience and observation, the light in empiricism, become unreliable because we cast a shadow over them. Here is Manzi, quoting Bacon, on this: “the human understanding is of its own nature prone to suppose the existence of more order and regularity in the world than it finds.” (p. 6). Second, a departure from seeking ultimate causes towards searching practical, useful, knowledge fundamentally changed how progress in science is attained. Manzi summarizes this key point as a change in perspective: "The ultimate goal of Baconian science is not philosophical truth; it is improved engineering." (p. 8). And third, the mother of all induction problems: generalizability. As Hume noted, the reliability of inductions must necessarily be contingent, makeshift, as there will always be a chance that an undiscovered causal rule lurks behind whatever we gauged with our experience. These three ideas posit a challenge: "[…] without rules that generalize from experience, we have nothing more than a catalog of data, but inductive evidence can never tell us with certainty that our generalizations are correct." (p. 15). Enter experimental science.

Experimentation cuts through the three features that make our observations of nature so difficult: high causal density, that is, multiple causal relationships interacting in non-linear ways; holistic integration, how complex phenomena only arises, emerges, along the web of the interaction of a systems' components - but not in isolation; and motivated reasoning, as Bacon already foresaw. Statistician George Box put it sardonically: "To find out what happens when you change something, it is necessary to change it", and experimentation does precisely that. Since James Lind's 1747 allocation of treatments for scurvy in the British ship Salisbury, Pierce's 1884 randomized experiment on the perception of weights, and Neyman & Fisher's field randomization of fertilizer, experimentation has fundamentally changed how we accumulate knowledge. By accounting for white-coat biases that can inadvertently assign treatments to non-comparable groups, and holding approximately equal the unobserved differences in the members of control and treatment groups, randomization opens a window to causality - despite the complexity of the environment and the motivated reasoning of experimenters. Randomization, therefore, "[…] is another statement of epistemic humility." (p. 76).

The promise of experimentation, however, is not without its caveats. Immediately after tracing the history and importance of randomization, Manzi jumps to the criticisms and limitations underlying experiments - drawing on Heckman's 1991 paper "Randomization and social policy evaluation" and other sources. The problems abound. Most notably, the generalizability from experimentation just doesn’t do away with Hume's problem of induction: a single, well-run randomized experiment cannot convincingly show all hidden conditionals. Worse: the problem remains even after an experiment's replications. As Manzi asserts: "There is never an absolutely reliable probability distribution for external validity developed through replication." (p. 86). Even worse still: some phenomena, like macroeconomic shocks or epidemiological contagions, are not susceptible to experimentation and remain the arena of stories built around observational data. Experimentation can only tackle fundamentally local phenomena; econometrician Charles Manski (2011) puts it starkly: "A randomized experiment has no predictive power when interactions are global". So in areas like economics, as Manzi himself notes, "the maze of causation is now far beyond anything that physicists or biologists typically have had to address." (p. 102).

Yet there is room for experimentation somewhere else, somewhere where it can actually yield practical, tangible, benefits: business. Manzi is the founder of Applied Predictive Technologies (APT), a firm that streamlines experimentation using software in the retail sector. He describes how a culture of experimentation rose in banking, and later moved to other sectors, changing the landscape of business strategy: "Though individual companies will surely come and go, the experimental revolution is likely to become a permanent feature of the business landscape. " (p. 148), and it has, as companies like Google or Netflix run thousands of experiments to improve their services using basically the same methods Neyman and Fisher devised decades ago. Manzi's practitioner's experience hits home here, showing what has come out of years of the experimental work in business: "First, innovative ideas rarely work. Second, those that do work typically create improvements that are small compared to the size of the strategic issues they are intended to address, or as compared to the size of the dreams of those who invent them." (p. 166). If experimentation becomes more widespread, this lesson will become even more valuable in the areas where randomization strategies are incipient, and even more where they have been already oversold.

The fertile ground for experimentation in business is less so in the areas of public policy. Manzi notably summarizes the available randomized control trial (RCT) evidence in areas like crime, education, welfare, political science and economics in the U.S. as of 2011. From here he sets out a course of reform that puts his evolutionary epistemology to practice along existing U.S. institutions, suggesting how to experiment more by facilitating state waivers on federal policies; to introduce an agency similar to the FDA that would promote and standardize experimentation at different levels of government; using immigration as recruitment of human capital; unbundling welfare to distinguish the effects of education, healthcare, and other policies; among other prescriptions.

"We need freedom because we are ignorant" proclaims Manzi in the introduction of the book. And right he is.

Now, there is one aspect of the book that was disappointing. Uncontrolled misses by an inch the 5th star for one simple reason: the unbalanced coverage in discussing experiments in business vis-a-vis experiments in science and policy. The sections of the book on business experimentation cannot help but sound like a sales pitch, as the rigorous tone disappears when talking about business experimentation. This seems like a missed opportunity to show the results of business experimentation, and it’s a shame that the reader walks out from reading the book without a balanced grasp of experimentation in science and business alike.

Besides this, after 3 readings of the book I still find Uncontrolled inspired, erudite, and relevant. An impressive accomplishment.

Coda:
Uncontrolled deserves a follow-up or a second edition. In recent years, two important things happened that deserve to be seen through the lens of this book. First, the conversation on experimental social science has changed a lot since the book was written, as RCTs have become more and more frequent and influential, especially in development economics. Subjects like microfinance microfinance (see American Economic Journal: Applied Economics, Vol. 7, No. 1 January 2015) or how the poor allocate cash transfers (see Banerjee et. al. (2017) "Debunking the Stereotype of the Lazy Welfare Recipient: Evidence from Cash Transfer Programs Worldwide") have been redefined by RCTs. Quasi experimental evidence has also become more widespread, and natural experiments could also help bridge the observational/experimental divide - something the book doesn’t touch upon. Second, the replication crisis in some subfields of experimental psychology, mostly (and fortunately) irrelevant outside of academia and the pop-psychology book market, has exposed big methodological and sociological problems in experimentation, and I for one would like to see these problems filtered through Manzi's clarifying lens.