Review
"At the core of the Lean Methodology is the scientific method: Creating hypotheses, running experiments, gathering data, extracting insight and validation or modification of the hypothesis. A/B testing is the gold standard of creating verifiable and repeatable experiments, and this book is its definitive text" -- Steve Blank, Adjunct professor at Stanford University, co-founded of multiple startups, father of modern entrepreneurship, author of The Startup Owner's Manual and The Four Steps to the Epiphany
"This book is a great resource for executives, leaders, researchers or engineers looking to use online controlled experiments to optimize product features, project efficiency or revenue. I know firsthand the impact that Ronny's work had on Bing and Microsoft, and I'm excited that these learnings can now reach a wider audience" -- Harry Shum, Executive Vice President, Microsoft Artificial Intelligence and Research Group
"A great book that is both rigorous and accessible. Readers will learn how to bring trustworthy controlled experiments, which have revolutionized internet product development, to their organizations" -- Adam D'Angelo, Co-founder and CEO of Quora and prior CTO of Facebook
"This book is a great overview of how several companies use online experimentation and A/B testing to improve their products. Kohavi, Tang and Xu have a wealth of experience and excellent advice to convey, so the book has lots of practical real world examples and lessons learned over many years of the application of these techniques at scale." -- Jeff Dean, Google Senior Fellow, and SVP, Google Research
"The secret sauce for a successful online business is experimentation. But it is a secret no longer. Here three masters of the art describe the ABCs of A/B testing so that you too can continuously improve your online services." -- Hal Varian, Chief Economist, Google, and author of Intermediate Microeconomics: A Modern Approach
"Do you want your organization to make consistently better decisions? This is the new bible of how to get from data to decisions in the digital age. Reading this book is like sitting in meetings inside Amazon, Google, LinkedIn, Microsoft. The authors expose for the first time the way the world's most successful companies make decisions. Beyond the admonitions and anecdotes of normal business books, this book shows what to do and how to do it well. It's the how-to manual for decision-making in the digital world, with dedicated sections written for business leaders, engineers, and data analysts." -- Scott Cook, Intuit Co-founder & Chairman of the executive committee.
"Online controlled experiments are powerful tools. Understanding how they work, what their strengths are, and how they can be optimized can illuminate both specialists and a wider audience. This book has the rare combination of being technically authoritative, enjoyable to read, and dealing with highly important matters" -- John P.A. Ioannidis, professor of Medicine, Health Research and Policy, Biomedical Data Science, and Statistics at Stanford University; he is one of the most-cited researchers across the scientific literature.
"Experiments are the best tool for online products and services. This book is full of practical knowledge derived from years of successful testing at Microsoft, Google and LinkedIn. Insights and best practices are explained with real examples, and pitfalls, their markers and solutions identified. I strongly recommend this book!" -- Preston McAfee, former Chief Economist and VP of Microsoft
"Which online option will be better? We frequently need to make such choices, and frequently err. To determine what will actually work better, we need rigorous controlled experiments, aka A/B testing. This excellent and lively book by experts from Microsoft, Google, and LinkedIn presents the theory and best practices of A/B testing. Must read for anyone who does anything online!" -- Gregory Piatetsky-Shapiro, Ph.D. is the president of KDnuggets, co-founder of KDD conference and SIGKDD, and LinkedIn Top Voice on Data Science & Analytics.
"In the past two decades the technology industry has learned what scientists have known for centuries: that controlled experiments are among the best tools to understand complex phenomena and to solve very challenging problems. The ability to design controlled experiments, run them at scale, and interpret their results is the foundation of how modern high tech businesses operate. Diane, Ya, and Ronny are three of the world's experts on the subject and between them have designed and implemented several of the world's most powerful experimentation platforms. This book is a great opportunity to learn from their experiences about how to use these tools and techniques." -- Kevin Scott, executive vice president and CTO of Microsoft
"Online experiments have fueled the success of Amazon, Microsoft, LinkedIn and other leading digital companies. This practical book gives the reader rare access to decades of experimentation experience at these companies and should be on the bookshelf of every data scientist, software engineer and product manager." -- Stefan Thomke, William Barclay Harding Professor, Harvard Business School, Author of Experimentation Works: The Surprising Power of Business Experiments
"A modern software-supported business cannot compete successfully without online controlled experimentation. Written by three of the most experienced leaders in the field, this book presents the fundamental principles, illustrates them with compelling examples, and digs deeper to present a wealth of practical advice. It's a 'must read'. -- Foster Provost, Professor at NYU Stern School of Business & co-author of the best-selling Data Science for Business.
"Ron Kohavi, Diane Tang and Ya Xu are the world's top experts on online experiments. I've been using their work for years and I'm delighted they have now teamed up to write the definitive guide. I recommend this book to all my students and everyone involved in online products and services." -- Erik Brynjolfsson, Professor at MIT and Co-Author of The Second Machine Age
"Experimentation is the future of digital strategy and 'Trustworthy Experiments' will be its Bible. Kohavi, Tang and Xu are three of the most noteworthy experts on experimentation working today and their book delivers a truly practical roadmap for digital experimentation that is useful right out of the box. The revealing case studies they conducted over many decades at Microsoft, Amazon, Google and LinkedIn are organized into easy to understand practical lessens with tremendous depth and clarity. It should be required reading for any manager of a digital business." -- Sinan Aral, David Austin Professor of Management, MIT and author of the forthcoming book 'The Hype Machine.'
"Indispensable book for any serious experimentation practitioner. This book is highly practical and goes in-depth like I've never seen before. It's so useful it's feels like you get a superpower. From statistical nuances to evaluating outcomes to measuring long term impact, this book has got you covered. Must-read." -- Peep Laja, Founder and Principal of CXL, and top conversion rate optimization expert
"Online experimentation was critical to changing the culture at Microsoft. When Satya talks about 'Growth Mindset,'
experimentation is the best way to try new ideas and learn from them. Learning to quickly iterate controlled experiments drove Bing to profitability, and rapidly spread across Microsoft through Office, Windows, and Azure." -- Eric Boyd, Corporate VP, AI Platform, Microsoft
"As an entrepreneur, scientist, and executive I've learned (the hard way) that an ounce of data is worth a pound of my intuition. But how to get good data? This book compiles decades of experience at Amazon, Google, LinkedIn, and Microsoft into an accessible, well-organized guide. It is the bible of online experiments." -- Oren Etzioni, CEO of Allen Institute of AI and Professor of Computer Science at University of Washington
"Internet companies have taken experimentation to an unprecedented scale, pace, and sophistication. These authors have played key roles in these developments and readers are fortunate to be able to learn from their combined experiences.". -- Dean Eckles, Associate Professor at Massachusetts Institute of Technology and former scientist at Facebook.
"A wonderfully rich resource for a critical but under-appreciated area. There are many real case studies in every chapter showing the inner workings and learnings of successful businesses. The focus on developing and optimizing an 'Overall Evaluation Criterion' (OEC) is a particularly important lesson." -- Jeremy Howard, founder of fast.ai, faculty at Singularity University, and former president and chief scientist of Kaggle
"There are many guides to A/B Testing, but few with the pedigree of Trustworthy Online Controlled Experiments. I've been following Ronny Kohavi for eighteen years and find his advice to be steeped in practice, honed by experience, and tempered by doing laboratory work in real world environments. When you add Diane Tang, Ya Xu to the mix, the breadth of comprehension is unparalleled. I challenge you to compare this tome to any other - in a controlled manner, of course." -- Jim Sterne, Founder of Marketing Analytics Summit and Director Emeritus of the Digital Analytics Association
"An extremely useful how-to book for running online experiments that combines analytical sophistication, clear exposition and the hard-won lessons of practical experience." -- Jim Manzi, Founder of Foundry.ai, Prior founder, CEO and Chairman of Applied Predictive Technologies (acquired by Mastercard), and author of Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society
"Experimental design advances each time it is applied to a new domain: agriculture, chemistry, medicine and now online electronic commerce. This book by three top experts is rich in practical advice and examples covering both how and why to experiment online and not get fooled. Experiments can be expensive; not knowing what works can cost even more." -- Art Owen, professor of Statistics at Stanford University, where among other things, he teaches experimental design.
"This is a must read book for business executives and operating managers. Just as operations, finance, accounting and strategy form the basic building blocks for business, today in the age of artificial intelligence, understanding and executing online controlled experiments will be a required knowledge set. Kohavi, Tang and Xu have laid out the essentials of this new and important knowledge domain that is practically accessible." -- Karim R. Lakhani, Professor and Director of Laboratory for Innovation Science at Harvard, Board Member, Mozilla Corporation
"Serious 'data-driven' organizations understand that analytics aren't enough; they must commit to experiment. Remarkably accessible and accessibly remarkable, this book is a manual and manifesto for high-impact experimental design. I found its pragmatism inspirational. Most importantly, it clarifies how culture rivals technical competence as a critical success factor." -- Michael Schrage, research fellow at MIT's Initiative on the Digital Economy and author of 'The Innovator's Hypothesis: How Cheap Experiments Are Worth More than Good Ideas'
"This important book on experimentation distills the wisdom of three distinguished leaders from some of the world's biggest technology companies. If you are a software engineer, data scientist, or product manager trying to implement a data-driven culture within your organization, this is an excellent and practical book for you." -- Daniel Tunkelang, former Director of Data Science and Engineering at LinkedIn, Tech Lead at Google, and Chief Scientist at Endeca.
"With every industry becoming digitized and data-driven, conducting and benefiting from controlled online experiments becomes a required skill. Kohavi, Tang and Yu provide a complete and well-researched guide that will become necessary reading for data practitioners and executives alike." -- Evangelos Simoudis, Co-founder and Managing Director Synapse Partners; author of The Big Data Opportunity in Our Driverless Future
"Ronny, Ya and Diane offer over 10 years of hard-fought lessons in experimentation, from beginner rules of thumb to advanced topics, in the most strategic book for the discipline yet." --- Colin McFarland, Director Experimentation Platform at Netflix
"The practical guide to A/B testing distills the experiences from three of the top minds in experimentation practice into easy and digestible chunks of valuable and practical concepts. Each chapter walks you through some of the most important considerations when running experiments--from choosing the right metric to the benefits of institutional memory. If you are looking for an experimentation coach that balances science and practicality, then this book is for you." -- Dylan Lewis, Experimentation Leader, Intuit
"The only thing worse than no experiment is a misleading one, because it gives you false confidence! This book details the technical aspects of testing based on insights from some of the world's largest testing programs. If you're involved in online experimentation in any capacity, read it now to avoid mistakes and gain confidence in your results." -- Chris Goward, Author of You Should Test That!, Founder and CEO of Widerfunnel
"This is a phenomenal book. The authors draw on a wealth of experience and have produced a readable reference that is somehow both comprehensive and detailed at the same time. Highly recommended reading for anyone who wants to run serious digital experiments." -- Pete Koomen, Co-founder, Optimizely
"Ronny, Ya, and Diane are pioneers of online experimentation. The platforms they've built and the experiments they've enabled have transformed some of the largest internet brands. Their research and talks have inspired teams across the industry to adopt experimentation. This book is the authoritative yet practical text that the industry has been waiting for." -- Adil Aijaz, Co-founder and CEO, Split Software.
Book Description
This practical guide for students, researchers and practitioners offers real world guidance for data-driven decision making and innovation.
From the Author
Additional information, including online chapters, errata, and FAQ is available at experimentguide.com .
The book has been translated to Chinese and Japanese.
About the Author
Ron Kohavi is a VP and Technical Fellow at Airbnb. He was previously a Technical Fellow and Corporate VP at Microsoft. Prior to Microsoft, he was the director of data mining and personalization at Amazon.com. He has a PhD in Computer Science for Stanford University. His papers have over 40,000 citations and three of his papers are in the top 1,000 most-cited papers in Computer Science.
Diane Tang is a Google Fellow, with expertise in large-scale data analysis and infrastructure, online controlled experiments, and ads systems. She has an AB from Harvard and MS/PhD from Stanford, and has patents and publications in mobile networking, information visualization, experiment methodology, data infrastructure, and data mining / large data.
Ya Xu heads Data Science and Experimentation at LinkedIn. She has led LinkedIn to become one of the most well-regarded companies when it comes to A/B testing. Before LinkedIn, she worked at Microsoft and received a PhD in Statistics from Stanford University. She is widely regarded as one of the premier scientists, practitioners and thought leaders in the domain of experimentation, with several filed patents and publications. She is also a frequent speaker at top conferences, universities and companies across the country.
