I have been running controlled experiments for over 12 years now, at Microsoft, Amazon, and Blue Martini software, and have been involved in thousands of A/B tests. Hundreds of experiments run on our experimentation system at Microsoft on any given day, and I have written multiple peer-reviewed papers on online controlled experiments, available at exp-platform.
Chris Goward's book was a pleasure to read, containing many gems, fun quotations, and cartoons, making what could have been a boring read into something enjoyable.
It is clear that Chris and team at WiderFunnel have been involved in a lot of real scenarios, and share many real useful examples. They also give good references to resources. I enjoyed reading the book, and learned many new things.
Why four and not five stars? Here are some downsides
- The math/stats is not just lightly covered, but wrong in a few places. For example, "Data is collected until statistical significance is achieved" is a common theme throughout the book, which is misleading. An example from Section 5.1 (False Positives) of the paper "Online Controlled Experiments at Large Scale" shows that running five variants and testing them six times (e.g., each day the of the week the experiment runs as suggested) will result in over 50% chance of getting a stat-sig positive result with 95% conf intervals (1-0.975)^30 on an A/A test (when there is no true difference).
Statements like "reached statistical significance" imply that once it is achieved, you remain stat-sig, which is not true.
- The discussion of metrics is superficial and unclear. For example, "If you were to include multiple visits and pageviews for the same people, the conversion rate reported could be skewed too low." It depends what you are optimizing for and why. Are you looking to reduce the variance? Is it really the metric the business is optimizing?
- Some statements are extreme, such as "Surveys and studies should never use percentages as findings unless their results are
statistically significant." The percentages with confidence intervals could be informative.
- The selling of WiderFunnel to the reader could have been avoided and left as implied. For example, "If I were Walmart, I would hire WiderFunnel to test..."
- The LIFT model is a nice conceptual model, but making claims like "Each person has a conversion tipping point, and the six conversion factors are both independent and cumulative" is much too strong. Do we now have a perfect six-factor linear separator for human decision making in the context of conversions?
- No discussion on protecting the live site from egregious errors due to bad experiments. Chris writes
"Once you’ve launched your test, do yourself a favor and take a break from watching the results for a few days."
We operate in the opposite way: we watch the first hours of an
experiment carefully to detect egregious errors that hurt users in order to abort those experiments.
If you're thinking about doing A/B testing and want some motivating examples, this is a great book to read.
Download the free Kindle app and start reading Kindle books instantly on your smartphone, tablet, or computer - no Kindle device required. Learn more
Read instantly on your browser with Kindle Cloud Reader.
Using your mobile phone camera - scan the code below and download the Kindle app.
Loading your book clubs
There was a problem loading your book clubs. Please try again.
Not in a club? Learn more
Join or create book clubs
Choose books together
Track your books
Bring your club to Amazon Book Clubs, start a new book club and invite your friends to join, or find a club that’s right for you for free.
![You Should Test That: Conversion Optimization for More Leads, Sales and Profit or The Art and Science of Optimized Marketing by [Chris Goward]](https://m.media-amazon.com/images/I/51xJjTM6VGL._SX260_.jpg)
Follow the Author
Something went wrong. Please try your request again later.
Customers who viewed this item also viewed
Page 1 of 1 Start overPage 1 of 1
About the author
Follow authors to get new release updates, plus improved recommendations.
Customer reviews
Top reviews from the United States
There was a problem filtering reviews right now. Please try again later.
Reviewed in the United States 🇺🇸 on August 17, 2014
Reviewed in the United States 🇺🇸 on January 21, 2019
Reviewed in the United States 🇺🇸 on November 25, 2014
Reviewed in the United States 🇺🇸 on July 30, 2014
Reviewed in the United States 🇺🇸 on July 23, 2013
Reviewed in the United States 🇺🇸 on September 22, 2013
Top reviews from other countries
Paul Brown
Fantastic book which challenged my idea and delivered insights
Reviewed in the United Kingdom 🇬🇧 on February 13, 2020
@Travelbusy
Great introduction to CRO
Reviewed in the United Kingdom 🇬🇧 on April 12, 2015
Great introduction to CRO
Reviewed in the United Kingdom 🇬🇧 on April 12, 2015
Great introduction to CRO. Nicley laid out chapters based around his lift model. Book also has excellent case studies, giving real world examples of implementing the lift model.
Reviewed in the United Kingdom 🇬🇧 on April 12, 2015
Images in this review
geek73
Lots of practical steps and examples
Reviewed in the United Kingdom 🇬🇧 on February 12, 2017
