Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are Kindle Edition

The basic thesis of "Everybody Lies" is that online data on human behavior, including Google searches and data from Facebook, shopping and pornographic sites, can reveal much about what we really think than data from surveys in which people might be too embarrassed to tell the truth. In our unguarded moments, when we are alone and searching Google in the privacy of our homes, we are much more likely to divulge our innermost desires. The premise is that truly understanding human behavior by way of psychology or neuroscience is too complicated right now, so it's much better to simply bypass that kind of understanding and look at what the numbers are telling us in terms of what people's online behavior. In doing this the author looks at a remarkable variety of online sources and studies by leading researchers, and one must congratulate him for the diversity and depth of material he has plumbed.

What has allowed us to access this pool of unguarded opinions and truckloads of data concerning human behavior is the Internet and the tools of "big" data. As the author puts it, this data is not just "big" but also "new", which means that the kind of data we can access is also quite different from what we are used to; in his words, we live in a world where every sneeze, cough, internet purchase, political opinion, and evening run can be considered "data". This makes it possible to test hypotheses that we could not have tested before. For instance, the author gives the example of testing Freud's Oedipus Complex through accessing pornographic data which indicates a measurable interest in incest. Generally speaking there is quite an emphasis on exploring human sexuality in the book, partly because sexuality is one of those aspects of our life that we wish to hide the most and are also pruriently interested in, and partly because investigating this data through Google searches and pornographic sites reveals some rather bizarre sexual preference that are also sometimes specific to one country or another. This is a somewhat fun use of data mining.

Data exploration can both reveal the obvious as well as throw up unexpected observations. A more serious use of data tools concerns political opinions. Based on Google searches in particular states, the author shows how racism (as indicated by racist Google searches) was a primary indicator of which states voted for Obama in the 2008 election and Trump in the 2016 election. That's possibly an obvious conclusion, at least in retrospect. A more counterintuitive conclusion is that the racism divide does not seem to map neatly on the urban-rural divide or the North-South divide, but rather on the East-West divide; people seem to be searching much more for explicitly racist things in the East compared to the West. There is also an interesting survey of gay people in more and less tolerant states which concludes that you are as likely to find gay people in both parts of the country. Another interesting section of the book talked about how calls for peace by politicians after terrorist attacks actually lead to more rather than less xenophobic Google searches; this is accompanied by a section that hints at how the trends can be potentially reversed if different words are used in political speeches. There is also an interesting discussion of how the belief that newspaper political leanings drive customer political preferences gets it exactly backward; the data shows that customer political preferences shape what newspapers print, so effectively they are doing nothing different from any other customer-focused, profit making organization.

The primary tool for doing all this data analysis is correlation or regression analysis, where you look at online searches and try to find correlations between certain terms and factors like geographic location, gender, ethnicity. One hopes that one has separated the most important correlated variable and has eliminated other potentially important ones.

There are tons of other amusing and informative studies - sometimes the author's own but more often other people's - that reveal human desires and behavior across a wide swathe of fields, including politics, dating, sports, education, shopping and sexuality. There's plenty of potentially useful material in these studies. For instance, some of the data that indicates gaps in educational or social attainment in different parts of the country are immediately actionable in principle. Google searches have also been used to keep track of flu and other disease epidemics. Sometimes finding correlations is financially lucrative; there is a story about how a horse expert found that success in horse races seems to correlate with one factor more than any other: the size of the left ventricle. Another study isolated the impact of the early growing season on the quality of wines. There is no doubt that financial firms, supermarkets, newspapers, hospitals and online purveyors of everything from pornography to peanuts are going to keep a close eye on this data to maximize their reach and profits.

Generally speaking I enjoyed "Everybody Lies"; for the scope of the material, the easy-going style and some of the counterintuitive observations it reveals. My main reservation about the book is that I think the author overstates his case and sometimes sounds a little too breathless about the great changes these tools are going to bring. More than once he uses the term "revolutionary" in describing these data tools, but I am much more suspicious of their ultimate utility. Firstly, data does not equal knowledge; rather, it is the raw material for knowledge. As the author himself acknowledges, understanding correlation is not the same as understanding causation, and it's in very few cases that a true causal relationship between people's Google searches and their true nature can be established. Part of the reason I think this way is because I don't believe that a person's Google search is as reflective of their innermost desires as the book seems to think, so what a person truly believes may go way beyond their online behavior. Consider the studies revealing people's sexual preferences for instance; how many of them point to trivial idiosyncrasies and how many are indicative of some deeper truth about human brains? The tools alone cannot draw this distinction. At the end of the day you could thus end up with a lot of data (including a lot of noise), but teasing apart the useful data points from the red herrings is a completely different matter. In this sense, looking at Google searches and other information can be a reductionist and simplistic approach.

Secondly, it's usually quite hard to control for all possible variables that may reflect a Google search; for instance in concluding that racism contributes the most to a particular political behavior, it's very hard to tease out all other factors that also may do so, especially when you are talking about a heterogeneous collection of human beings. How can you know that you have corrected for every possible factor? Thirdly and finally, the "science" part of "data science" still lacks rigor in my opinion. For instance, a lot of the conclusions the book talks about are based on single studies which don't seem to be repeated. In some cases the sample sizes are large, but in other cases they are small. Plus, people's opinions can change over time, so it's important to pick the right time window in which to do the study. All this points to great responsibility on the part of data scientists to make sure that their results are rigorous and not too simplistic, before they are taken up by both politicians and the general public as blunt instruments to change social policies. This responsibility increases especially as these approaches become more widespread and cheaper to use, especially in the hands of non-specialists. When you are in possession of a hammer, everything starts looking like a nail.

Considering all these caveats, I thus find tools like those described in this volume to be the starting points for understanding human behavior, rather than direct determinants of human behavior. The tools themselves can tell you what they can be used for, not necessarily what problems would benefit the most from their application. The many interesting studies in this book certainly answer the "what" quite well, but most of them are still quite far from answering the "how" and especially the "why". They point out the path to the door, but don't necessarily tell us which door to open. And they can be especially impoverished in illuminating what lies beyond; for that only a true understanding of the human mind will pave the way.

If you've ever read Seth SD'd New York Times Op Eds, you'll recognize the endearingly self-deprecating writing style only a neurotic NYC Jew raised on Woody Allen and The Yankees can pull off. The topic itself is interesting enough: using aggregate google search data analytics as a tool for sociological study. But the author ups the ante with lolzy personal anecdotes, intrinsically comical studies (hint: penis sizes and porn), and that endlessly lovable sense of humor. The book's density is offset by crystalline prose and rapid fire storytelling. Give it a read.

This book is both data-rich and funny ... at the same time! The author is a PhD in economics from Harvard using cutting edge methods to study human behavior. Boy, is he finding great stuff ... and by great, I mean the stuff we need to know but would otherwise never find out. An essential and enjoyable read.

I read it in two days. It is an easy and fun read even for the general reader.

“Everybody Lies” is a fascinating dive into the world of “Big Data”. The core premise of the book is that by mining large data sets we can answer questions more accurately than through other methods. Behavioral and psychological questions can be addressed without the filter of a poll or questionnaire, where “everybody lies”. Thus, in theory, we capture a more accurate representation of people’s real prejudices and desires through big data searches than through polling.

Seth Stephens-Davidowitz uses quirky and often humorous examples to show the power of big data. One example from the book revealed that I was one of the 7% who finished “Thinking, Fast and Slow” (I am not sure whether that is a good or bad thing).

The data is the data, but the interpretation is subjective. My concern is that the subjective conclusions drawn from the data will be presented as fact rather than what they are – subjective interpretations of the data (however statistically significant). As such, there is a danger that such information will be misused. We still need to be cautious in determining the meaning of the data.

Seth Stephens-Davidowitz brings the topic to life with terrific story telling about a wide number of subjects. The author has performed a great service by making this very important topic comprehensible to “the rest of us”.

If you are like me, you wake up in a panic and google things at 4am and wonder, is it just me or does everyone else have weird secret google habits. Turns out, lots of people google their deep secrets (though most often at a more reasonable time of day).

Seth Stephens-Davidowitz takes what we are all thinking about but never talk about and puts it in writing. Detailed, thoughtful, and surprising, Seth Stephens-Davidowitz has a way of making even the most upsetting facts seem accessible and even normal.

I'm not a data person, but i've already suggested Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are for my Book Club because it's a must read for anyone who is looking for an accessible, fun, and informative popular science book that still challenges you to question what you think you know. A great conversation starter!