Web Scraping with Python: Collecting Data from the Modern Web 1st Edition

Q&A with author Ryan Mitchell

What got you interested in web scraping?

In 2011, I started working for a company called Abine, that offered a service to remove customers’ personal information from various sites on the Internet. In the early days of the company, the process of looking for someone’s personal information on all of these sites, filling out all these opt-out forms, faxing emailing, compiling reports to send back to the customers -- it all took a lot of time! I started looking into ways to streamline these processes, and add additional features. I built bots that could search for profiles, store information in our database, fill out web forms, create documents, and send the emails and faxes automatically. Some of these sites were fairly bot-resistant, so I had to learn, and even invent, some interesting techniques to deal with them. I really fell in love with building bots and scraping the web, and continued to do it even after I left the company!

Why is Python such a good fit for web scraping and building web crawlers?

I’ll be honest: As far as high performance programming languages go, Python does not win many speed contests. But with web scraping, you’re not looking for speed -- sending and receiving data across the Internet will be thousands of times slower than any relatively tiny differences in language performance, so you can throw that metric out the window! What you need is something that’s lightweight, easy to deploy to remote machines, that can be installed and run anywhere, that’s easy to write and modify, and, perhaps most importantly: that has a plethora of well-document tools for just about any situation. Python has all of these in spades.

What’s the most interesting way you’ve used web scraping, for professional or side projects?

One of my favorite scraping projects, and something I introduce in Web Scraping with Python, is scraping Wikipedia for historical edits by IP address, time of the edit, and language. You can resolve the IP address to a geographic location, and explore when and where speakers of different languages are making edits. Lots of interesting sociological research potential there!

A recent hobby of mine has also been automated CAPTCHA solving. I really enjoy analyzing new types of CAPTCHAs for vulnerabilities, writing scripts to pre-process the images, creating data sets for machine learning algorithms, and seeing how high I can get the success percentage of my bots! No real practical applications these days, but you never know when it will come in handy.

What information do you hope that readers of your book will walk away with?

I try to stress a couple of things throughout the book:

First, no website is bot-proof. Attempts to make websites more bot-proof generally also result in a loss of usability for human users. That loss of usability may be in the form of slower loading times, poor browser compatibility, lack of accessibility for users with mobility or visual impairments, or users on mobile devices. And many of these measures have no real deterring effect on web scrapers. If you can view the data in a browser, you can capture it with a scraper.

Second, writing web scrapers that capture the data you want often involve combining multiple techniques, some creative thinking, and a dash of laziness. I can’t count the number of times people have asked me to build a bot, or to help them build a bot, to collect data that could be easily obtained through an API! So sometimes your data collection problem can be solved using the information from only a single chapter in the book. On the other hand, I also provide an example of a web scraper that uses JavaScript execution, HTML parsing, DOM interaction, and optical character recognition, all in one piece of code, in order to extract the text from book previews on Amazon! (Sorry, Amazon!) When faced with a web scraping problem you should always 'work the steps' to try formulate a data extraction and processing plan -- it’s not just about learning a single library or command!

What’s the most exciting or important thing happening in your space right now?

Like many fields, especially computer science fields, there’s a lot being done with machine learning and big data. The percentage of page requests performed by humans and bots is about 50/50 right now, and as more humans are getting on the Internet, more bots are too -- and outpacing them! There’s just so much data, and so many machines collecting that data, and so many connections we haven’t been able to make before, waiting to be made. And these aren’t just data scientists and server farm owners making them, either! The kind of research that once might have required months or years of surveys and data collection are now just a Python script, a database, and a weekend of coding away!