Perl & LWP [Paperback]

Sean M. Burke (Author)

Book Description

Publication Date: June 27, 2002 | ISBN-10: 0596001789 | ISBN-13: 978-0596001780 | Edition: 1

Perl soared to popularity as a language for creating and managing web content, but with LWP (Library for WWW in Perl), Perl is equally adept at consuming information on the Web. LWP is a suite of modules for fetching and processing web pages.

The Web is a vast data source that contains everything from stock prices to movie credits, and with LWP all that data is just a few lines of code away. Anything you do on the Web, whether it's buying or selling, reading or writing, uploading or downloading, news to e-commerce, can be controlled with Perl and LWP. You can automate Web-based purchase orders as easily as you can set up a program to download MP3 files from a web site.

Perl & LWP covers:

• Understanding LWP and its design
• Fetching and analyzing URLs
• Extracting information from HTML using regular expressions and tokens
• Working with the structure of HTML documents using trees
• Setting and inspecting HTTP headers and response codes
• Managing cookies
• Accessing information that requires authentication
• Extracting links
• Cooperating with proxy caches
• Writing web spiders (also known as robots) in a safe fashion

Perl & LWP includes many step-by-step examples that show how to apply the various techniques. Programs to extract information from the web sites of BBC News, Altavista, ABEBooks.com, and the Weather Underground, to name just a few, are explained in detail, so that you understand how and why they work. Perl programmers who want to automate and mine the web can pick up this book and be immediately productive. Written by a contributor to LWP, and with a foreword by one of LWP's creators, Perl & LWP is the authoritative guide to this powerful and popular toolkit.

Editorial Reviews

Amazon.com Review

Perl & LWP sets out to unwrap the Library for the Web in Perl (LWP), which is a collection of modules that make it easier to access and pick apart Web pages (and FTP-accessible files, and outgoing e-mail messages) from within your Perl programs. The book succeeds wonderfully, not only in conveying the technical aspects of LWP programming, but in making clear the fun of doing work that's very well suited to Perl. Sean Burke assumes that his readers know something about Perl, albeit not much, and a similar amount about HTML. He does a great job of explaining how LWP functions fit into Perl programs, and how you can use them to make reference to Internet resources far more easily than before.

Burke's narrative takes the form of a guided tour in which he introduces his readers to aspects of the LWP modules one by one. His tone is generally straightforward (sharp commentary alternates with brief code listings, with occasional passages of reference material), but there's sometimes an undercurrent of exuberance that makes the reader want to get going with his or her own programming right away. Overall, the emphasis is on teaching both LWP and Perl itself to the extent necessary to do LWP work. Because of the concise and nicely indexed code modules, though, you'll find this book useful as a reference after you're under way with LWP. --David Wall

Topics covered: How to program with LWP and Perl itself. All of LWP's strong points--including HTML parsing (with tokens and trees as well as with regular expressions), HTML generation and modification, manipulation of HTML forms, and the operation of spiders--are covered. This book has more of a tutorial tone than any similar reference material on the Internet.

About the Author

Sean Burke is an active member in the Perl community and one of CPAN's most prolific module authors. He has been a columnist for The Perl Journal since 1998, and is an authority on markup languages. Trained as a linguist, he also develops tools for software internationalization and Native language preservation.

Product Details

• Paperback: 264 pages
• Publisher: O'Reilly Media; 1 edition (June 27, 2002)
• Language: English
• ISBN-10: 0596001789
• ISBN-13: 978-0596001780
• Product Dimensions: 9.2 x 7.1 x 0.7 inches
• Shipping Weight: 1.1 pounds (View shipping rates and policies)
• Average Customer Review: 4.6 out of 5 stars  See all reviews (11 customer reviews)
• Amazon Best Sellers Rank: #689,122 in Books (See Top 100 in Books)

More About the Author

Sean Michael Burke is the author of "Perl & LWP" (softcover version, and free online version) and "RTF Cookbook", and was a columnist for "The Perl Journal" for several years. His columns often focused on processing (human) languages, and most are collected in "The Best of the Perl Journal: Games, Diversions, and Perl Culture" and free online.

He has a master's degree in linguistics from Northwestern University, with the thesis "The Design of Online Lexicons", which appears as a chapter in "A Practical Guide to Lexicography". He has written several dozen Perl Archive (CPAN) modules including Maketext, the basis of Perl software localization; HTML-Tree, Perl's notably robust HTML parser; and the specification and modern implementation for Pod, Perl's documentation system.

Burke currently works for several projects for the preservation of endangered North American languages. He wrote "RTF Pocket Guide" after having learned RTF (a text format like .doc) as part of those projects, for the step where lexicon databases need to be turned into conventionally formatted dictionaries for publication. Examples now in print: "Dictionary of Alaskan Haida" and Dictionary of Jicarilla Apache: Abáachi Mizaa Iłkee' Siijai".

He makes programmers think he's basically a linguist and linguists think he's basically a programmer. He is a sparklingly charismatic yet always modest person who really does enjoy writing about himself in the third person like this. He currently lives on a small island off the coast of Canada.

(This Sean Burke is not the (Irish scholar) Sean Burke who wrote "Death and Return of the Author", "The Ethics of Writing", and "Deadwater"; nor is he the (Canadian athlete) Sean Burke who was a goalee in the National Hockey League; but he assumes they're nice people anyway.)

Customer Reviews

Most Helpful Customer Reviews

17 of 17 people found the following review helpful:

5.0 out of 5 stars Excellent coverage of LWP, packed full of useful examples, July 16, 2002

By 

Gavin Estey (Astoria, NY, USA) - See all my reviews

This review is from: Perl & LWP (Paperback)

I was definitely interested when I first heard that O'Reilly were publishing a book on LWP. LWP is a definitive collection of perl modules covering everything you could think of doing with URIs, HTML, and HTTP. While 'web services' are the buzzword friendly technology of the day, sometimes you need to roll your sleeves up and get a bit dirty scraping screens and hacking at HTML. For such a deep subject, this book weighs in at a slim 242 pages. This is a very good thing. I'm far too busy to read these massive shelf-destroying tomes that seem to be churned out recently.

It covers everything you need to know with concise examples, which is what makes this book really shine. You start with the basics using LWP::Simple through to more advanced topics using LWP::UserAgent, HTTP::Cookies, and WWW::RobotRules. Sean shows finger saving tips and shortcuts that take you more than a couple notches above what you can learn from the lwpcook manpage, with enough depth to satisfy somebody who is an experienced LWP hacker.

This book is a great reference, just flick through and you'll find a relevant chapter with an example to save the day. Chapters include filling in forms and extracting data from HTML using regular expressions, then more advanced topics using HTML::TokeParser, and then my preferred tool, the author's own HTML::TreeBuilder. The book ends with a chapter on spidering, with excellent coverage of design and warnings to get your started on your web trawling.

14 of 14 people found the following review helpful:

5.0 out of 5 stars This book can teach you expert-level web scraping/munging., July 12, 2003

By 

wickline "mwickline" (Portland, Oregon USA) - See all my reviews

This review is from: Perl & LWP (Paperback)

If you aren't yet comfortable using object-oriented Perl modules, the multitude of examples will at least allow you see how it's done even if you're a bit fuzzy on what's happening 'underneath' when you call object methods. If you're comfortable learning how to do something without knowing exactly why it works, then the author's clear step-by-step explantions and numerous progressively more powerful examples should make this book accessible even to relatively innexperienced Perl programmers.

More experienced programmers will understand better why things work, but any Perl programmer will set this book down feeling empowered to turn the web into their own valet. No longer do you need to check multiple sites looking for interesting information. Instead, you can readily author code to do that for you and alert you when items of interest are found. You can use these tools to free up personal time, to harvest information to inform business decisions, to automate tedious web application testing, and a zillion other things.

The author's clear exploration of the relevant Perl modules leaves the reader with a good depth of understanding of what these modules do, when you might want to use which module, and how to use them for real world tasks. Before reading the book, I knew of these modules, but they were a rather intimidating pile. I'd used a few of them on occasion for rather limited projects, but was reluctant to invest the time required to read all of the documentation from the whole collection. Mountains of method-level documentation do not a tutorial make. This book takes all of that information, selects the most important parts, and ensures that those parts are covered in progressively more powerful and/or flexible examples.

If you know Perl and you're sick of 'working the web' to get information and you want the web to work for you instead, then you need this book. I had a personal project that was on the back burner for a couple of years because it just sounded too hard. The weekend after I finished this book, I wrote what I had previously thought to be the hard part of that project and it was both easy and fun. This book makes hard things not just possible, but actually easy.

-matt

14 of 14 people found the following review helpful:

5.0 out of 5 stars Very Informative and useful, August 7, 2002

By 

"sherzodr" - See all my reviews

This review is from: Perl & LWP (Paperback)

As a web programmer, I had dealt with several such projects dealing with web automation and writing simple crawlers even before I read "Perl & LWP". The book was the first book I've read on the subject, and I'm by no means disappointed. The book is very well organized, very informative and nails the subject in the head. I am pleased.

I noticed some inaccuracies in the discussions, some chopped off paragraphs and sentences. But this doesn't affect the usability of the book much. Author Sean Burke does a great job in walking one through the most of the aspects of web automation and data extraction in the web using Perl and LWP (libwww in Perl ).

The codes the book gives are very well organized, well written and easily debugable. The steps are pretty consistent across all the examples:
a) Inspect the HTML source code of the page;
b) Determine the tokens and patterns of interest;
c) Write the first code;
d) Fine tune the code;

As usual, I'll be commenting on individual chapters to give you an idea of the
coverage of the book in more details...