sed & awk (Nutshell Handbooks) [Paperback]

Dale Dougherty (Author), Arnold Robbins (Author)

Book Description

Publication Date: March 8, 1997 | ISBN-10: 1565922255 | ISBN-13: 978-1565922259 | Edition: Second Edition

sed & awk describes two text processing programs that are mainstays of the UNIX programmer's toolbox.

sed is a "stream editor" for editing streams of text that might be too large to edit as a single file, or that might be generated on the fly as part of a larger data processing step. The most common operation done with sed is substitution, replacing one block of text with another.

awk is a complete programming language. Unlike many conventional languages, awk is "data driven" -- you specify what kind of data you are interested in and the operations to be performed when that data is found. awk does many things for you, including automatically opening and closing data files, reading records, breaking the records up into fields, and counting the records. While awk provides the features of most conventional programming languages, it also includes some unconventional features, such as extended regular expression matching and associative arrays. sed & awk describes both programs in detail and includes a chapter of example sed and awk scripts.

This edition covers features of sed and awk that are mandated by the POSIX standard. This most notably affects awk, where POSIX standardized a new variable, CONVFMT, and new functions, toupper() and tolower(). The CONVFMT variable specifies the conversion format to use when converting numbers to strings (awk used to use OFMT for this purpose). The toupper() and tolower() functions each take a (presumably mixed case) string argument and return a new version of the string with all letters translated to the corresponding case.

In addition, this edition covers GNU sed, newly available since the first edition. It also updates the first edition coverage of Bell Labs nawk and GNU awk (gawk), covers mawk, an additional freely available implementation of awk, and briefly discusses three commercial versions of awk, MKS awk, Thompson Automation awk (tawk), and Videosoft (VSAwk).

Editorial Reviews

From the Publisher

sed & awk, one of the most popular books in O'Reilly & Associates' Nutshell series, describes two text processing programs that are mainstays of the UNIX programmer's toolbox. sed is a "stream editor" for editing streams of text that might be too large to edit as a single file, or that might be generated on the fly as part of a larger data processing step. The most common operation done with sed is substitution, replacing one block of text with another. awk is a complete programming language. Unlike many conventional languages, awk is "data driven" -- you specify what kind of data you are interested in and the operations to be performed when that data is found. awk does many things for you, including automatically opening and closing data files, reading records, breaking the records up into fields, and counting the records. While awk provides the features of most conventional programming languages, it also includes some unconventional features, such as extended regular expression matching and associative arrays. sed & awk describes both programs in detail and includes a chapter of example sed and awk scripts. This new edition covers features of sed and awk that are now mandated by the POSIX standard. This most notably affects awk, where POSIX standardized a new variable, CONVFMT, and new functions, toupper() and tolower(). The CONVFMT variable specifies the conversion format to use when converting numbers to strings (awk used to use OFMT for this purpose). The toupper() and tolower() functions each take a (presumably mixed case) string argument and return a new version of the string with all letters translated to the corresponding case. In addition, this edition covers GNU sed, newly available since the first edition. It also updates the first edition coverage of Bell Labs nawk and GNU awk (gawk), covers mawk, an additional freely available implementation of awk, and briefly discusses three commercial versions of awk, MKS awk, Thompson Automation awk (tawk), and Videosoft (VSAwk).

From the Back Cover

The book begins with an overview and a tutorial that demonstrate a progression in functionality from grep to sed to awk. sed and awk share a similar command-line syntax, accepting user instructions in the form of a script. Because all three programs use UNIX regular expressions, an entire chapter is devoted to understanding UNIX regular expression syntax. Next, the book describes how to write sed scripts. After getting started by writing a few simple scripts, you'll learn other basic commands that parallel manual editing actions, as well as advanced commands that introduce simple programming constructs. Among the advanced commands are those that manipulate the hold space, a set-aside temporary buffer. The second part of the book has been extensively revised to include POSIX awk as well as coverage of three freely available and three commercial versions of awk. The book introduces the primary features of the awk language and how to write simple scripts. You'll also learn: common programming constructs; how to use awk's built-in functions; how to write user-defined functions; debugging techniques for awk programs; how to develop an application that processes an index, demonstrating much of the power of awk; and FTP and contact information for obtaining various versions of awk. Also included is a miscellany of user-contributed scripts that demonstrate a wide range of sed and awk scripting styles and techniques.

See all Editorial Reviews

Product Details

• Paperback: 434 pages
• Publisher: O'Reilly Media; Second Edition edition (March 8, 1997)
• Language: English
• ISBN-10: 1565922255
• ISBN-13: 978-1565922259
• Product Dimensions: 7.1 x 0.9 x 9.2 inches
• Shipping Weight: 1.6 pounds (View shipping rates and policies)
• Average Customer Review: 4.3 out of 5 stars  See all reviews (53 customer reviews)
• Amazon Best Sellers Rank: #64,379 in Books (See Top 100 in Books)
  • #24 in Books > Computers & Technology > Operating Systems > Unix
  • #49 in Books > Computers & Technology > Operating Systems > Linux

More About the Author

Discover books, learn about writers, read author blogs, and more.

Most Helpful Customer Reviews

61 of 64 people found the following review helpful

5.0 out of 5 stars 2 cute little guys who will work hard for you July 1, 2004

By Alexander E. Paulsen

Format:Paperback

I've had this book for 3-4 years now and find it to be my handiest reference. I even use ot more than my vi book.

The raw power of sed and awk will humble even the proudest VB programmer and this book will help you get the most out of both the "Aho, Weinberg and Kernigan" and the "stream editor".

The first chapter section is entitled "May You Solve Interesting Problems" a paraphrase of the old Chinese curse, but with sed and awk no problem seems insurmountable and will make all problems appear interesting.

This book is an outstanding reference and will get you up and running with both of these handy little programs in no time.

Just to give you an example - the first time I ever used these programs (sed and awk) I developed a cron script to query a database every day at midnight, sort the results, grabbed lines with tagged values within certain limits, added a few totals, did some averaging, "starred" the lines that departed from certain parameters, then formatted the results into a report with a header showing the dates, times and query results, statistics then packed them into a report file and e-mailed them to a dozen recipients and added a few recipients if the data was outside certain values.

I did this remarkable feat by copying a few examples from the book, changing a thing or two and in the end had a little shell script that was a few K-bytes at the most.

This VB guy from corporate worked on a VB version of the same function and ended up with six revisions, several megs in the executable and never was able to get it formatted right. Not to say that it couldn't be done but..... I wrote mine before lunch one day. He worked on his for weeks and weeks....

Such is the power contained within should you want to tap it Grasshopper.

Great book. Amaze your friends and colleagues. Have the office women ( or men) throw flowers at your feet as you enter every day - well maybe not, but this will make you more productive.

You real nerds will hardly get finished with a single box of Pop Tarts before you are writing really cool scripts when using this book.

Like all O'Reilly books this one is terse, practical and highly usable - just like UNIX. Read more ›

34 of 35 people found the following review helpful

4.0 out of 5 stars You'll not regret this one! September 25, 2001

By Ales Kavsek

Format:Paperback

My primarily profession is database administration on Windows NT/2000 platform so I'm pretty new to Unix tools. Several months ago I needed some inexpensive, yet powerful tools to manipulate some large text files for our data warehouse project -- basically I was looking for a way to avoid expensive commercial ETL tools. PERL was my first choice, but it seemed to me that learning curve is too steep for what I needed, so I ended with GNU sed and awk.
I was happy with the GNU version of Guide for Awk "Effective AWK Programming" written by one of the author of this book, Mr. Arnold D. Robbins -- but I needed some good reference for sed as well and this book was logical choice for me. Not only that it helped me to learn SED, now I'm using the book mostly as reference and user guide for awk. It's well written and will certainly satisfy both the advanced and the novice users.
The only objections that I have are examples at the end of the book. Not many readers will find them interesting and useful, especially chapter 12 Full-Featured Applications.
Overall good choice if you want to learn either of the two tools!

32 of 34 people found the following review helpful

4.0 out of 5 stars Powerful tools for text manipulation April 15, 2000

By Primoz Peterlin

Format:Paperback

Unix has earned itself quite a reputation for its potent tools, used for batch editing of text files (like program output). Sed and Awk are two of these tools. Sed is a direct descendent of Ed, the original Unix line editor, which employs regular expressions, a powerful method for description of patterns in text, for operations like substitute, append or delete. Awk is a complete scripting language with programming structures like conditionals, loops, functions etc., developed in 1970's by Alfred Aho, Brian Kernighan and Peter Weinberger (hence A-W-K). The trio has also written a book on Awk.

Dale Dougherty (in the 2nd edition with Arnold Robbins, maintainer of GNU Awk and author of several more books on Awk programming language) have made a good job in making a thoroughly readable tutorial on Sed and Awk. However, it remains a mystery to me how they succeeded to fill no less than 407 pages with it. Mind you, Sed and Awk are not really some big monsters. There exist something like two dozens of operators in Sed (most of them you will probably never use), and the syntax of Awk mimics those of C programming language, so it is likely that you know it already. Once you grok the idea of regular expressions, you should become a proficient user of Awk in about 30 minutes.

In conclusion, go buy the book if your need to manipulate text files on Unix and you think you need a lengthy tutorial with a gentle learning curve. Otherwise, short references on Awk and Sed, like the ones in "Unix Power Tools" and a bunch of examples showing some tricks you might not think of, will probably be more useful. In addition, it is good to know that during the nineties, much of the focus has drifted from Awk to Perl, so you might consider a book on Perl as well.

19 of 20 people found the following review helpful

3.0 out of 5 stars Useful to learn it, not for experienced users, and not a reference book either. July 1, 2007

By GPK

Format:Paperback

I have a science background and am a regular user of unix systems, although I am not a professional IT'er. Therefore I was already familiar with sed and awk. I was looking for a single book that would give a good overview of all functions on a more-than-beginners-level and at the same time could serve as a reference. Perhaps not as 'the ultimate reference' but good enough for professionals who want to use sed and awk in a pragmatic way. With this I mean: a book you can turn to if you just want to write a little program/script that get's the job done.

This is not that book. I found this book useful in explaining the basics of sed and awk. There is also a quick reference section on both sed and awk. However, my overall observation is that the level of this book is not very advanced and therefore did not do the job for me. I would recommend this book to beginners and not to people who have already used sed and awk themselves. The level is basic, the reference sections are too brief and incomplete to be really 'reference sections'. I am still looking for a real reference book.

Tip: if you want to get started with sed and awk, search the internet for "sed one-liners" and "awk one-liners": these are one-line example 'programs' or commands that most likely will do what you are looking for. And if not, they are helpful to get you going and make some trivial changes so that the program does do what you need. No need for buying a book.