Chapter 1Introduction
Linux is a solid operating system. It is easy to use and install, has very powerful capabilities, runs fast on almost any hardware, and rarely crashes. It has few bugs and its widespread support from a cast of thousands ensures that any remaining bugs get fixed as soon as they are discovered. It is highly versatile and can be made as secure as any UNIX system.
Unfortunately, UNIX and Linux machines are broken into every day, not because they are inherently insecure, but because the steps required to expose a system to the real world safely-the modern Internet-are not always so obvious. The single goal of this book is to teach any Linux or UNIX system administrator how to secure his systems, keep them secure, and feel confident that all necessary steps have been taken.1.1
Who Should Read This Book?
This book will aid Linux and UNIX System Administrators (SysAdmins) in making their systems and networks as secure as possible from intruders and improper action of the users. It covers both quick and simple solutions, and some more involved solutions to eliminate every possible vulnerability.
It is organized to allow the busy SysAdmin to increase the security of the systems one piece at a time. It is recognized that one cannot take a system down for a week and work exclusively on its security for that week. In the real world, a SysAdmin's time is divided up by many tasks that cannot wait and systems are too critical to stay down for long.
In the real world, some systems will be broken into despite the best efforts of talented SysAdmins. This book devotes over 60,000 words to dealing with a possible break-in. It deals with how to prepare for it, how to detect it, and how to recover from it quickly and completely with minimal loss of confidential data and money, with minimal inconvenience to one's customers and employees, and with minimal publicity. This is considered one of the unique features of this book. On March 30, 2000, 350 "hackers" from around the world gathered in Israel for a conference. Organizers there said that they were able to break into 28 percent of Israeli computers that they tried and that this percentage was typical worldwide. This was with the permission of the computers' owners, who were convinced that their computers were invulnerable. The quoted statistics were not broken down by operating system type. Both John Draper ("Captain Crunch") and Kevin Mitnick were there.
The book is designed to be used by both the veteran of many years of Linux and UNIX experience, as well as the new SysAdmin. It does assume that the reader is somewhat knowledgeable in system administration; Prentice Hall has other fine books to help people hone their SysAdmin skills. There are many useful details here, both for the person with a single Linux box at home and for those supporting multinational corporations and large government agencies with very large networks comprised of multiple types of operating systems.1.2
How This Book Is Organized
Part I is concerned with increasing the security of your systems. This book is organized with the understanding that some SysAdmins have only a little time right now, but certainly want to fix the most severe holes immediately, before someone breaks into their systems. (The smaller holes also need to be closed, but statistically there is more time to address them before a cracker is likely to try them. Crackers, sometimes incorrectly called hackers, are people who break into computer systems without permission for the fun, challenge, fame, or due to a grudge.) These urgent quick-to-do items are covered in Chapter 2 "Quick Fixes for Common Problems" on page 15. That chapter starts with a discussion of basic security concepts to bring those new to Linux security up to speed and to serve as a "refresher" for veterans. The author estimates that applying just the quick fixes may reduce a system's vulnerability by 70 to 90 percent, based on published reports and incidents discussing probable "points of entry." Many of these solutions are independent from each other so that a SysAdmin may pick the solutions most appropriate to his or her situation and may implement these in almost any order.
The book then progresses into more involved procedures that can be done to increase security, allowing the system administrator to progress to as secure a system as time and desire allows. It even addresses some simple kernel modifications to increase security still further. It can be treated as a workbook, to be worked through a bit at a time, or as a reference book, with relevant areas picked from the Table of Contents or from the extensive Index. Part II deals with preparing for an intrusion. No computer or network is completely secure and anyone who thinks that theirs is 100 percent secure is, well, probably due for some "education." Most computer security books deal almost exclusively with securing systems and devote only a few pages to dealing with an intrusion, that 10-40 percent of their readers will suffer. This author considers this to be a naive disservice. (All other common platforms are considered even more vulnerable.) In many of the cases that this author has been asked to analyze, the vulnerability that allowed the break-in turned out to be a bug in system software that had not been well-known at the time. This proves the point that just securing a system is not sufficient.Innovative solutions are presented to even the most daunting problems, such as keeping customers' credit card numbers secure even if the Web server and the entire internal network are completely compromised! This solves a major widespread problem with e-commerce companies.
This book is called Real World Linux Security: Intrusion Prevention, Detection, and Recovery because in the real world a significant percentage of computers are broken into and the prepared SysAdmin is well prepared for this. Perhaps 5-25 percent of SysAdmins who have secured their Linux boxes still will have to deal with an intrusion. Even the author's own quiet site on a Dynamic IP over PPP suffers weekly intrusion attempts (with no successes so far), but it has been prepared for intrusion attempts and even for fast recovery from a possible successful intrusion.
Switching to another platform will not reduce this risk, in my opinion. I have seen many reports of security bugs in various competing systems. Almost weekly I see a report on a newly discovered severe vulnerability in software long running and widely distributed on these closed-source platforms. Software written by independent vendors also has its share of problems.
Part III deals with detecting intrusions (both attempts and successes) and sophisticated notification and logging in detail. Part IV discusses recovering from intrusions successfully, completely, and quickly! It also covers tracking down the intruder and dealing with law enforcement officers and the courts, and what to expect from them. Outages can cost millions of dollars a day in lost revenue and bad publicity can mean more lost business and worse-the dismissal of the SysAdmins. A quick recovery may get no publicity and might even be blamed on a glitch in the Internet.
This book covers many security problems. These include problems of incorrect configuration, some services whose design prevents them from being made secure, some inherent limitations in the TCP/IP, UDP/IP, ICMP/IP, ARP, and related protocols, bugs in programs that have come with various Linux distributions or which get installed on Linux systems, and even some physical security and human factors (social engineering) matters.
Please do not get the idea that Linux is a hard-to-configure, buggy, half-baked idea not worthy of your attention! Nothing could be further from the truth. Many security experts consider Linux and FreeBSD UNIX to be the most secure general purpose operating systems. This is because the open source allows many more talented white hats to inspect each line of code for problems and to correct these problems and "fold the fixes back into the master code base" maintained by Linus, the Free Software Foundation, and the creators of the major distributions.
There now is much sharing of code between Linux and the various BSD releases of UNIX and even versions of UNIX supported by the various vendors. This is to the advantage of all users of these systems, since there are more developers improving the code. By following the steps in this book, even a major intrusion can be detected and recovered from in a few minutes, rather than the many hours or days that The White House, Lloyd's of London, eBay, and other major, but apparently unprepared, sites required to recover.1.2.1 Conventions in This Book
The Table of Contents is designed to allow one to scan it quickly for applicable issues. The Index is extensive and most items are cross-referenced, both by the subsystem or program that is affected and the type of problem, e.g., vulnerability. Some Internet resources (URLs) are listed in whatever sections discuss them; many popular Internet resources are discussed in Appendix A. Many URLs are listed in the Index too. Appendix B discusses non-Internet resources; these include books, CD-ROMs, and videos; some of these are free for the asking. Other appendices contain source code or other data that is too massive to appear in running text. These items also appear on the companion CD-ROM as do a number of open-source tools that are discussed in the text. These are mirrored on the associated Web site, realworldlinuxsecurity. The Web site also will contain the latest information and errata. There is also a Glossary of Acronyms.
The three-headed dog on the book's cover is Cerberus from Greek mythology. He guards the entrance of Hades to keep the evil demons from escaping into our world and wreak havoc, chaos, pain, and disaster. He also prevents the living from entering Hades. This is not unlike the security aspects of a system administrator's job and it certainly seems to require three heads to keep ahead of the problems.
Not too many people understand that TCP/IP is the Transmission Control Protocol (TCP) running on top of the Internet Protocol (IP). This means that an incoming TCP/IP packet is first processed by the IP layer of the communications "stack," then by the TCP layer, and then is passed to the program listening on that port. Similarly, UDP/IP is the User Datagram Protocol (UDP) on top of IP and ICMP/IP is the Internet Control Message Protocol on IP. For brevity, these will be referred to as TCP, UDP, and ICMP throughout this book.1.2.2 Background
You can assume that there are crackers out there with copies of all of the proprietary source code from the UNIX vendors, other operating systems, routers, etc. so the crackers know their vulnerabilities. Unlike Linux, though, there will be far fewer white hats looking over the proprietary code for vulnerabilities and working to get them fixed. While working for free, the Linux volunteers are some of the very best programmers in the world and our goal is the very best code. We will not be limited by time-to-market, development costs, or similar limitations of the commercial world.
While this book is written for the Linux SysAdmin, 95 percent of it is applicable to most UNIX systems as well. The principal difference is that most UNIX SysAdmins do not have access to source code and will need to get most fixes from their vendor. Most vendors release fixes for security holes quite quickly and many of their clients have support contracts to cover this. Some of the security problems to be explored are inherent to the various services and protocols and very similar problems will be found on all platforms, including UNIX, Macs, Windows, VMS, and any other platform supporting the same services.
This book covers types of intruders and their goals, types of security holes and how to plug them, and where to look on the Internet to keep up-to-date on the latest holes and plugs. In many cases, system administration duties are divided between people with different titles, such as Network Administrator, Database Administrator, Webmistress, operator, etc. This book is for these people too. Additionally, it addresses issues of program design that every programmer writing applications, CGIs, shell scripts, etc. must know to avoid creating a security hole.
It is important that the SysAdmin ensure that users have been taught about security too. A user's files or program with improper security can allow intrusion not only into his data but also to the rest of the system and network. This is because some security holes require access to some user's account.
It is important that there be no unauthorized and no unanalyzed bridges between the Internet and internal LANs or WANs, sometimes called Intranets. Producing a written policy to help ensure security, while possibly boring, is an important part of security. If it is on paper, people are less likely to disregard it, particularly if disregarding it could cause a problem that they could be "blamed" for. An entire chapter is devoted to policy.
Intranets are trusted in that confidential unencrypted data flows along them. If the bridging system is not secure, a cracker can come in over the Internet and sniff the Intranet, see the confidential data, and probably break into the important systems.1.3
What Are You Protecting?
There are essentially four things that you need to protect against.1.
An intruder reading your confidential data
An intruder could see your product designs, competitive plans for the future, names and addresses of customers, customers' credit card and bank account information, your bank account numbers and contents, sensitive system data including modem phone numbers, passwords, etc.
Frequently the greater harm will happen if the intruder makes the data available to others. While a cracker herself knowing about your product design may not be a severe problem, publishing it on the Internet where your competitors can get it is a severe problem. If your customers' credit card numbers are revealed and it becomes publicly known (as has happened to America Online) people will be afraid to do business with you.2.
An intruder changing your data
This is perhaps the most scary and damaging intrusion. An intruder can alter designs and data without your people discovering it. This could cause loss of life and very severe liability. What if the formulation of a pharmaceutical company's medicine is changed, the design of a automobile or airplane is changed, or the program operating a factory or patient X-ray or Gamma ray device is changed. Patients' medical records could be altered. Any of these situations could result in death. They also could result in large lawsuits.
An intruder may not even realize the harm that his actions could do. In a case in Berkeley, California, crackers were in a system that controlled a cyclotron that sometimes was used for cancer treatments. Intruders have caused banks' ATMs to spit out money to no one in particular and made embarrassing changes to agencies' Web pages, including the Central Intelligence Agency's. 3.
An intruder removing your data
The harm here is self-evident and a good backup program limits the damage that can be done if it is detected.4.
Denial of Service
This is when an intruder causes a computer or network to be "less available" or "not available." Less available includes the system slowing down substantially because of intruder-induced loads or rescheduling, less modems or ports being available to legitimate users, due to intruders shutting some down, etc. Not available means that the intruder has caused the system to crash or go down.
An intruder may think it amusing to crash the computer controlling a phone company exchange. Unfortunately, this blocks 911 emergency calls and interrupts the Air Traffic Control System voice and radar circuits between a control tower and remote radio antennas and other control towers. This could cause loss of life. Note that any interference with the operation of an aircraft in the U.S. that causes loss of life is a federal felony that carries the death penalty.5.
An intruder launching other attacks from your site
This could result in Denial of Service both due to loss of bandwidth and from other sites blocking your site as "a cracker site." This attack could result in bad publicity and possible legal liability.
Any of these attacks can cause less severe problems, such as the bankruptcy of a company, or firing of a SysAdmin. Certainly, this latter problem is the most severe of all.1.4
Who Are Your Enemies?1.
Crackers
Frequently, crackers regard the companies or agencies whose computers they break into as evil or simply unimportant. Sometimes their actions are benign (in that they do not damage or publish confidential data or cause Denial of Service) but do cost time and money for SysAdmins to lock them out. Sometimes their goal is to cause as much damage as possible. Their attacks occur essentially randomly but gravitate towards "big name" sites, typically large well-known companies and government agencies. They are very hard to catch.
Frequently, they will connect in through a laptop connected to a pay phone. Other times they will come in through a compromised system from a second compromised system or even via a long string of compromised systems. Sophisticated attacks use long chains of compromised systems, making it difficult or impossible to trace and catch the crackers. Crackers have periodically posted customer credit card numbers, purloined from compromised systems, for years. However, in late 1999 there were a number of cases where crackers obtained large numbers of stolen credit cards from merchants such as Pacific Telephone and an airline and demanded millions of dollars to not post the card numbers. No money was paid and valid card numbers were posted by the crackers. Clearly, the motive was greed and theft.
Some will break into systems to have a "base of operations" from which to attack other systems. Their goal is not to be detected on these base systems; unless the SysAdmin is especially vigilant, they could be "in" for months or years. They may use these systems to have an untraceable account or they may use them later in a massive Distributed Denial of Service (DDOS) attack against another computer. Techniques for reliably detecting even these "quiet crackers" will be covered in depth.2.
Disgruntled current employees
These attacks, too, are hard to predict, but proper auditing can both catch them and reduce the likelihood of attack due to fear of being caught. Frequent backups done and stored in such a way that no one person can cause them to be lost or invalid is strongly recommend.
Certainly, if an action such as a poor review, reprimand, or unpleasant assignment is about to be given to an employee with access to important data or hardware, it would be prudent to make system backups, possibly alter door access codes, etc.3.
Disgruntled former employees
These attacks can be predicted somewhat by assuming that the first thing a fired employee might do is try to harm the system. Most SysAdmins have had the sad job of being asked to disable someone's computer access while they were in with their boss or Human Resources being fired. Naturally anyone who might unknowingly give this employee access should be informed of the termination. This includes vendors who have access codes and other employees.
This brings to mind the sad case of an airline employee in California who was fired but nobody bothered to tell the other employees or the security personnel. The public is not aware that, as a "courtesy," airline employees were not required to pass through the metal detectors. This now-fired employee took advantage of this to bring a gun on a flight and shot the flight crew to death. The jet crashed and no one survived. Security is serious business.4.
Competitors
Your competitors will try to get your product designs, customer lists, future plans, etc. This information is usually used to steal your designs and customers, but sometimes embarrassing information is made public.
While not strictly your competitors, headhunters will do almost anything to get the names and phone numbers of your employees so that they may hire them away. Some companies post their employee names and numbers on their Web sites. It is recommend to not do this to prevent their being targets of "raids." You may want to post the names and numbers of a few employees who interact with people outside of the company.5.
Spies
Despite the fall of the Soviet Union and peace in the Middle East, there is plenty of spying going on throughout the world. Some of it is one country spying on another. There is an abundance of activity where one country spies on other countries' industries to gain illicit advantage. There is no shortage of industrial spying.6.
Criminals
While crackers are usually not motivated by money, the criminal element may be, breaking into computers for the sole purpose of theft, extortion, and other criminally profitable ventures. Organized crime may be involved.7.
Extremists
Some individuals on what they consider to be a moral or religious crusade may try to intrude into your system. These are not just maniacs from the other side of the globe. There are many groups whose members in the past have done criminal acts either against computers or even against physical objects. These include various anti-government types, "activists," those against big business or certain industries, political extremists, pro this, and anti that. If one is the SysAdmin at a company or agency that may be a target of an extremist group, one needs to take precautions. Almost no one is immune.1.5
What They Hope to Accomplish
What do they hope to accomplish? Crackers want to leave their mark so their cracker friends will see it and, hopefully, so that they make the news. The quiet ones just want your systems' CPU cycles and your network bandwidth to use to attack other systems. Disgruntled and fired employees obviously want to harm you. Deleting or altering critical data or posting confidential data usually is their goal. The FBI claims that 80 percent of attacks are internal; this author's experience is different.
Competitors are most interested in increasing their profits and market share but will stoop to lowering yours to weaken you. They will use any information that they can obtain. They most commonly are after customer lists and plans for future products or marketing campaigns. It truly is scary how much absolutely critical and confidential data resides on many executives', sales people's, and engineers' laptops, there for the grabbing!Unlikely? Two of my clients have been relieved of their laptops at gun-point; one in his up-scale hotel in the U.S., and one in his hotel room in Africa in the middle of the night, both in the past three years. A third client caught a competitor trying to obtain its confidential computer files. 337,000 laptops were stolen in the U.S. alone in 1999.
Usually, spies want information to use to their advantage. Clearly, criminals will try to subvert systems to allow theft; obtaining credit card numbers is a common theft. Extortion, too, probably is popular, though only the unsuccessful ones are publicized. This author suspects that the successful ones are paid quietly.
Extremists, sometimes considered terrorists, want to cause you harm and gain publicity for their causes. Defacing Web sites is common and this book discusses some excellent strategies to prevent it. Many will try to disrupt your operations, shutting down your e-mail, your Web site, your manufacturing plants, or erasing or altering data from your systems.1.6
Costs: Protection versus Break-Ins
Many people do not realize the cost of being cracked: lost customers, stolen merchandise, bad publicity, loss of human life, etc. These costs can be calculated and the cost of protection may be calculated too.
The book covers how to calculate the cost of protection versus the cost of break-ins, how to prevent most break-ins, and how to prepare for break-ins and how to recover from break-ins quickly, sometimes within a few minutes. You will not need to be down for the days at a time that eBay, The White House, and many other high-profile sites have been down for.1.7
Protecting Hardware
Physical security for the systems is discussed, as is access to floppy, CD-ROM readers, tape drives, and other issues. Many people are not aware that almost any system can be taken over in one minute merely by inserting a rogue floppy or CD-ROM and pressing the reset button or momentarily interrupting power at the switch, the plug, or the building's main breaker. Many systems are so accommodating that they will ask the intruder what file on the disk she would like to boot as the kernel. Without special configuration, simply supplying the single parameter when booting Linux will bypass all password checks. The name floppy is the original name of the magnetic disk media using a flexible plastic envelope. It is the author's recollection that IBM coined the term diskette as sounding more professional when they made business presentations in three-piece suits.1.8
Protecting Network and Modem Access
Topics include protecting access to your modems and networks from a variety of attacks. These attacks range from a rogue easily determining what the phone number of your modem is to intruders tapping your phone lines. Strategies from making wardialing attempts to find your modems difficult to preventing intruders bridging into your internal networks to get at unprotected systems are covered. Wardialing means dialing a large number of phone numbers in sequence to determine which ones have modems attached. Frequently, the wardialing software can analyze the type of system that answered by the login prompt. It will try common account names and passwords and system-specific attacks.1.9
Protecting System Access
This book covers how to protect access to a system via proper password policy and configuration, disabling insecure services and software, upgrading insecure versions of programs, logging out inactive users, discovering new intrusion techniques (including the occasional security bug in trusted software) before crackers can use them against a system, and avoiding the various traps that crackers plant.
Also discussed are a variety of techniques and tools that further reduce vulnerability. Many Linux systems on the Internet (and non-Linux system as well) offering telnet and FTP can be cracked via exhaustive password searches. Many systems have weak passwords that do not require exhaustive searches. On many systems, an intruder merely uses anonymous FTP to get a copy of the encrypted password file and cracks it on his own system with widely available tools. Then he "owns" that system (controls it with unauthorized root). On others, he uses known vulnerabilities in the POP or IMAP daemons, named, sendmail, or he breaks a CGI program. It is explained in detail how to build up a number of concentric walls in one's systems and network, each one of which must be penetrated in turn, before a break-in can occur. I call these concentric walls "Rings of Security" throughout the book. A single wall with many places where it might be broken would require only one break for a cracker to gain full access. However, these "Rings of Security" will stop most crackers from causing major problems because it is unlikely that a cracker will be able to break through all of them in turn. Many leading security experts use the term security in depth for the same meaning, including Kurt Seifried of SecurityPortal and Mike Warfield of Internet Security Systems. It is very fortunate that both of these well-known experts found time to review this book and offer many suggestions.1.10
Protecting Files
On many systems, users are in need of security education regarding how to set proper permissions on their files, changing the initial password, and why some passwords are better than others. Sometimes a user will find a "cool" CGI script or program and ask you to install it or have the access to install it herself. Frequently these programs have severe security holes that are not obvious.
My favorite is the CGI script technique which is used to generate e-mail from a browser user's form submission. The problem with many of these is that the script simply drops the user's fields into a Mail command. All an intruder needs to do is to put a semicolon or newline in the middle of the right field and any text after that will be interpreted as a shell command. Even if the command is not running as root, the user may have access to the database or other critical data. Even experienced programmers may not understand all of the security issues.
There are many, many issues that must be attended to in order to maintain security on a Linux system on the Internet. This is called hardening a system. Similarly to securing your house against invasion, some things are simple, easy, and inexpensive such as locking the doors when you go out. Installing a deadbolt lock will improve security. Adding an alarm system is even better. Arranging 24 x 7 armed guards is the ultimate protection but unavailable to most and not cost effective, except those at particular risk such as the rich and powerful. You must get "it" right 100 percent of the time, but the cracker only has to get lucky once.1.11
Preparing for and Detecting an Intrusion
Periodically, security holes in programs are detected and, unfortunately, some are discovered by crackers poring over the source or experimenting. While some detractors claim that this is a weakness of Linux, the reality is that with so many people looking at the code, problems are found and fixed quickly, frequently within a day. It is this author's experience that a closed-source vendor will take from a month to a year to fix many serious problems. An intelligent person does not leave burning candles unattended nor does she smoke in bed, but still installs smoke detectors and carries insurance. We look at many important steps to take in preparing for a possible intrusion and for detecting attempts and even the rare successful intrusion.1.12
Recovering from an Intrusion
The White House Web server was down for days after it seemed to have been broken into. We will look at some techniques to detect and recover even from a successful intrusion. These techniques will allow detection and recovery in only a few minutes with minimal loss of data. The White House system and many other large sites that were down for one or more business days following break-ins could have benefited from these techniques.
