Chapter One

Me, a wizard? Until a week ago, I was an astronomer, contentedly designing telescope optics. Looking back on it, I'd lived in an academic dreamland. All these years, never planning for the future, right up to the day my grant money ran out.

Lucky for me that my laboratory recycled used astronomers. Instead of standing in the unemployment line, I found myself transferred from the Keck Observatory at the Lawrence Berkeley Lab, down to the computer center in the basement of the same building.

Well, hell, I could fake enough computing to impress astronomers, and maybe pick it up fast enough that my co-workers wouldn't catch on. Still, a computer wizard? Not me -- I'm an astronomer.

Now what? As I apathetically stared at my computer terminal, I still thought of planetary orbits and astrophysics. As new kid on the block, I had my choice of a cubicle with a window facing the Golden Gate Bridge, or an unventilated office with a wall of bookshelves. Swallowing my claustrophobia, I picked the office, hoping that nobody would notice when I slept under the desk. On either side were offices of two systems people, Wayne Graves and Dave Cleveland, the old hands of the system. I soon got to know my neighbors through their bickering.

Viewing everyone as incompetent or lazy, Wayne was crossthreaded with the rest of the staff. Yet he knew the system thoroughly, from the disk driver software up to the microwave antennas. Wayne was weaned on Digital Equipment Corporation's Vax computers and would tolerate nothing less: not IBM, not Unix, not Macintoshes.

Dave Cleveland, our serene Unix buddha, patiently listened to Wayne's running stream of computer comparisons. A rare meeting didn't have Wayne's pitch, "Vaxes are the choice of scientists everywhere and help build strong programs twelve ways." Dave retorted, "Look, you keep your Vax addicts happy and I'll handle the rest of the world." Dave never gave him the satisfaction of getting riled, and Wayne's complaints eventually trailed off to a mutter.

Great. First day on the job, sandwiched between two characters who were already ruining my daydreams with their periodic disputes.

At least nobody could complain about my appearance. I wore the standard Berkeley corporate uniform: grubby shirt, faded jeans, long hair, and cheap sneakers. Managers occasionally wore ties, but productivity went down on the days they did.

Together, Wayne, Dave, and I were to run the computers as a lab-wide utility. We managed a dozen mainframe computers -- giant workhorses for solving physics problems, together worth around six million dollars. The scientists using the computers were supposed to see a simple, powerful computing system, as reliable as the electric company. This meant keeping the machines running full time, around the clock. And just like the electric company, we charged for every cycle of computing that was used.

Of four thousand laboratory employees, perhaps a quarter used the main computers. Each of these one thousand accounts was tallied daily, and ledgers kept inside the computer. With an hour of computing costing three hundred dollars, our bookkeeping had to be accurate, so we kept track of every page printed, every block of disk space, and every minute of processor time. A separate computer gathered these statistics and sent monthly bills to laboratory departments.

And so it happened that on my second day at work, Dave wandered into my office, mumbling about a hiccup in the Unix accounting system. Someone must have used a few seconds of computing time without paying for it. The computer's books didn't quite balance; last month's bills of $2,387 showed a 75-cent shortfall.

Now, an error of a few thousand dollars is obvious and isn't hard to find. But errors in the pennies column arise from deeply buried problems, so finding these bugs is a natural test for a budding software wizard. Dave said that I ought to think about it.

"First-degree robbery, huh?" I responded.

"Figure it out, Cliff, and you'll amaze everyone," Dave said.

Well, this seemed like a fun toy, so I dug into the accounting program. I discovered our accounting software to be a patchwork of programs written by long-departed summer students. Somehow, the hodgepodge worked well enough to be ignored. Looking at the mixture of programs, I found the software in Assembler, Fortran, and Cobol, the most ancient of computer languages. Might as well have been classical Greek, Latin, and Sanskrit.

As with most home-brew software, nobody had bothered to document our accounting system. Only a fool would poke around such a labyrinth without a map.

Still, here was a plaything for the afternoon and a chance to explore the system. Dave showed me how the system recorded each time someone connected to the computer, logging the user's name, and terminal. It timestamped each connection, recording which tasks the user executed, how many seconds of processor time he used, and when he disconnected.

Dave explained that we had two independent accounting systems. The ordinary Unix accounting software just stored the timestamped records into a file. But to satisfy some bureaucrat, Dave had built a second accounting system which kept more detailed records of who was using the computer.

Over the years, a succession of bored summer students had written programs to analyze all this accounting information. One program collected the data and stashed it into a file. A second program read that file and figured how much to charge for that session. Yet a third program collected all these charges and printed out bills to be mailed to each department. The last program added up all user charges and compared that total to the result from the computer's internal accounting program. Two accounting files, kept in parallel by different programs, ought to give the same answer.

For a year, these programs had run without a glitch, but weren't quite perfect this week. The obvious suspect was round-off error. Probably each accounting entry was correct, but when added together, tenths of a penny differences built up until an error of 75 cents accumulated. I ought to be able to prove this either by analyzing how the programs worked, or by testing them with different data.

Rather than trying to understand the code for each program, I wrote a short program to verify the data files. In a few minutes, I had checked the first program: indeed, it properly collected the accounting data. No problem with the first.

The second program took me longer to figure out. In an hour I had slapped together enough makeshift code to prove that it actually worked. It just added up time intervals, then multiplied by how much we charge for computer time. So the 75-cent error didn't come from this program.

And the third program worked perfectly. It looked at a list of authorized users, found their laboratory accounts, and then printed out a bill. Round-off error? No, all of the programs kept track of money down to the hundredths of a penny. Strange. Where's this 75-cent error coming from?

0 Well, I'd invested a couple hours in trying to understand a trivial problem. I got stubborn: dammit, I'd stay there till midnight, if I had to.

Several test programs later, I began actually to have confidence in the mishmash of locally built accounting programs. No question that the accounts didn't balance, but the programs, though not bulletproof, weren't dropping pennies. By now, I'd found the lists of authorized users, and figured out how the programs used the data structures to bill different departments. Around 7 P.M. my eye caught one user, Hunter. This guy didn't have a valid billing address.

Ha! Hunter used 75 cents of time in the past month, but nobody had paid for him.

Here's the source of our imbalance. Someone had screwed up when adding a user to our system. A trivial problem caused by a trivial error.

Time to celebrate. While writing this first small triumph into the beginning pages of my notebook, Martha, my sweetheart, stopped by and we celebrated with late-night cappuccinos at Berkeley's Cafe Roma.

A real wizard would have solved the problem in a few minutes. For me, it was unknown territory, and finding my way around hadn't been easy. As a consolation, I'd learned the accounting system and practiced a couple obsolete languages. Next day, I sent an electronic mail message to Dave, preening my feathers by pointing out the problem to him.

Around noon, Dave stopped by to drop off a pile of manuals, and casually mentioned that he had never added a user named Hunter -- it must have been one of the other system managers. Wayne's curt response: "It wasn't me. RTFM." Most of his sentences ended with acronyms, this one meaning, "Read the fucking manual."

But I'd read the manuals. Operators weren't supposed to add a new user without an account. At other computer centers, you just log into a privileged account and tell the system to add a new user. Since we also had to make several bookkeeping entries, we couldn't run such a vanilla system. Ours was complex enough that we had special programs which automatically did the paperwork and the systems juggling.

Checking around, I found that everyone agreed the automatic system was so superior that nobody would have manually added a new user. And the automatic system wouldn't make this mistake.

Well, I couldn't figure out who had made this goof. Nobody knew Hunter, and there wasn't an account set for him. So I erased the name from the system -- when he complained, we could set him up properly.

A day later, an obscure computer named Dockmaster sent us an electronic mail message. Its system manager claimed that someone from our laboratory had tried to break into his computer over the weekend.

Dockmaster's return address might have been anywhere, but signs pointed to Maryland. The e-mail had passed through a dozen other computers, and each had left a postmark.

Dave answered the message with a noncommittal "We'll look into it." Uh, sure. We'd look when all our other problems disappeared.

Our laboratory's computers connect to thousands of other systems over a dozen networks. Any of our scientists can log into our computer, and then connect to a distant computer. Once connected, they can log into the distant computer by entering an account name and password. In principle, the only thing protecting the networked computer is the password, since account names are easy to figure out. (How do you find account names? Just use a phone book -- most people use their names on computers.)

Dockmaster's electronic mail message was a curiosity, and Dave passed it to Wayne, attaching a question, "Who's Dockmaster?" Wayne forwarded it to me with his guess -- "Probably some bank."

Eventually, Wayne bounced the message to me. I guessed Dockmaster was some Navy shipyard. It wasn't important, but it seemed worth spending a few minutes looking into.

The message gave the date and time when someone on our Unix computer tried to log into Dockmaster's computer. So I scrabbled around the accounting files, looking at Saturday morning's records. Again, the two accounting systems disagreed. The stock Unix accounting file showed a user, Sventek, logging in at 8:25, doing nothing for half an hour, and then disconnecting. No timestamped activity in between. Our home-brew software also recorded Sventek's activity, but it showed him using the networks from 8:31 until 9:01 A.M.

Jeez. Another accounting problem. The time stamps didn't agree. One showed activity when the other account said everything was dormant.

Other things seemed more pressing, so I dropped the problem. After wasting an afternoon chasing after some operator's mistake, I wasn't about to touch the accounting system again.

Over lunch with Dave, I mentioned that Sventek was the only one connected when Dockmaster reported the break-in. He stared and said, "Joe Sventek? He's in Cambridge. Cambridge, England. What's he doing back?" Turned out that Joe Sventek had been the laboratory's Unix guru, a software wizard who built a dozen major programs over the past decade. Joe had left for England a year ago, leaving behind a glowing reputation throughout the California computer community.

Dave couldn't believe Joe was back in town, since none of Joe's other friends had heard from him. "He must have entered our computer from some network," Dave said.

"So you think Joe's responsible for this problem?" I asked Dave.

"No way," Dave replied. "Joe's a hacker of the old school. A smart, quick, capable programmer. Not one of those punks that have tarnished the word 'hacker.' In any case, Sventek wouldn't try to break into some Maryland computer. And if he did try, he'd succeed, without leaving any trace."

Curious: Joe Sventek's been in England a year, yet he shows up early Saturday morning, tries to break into a Maryland computer, disconnects, and leaves behind an unbalanced accounting system. In the hallway I mention this to Wayne, who's heard that Joe's on vacation in England; he's hiding out in the backwoods, far away from any computers. "Forget that message from Dockmaster. Sventek's due to visit Berkeley RSN and he'll clear it up."

RSN? Real Soon Now. Wayne's way of saying, "I'm not sure when."

My worry wasn't Sventek. It was the unbalanced accounts. Why were the two accounting systems keeping different times? And why was some activity logged in one file without showing up in the other?

Back to the accounting system for an afternoon. I found that the five-minute time difference between the time stamps came from our various computers' clocks drifting over the months. One of our computer's clocks lost a few seconds every day.

But all of Sventek's activities should have appeared in both tallies. Was this related to last week's accounting problem? Had I screwed things up when I poked around last week? Or was there some other explanation?

Copyright © 1989, 1990 by Clifford Stoll