Customer Reviews

Measuring and Managing Performance in Organizations

5 star:		(14)
4 star:		(0)
3 star:		(0)
2 star:		(0)
1 star:		(0)

 

 

This book provides an amazingly convincing explanation for why employee incentive programs often do more harm than good. It's often because knowledge work is too complicated to benefit from any simple measures.

The core argument of the book uses some mathematical reasoning that will be accessible to anyone who stayed awake through Economics 101. This is illuminating enough, but then Austin continues to add on additional insights.

I've placed this book on my shelf next to The Logic of Failure (Doerner) and Normal Accidents (Perrow). All of these books provide solid scientific arguments for the limits of management.

As a software tester, the most obvious application of the book is as an explanation of exactly when counting defects (found by testers, or introduced by programmers) is likely to lead to trouble.

Robert Austin presents an idealized model of a managed organization. Instead of looking at an organization made up of thousands of employees and a few hundred managers arranged in a hierarchy, Austin's model consists of three participants: a principal, i.e. a manager, and an agent, i.e. an employee, and finally a customer who buys the goods or services provided by the agent under the supervision of the principal.

He also assumes that an agent's job consists of two activities and the customer is happy if the agent performs well in both. Austin looks at the cases where the principal can monitor neither of the two activities, where she can monitor only one of the two activities, or where she can monitor both activities. According to the model the agent will behave differently in all three cases.

If the principal cannot (or will not) measure either activity, then we have delegated management, if she can measure both activities, then we have a fully supervised model, and if she can measure only one of two activities, we have a dysfunctional model.

When delegating management, the assumption is that agents want to work well, that they are not deriving maximum satisfaction by exerting the least amount of effort.

When supervising, the principal evaluates overall performance by measuring certain aspects of the agent's activity. Austin's conclusion is that measuring performance won't work unless you can measure everything employees should be doing (i.e. full supervision). Incomplete measurement is not only useless, it is dangerous since it motivates agents to make efforts only for what is measured.

For example, if a help desk line measures performance by the number of calls an employee takes, then employees are motivated to spend very little time per call. The customer is left dissatisfied, but the measurements show that the agent is providing first class results. Austin calls this situation dysfunctional.

Throughout the book, Austin emphasizes dysfunction to the point where it seems he dismisses any and all attempts at measurement, but to quote Austin, the central message of the book is that "organizational measurement is hard". It's not impossible.

He suggests one method, probabilistic measurement, to mitigate dysfunction. For instance, if dysfunction comes from being unable to measure everything an agent does, e.g. you just can't have your supervisors listen to all help desk calls, the principal can carry out random samplings of performance, e.g. you can record all the calls and listen to a random selection of them each day. The agent will then expend effort along those dimensions that cannot be completely measured simply because he knows they might be.

All in all, an effectively simplified model of organizations sure to spark healthy and constructive debate.

Vincent Poirier, Dublin

In my role as a methodologist, business process modeler, and designer of metrics and measurement programs I have long been concerned with the preverse and unanticipated effects of such measurement programs. For the first time Austin has identified what is going wrong with most types of measurements and offers a model for how to correctly construct a non-disfunctional approach to measuring things in the real world. I now understand what is wrong with the Consumer Price Index, why my marriage failed, and a lot of other inexplicable things about the world around me. I would urge every manager and professional to read at least the first few chapters of this book in order to understand the tremendous harm incorrect measurement can do and how collect and use measurements properly

This book is not - a light read - long - mathematical - about software specific issues and the arcana of that discipline - a cookbook for deciding what to measure, how to measure, how to analyze, how to report

This book describes - the uses of measurement, informational vs motivational - a (increasingly elaborated) measurement model - an objective definition of dysfunction and how it arises because of measurement - a model of "supervision" and how measurement supports (or interferes with) various kinds of supervision - a suggestion about organizational incentives - some strengths & weaknesses of well known assessement systems; e.g., ISO, SEI - the interview method and answers applying the model with 8 well-known writers on software and software management issues.

The messages I got - setting up measurement systems is not easy. There are many pitfalls - picking the goal(s) that the measures will support is critical - picking the measures. Some things are too expensive to measure - deciding how much to spend - deciding what to report to whom - (to my own chagrin) that I had personnally and fully encountered most pitfalls - it's easy for those measured to subvert the measuring - partial measurement may make things worse - informational measurement (measuring and results stay with those measured) is less likely to be subverted - purely economic models are not fully adequate explanations of employee-employer relationships.

I teach courses on software metrics and do some research on software-related measurement. As Austin points out in his book, many of the well-known advocates of metrics in the software community are blind to the issues that he raises, or they dismiss the issues as social science hooey that won't affect serious engineering. They are so, so wrong. This is a useful, readable book, that teaches hard lessons.

While research the issue of developer productivity and metrics for a client during this past week, I ran across a reference to this book, so I bought it. I started reading it on a cross-country flight and finished it that same day.

It was a revelation.

I have long been leery of most metrics used in software development, particularly when used to motivate (or punish) developers. This is particularly true since I've been working since the mid-1990s both as a consultant to organizations with troubled IT projects and as an expert witness in lawsuits that involve troubled, disputed, or failed IT projects. Developing software is hard, and measuring what's been accomplished in useful ways is even harder. But my objections have been largely intuitive, observational, and anecdotal.

Austin, by contrast, brings tremendous rigor and a remarkable depth and breadth of research to his approach. His use of the principal-agent model is particularly effective; I will probably be sketching figure 9.2 (or related figures) for years to come.

This is not an easy book to read; it's not a collection of breezy success (or failure) stories with some extrapolated maxims to explain them. It is almost like reading a work of philosophy, but there is nothing abstract or ethereal about it. I learned much, even as I had my own observations and suspicions confirmed. It has my strongest recommendation.

If you thought that performance review processes and incentive systems are often flawed, but wanted to understand why, Austin's book will give you a model to understand dysfunctional measurement processes, and a path to understanding how to avoid the dysfunctions and really help organizations and teams perform in a way that improves quality and gives customers what they want. Though the core of this book is an economic model, the book is quite readable and enjoyable. While this book was written before many people had heard of agile or Scrum, the model in the book also helps you understand why the approach of self-organizing teams that agile methods advocate can be very effective. Many of the conclusions in this book seem like common sense, but like many things, common sense is not synonymous with common practice, especially among people want to measure things. If you manage people consider reading this book to get a deeper understanding about incentive systems in addition to Behind Closed Doors: Secrets of Great Management (Pragmatic Programmers) which will help you to learn how to effectively manage people day-to-day to improve performance.

This book is filled with both humorous and chilling examples of measurement dysfunction that make the sometimes academic approach quite palatable. Dr. Austin identifies three different types of performance measurement based on the intent of the measurement - measurement for motivation, process improvement, or process coordination. It is measurement for motivation that causes the dysfunction that this book so convincingly describes.

For example, if we record the fact that 10 widgets are produced on machine A and we are comparing this against the 10 widget benchmark for bonuses, it is very likely that other perspectives like quality will suffer in the drive to make the 10 widget goal. Austin makes the point that the discovery that every time our overall performance is excellent we have produced 10 widgets does not imply that producing 10 widgets will guarantee excellent overall performance.

If we record the fact that 10 widgets are produced on machine A while only eight widgets are produced in the same time using competing technology on machine B, this is measurement for process improvement and can be very useful - provided it is limited in scope and used purely for the stated purpose.

If we record the fact that 10 widgets are produced on machine A and convey this information to the widget packaging department to ensure that enough widget cases are ready, this is measurement for process coordination, and is also potentially useful on its own.

The idea that the intent or goal of the measurement is of paramount importance is one important lesson from this book.

Austin does make some recommendations about developing effective performance measurement systems.

Understanding the costs involved with "perfect" measurements is part of the solution. Substituting a cheaper approximation for a key measurement is bound to cause problems - witness the measurement of nitrogen instead of protein in wheat gluten used in pet food. The incorrect justification for cheaper approaches is a thread surfaces in other areas - reusing financial figures as a proxy for management accounting leads to flawed descision-making emphasizing short term financial gain - reuse of software components leads to products that are hard to use.

One effective technique is using the end customer as the ultimate judge of quality and performance - the kind of approach described 10 years later in Fred Reichheld's The Ultimate Question: Driving Good Profits and True Growth

A principal of a company once told me that the primary job of a manager is to get the employee to do what the manager wants him to do. From there to effective management in real life comes a lot of confusion.

Robert Austin sorts it all out with a surprisingly simple model, and a strong does of honesty. Managers and workers -- participants in the serious game of work in organizations -- put aside illusions and read this book. And anyone who thought they were helping by designing a measurement program, pay attention too.

Measurement and management can work, but only if you know what you're doing.

No, really. I've never looked at the world the same since reading this book many years ago. Austin, a former executive at Ford Motors Europe and now on the faculty of Harvard Business School, applies agency theory, an application of game theory, to incentives and performance measurement. Sounds dry, but this book is short, readable, and to the point. It explained so much of the dysfunction I see around me in both work and life in general. I ended up giving a couple of talks about it, recommending it to others, blogging about it (I write under the name "Chip Overclock"), etc. I'm a product developer by trade, and this book applies directly to how large product development organizations are run, or mis-run. I was found it so fascinating that I ordered a copy of Austin's Ph.D. thesis, from which this book was derived, and read that too.