Customer Reviews

The Failure of Risk Management: Why It's Broken and How to Fix It

5 star:		(37)
4 star:		(2)
3 star:		(5)
2 star:		(3)
1 star:		(4)

 

 

How do we know if our risk management methods are working? Would we notice if they were not working? What are the consequences if they are not working? These are the three basic questions that Douglas Hubbard asks in his book The Failure of Risk Management.

In this book Mr. Hubbard lays out the basics of risk management and explains why many risk management methods are worse than useless. He also provides some ideas and first steps to fix the problem.

Here's a brief walk though 'The Failure of Risk Management':

Part I introduces the history of risk management and the problems with modern risk management methods. Independent events, for instance, are often times not independent at all. This common-mode failure is unaccounted for by many managers, yet can be devastating in an emergency.

Part II of the book goes in depth with some of the problems and failures of risk management, and to me was the most interesting part of the book. Chapter 4 is called The "Four Horseman" of Risk Management, and describes the differences between what the author considers the four main classes of risk managers. The four classes are actuaries, "war quants," economists, and management consultants. Each group has distinctly different methods and areas of expertise, as well as different levels of validation.

Chapter 5 is about how risk should be defined, and why different people may actually be talking about different things when they discuss volatility and risk. Chapter 6 breaks down why humans are not good at subjective methods (which lays the ground work for later chapters introducing quantitative methods). There are a few "calibration" tests available for you to see how overconfident you are in your decision making. These are pretty interesting, and even after reading about overconfidence I still did poorly on them.

Chapters 7, 8, and 9 talk about problems with subjective scoring methods, problems with describing one-off events, and the problems with some quantitative models. The author talks about "black swans," as described by Nassim Nicholas Taleb, and how they relate to modeling. Many times people believe that events can't be modeled, but the author believes this is not so.

The last section of the book, Part III, gives some ideas on how to fix risk management. Adding empiricism is a big start, as well as calibration of subjective human inputs. Many companies build and use models, but then they don't actually bother to see how well the things have performed in the past. I will leave the rest of the solutions for you to read in the book.

Recommendation:

First off, the author says this book is geared towards all types of risk management, and all types of industries, and I think this is true. The author uses a wide variety of examples from airplane engine failures to volcano eruptions. But I still feel like this book is more geared towards enterprise risk management, and less towards the already quant heavy fields such as actuarial science or credit risk management. But it was an interesting read nonetheless.

It seems like in the past 20 years there have been several so-called "once-in-a-lifetime events," such as the floods of Hurricane Katrina or any of the financial crisis, including 1987, 1998, 2000, or 2008. I wish I had the money to buy this book for every person who ever said "no one saw that coming."

I think this is a great book for anyone who deals with the potential for risk, loss, or damage - no matter if it is financial, personal, or physical. When the stakes are high we should be careful relying on a risk matrix developed by a management consultant, and Douglas Hubbard will tell you why. If you work in risk management, or if you have influence on the operational strategy of some organization, then this book is a must read.

I have been involved in business consulting, investment management, business valuation and corporate governance for most of the past 25 years, and I can say without hesitation that Doug Hubbard's book on The Failure of Risk Management is an outstanding and elucidating work. I have never been a risk manager per se, but I have frequently been deeply involved in risk assessment and risk management activities, so I do have firsthand experience in this topic.

This book is an eye opener from the outset. In Part One of his book ("An Introduction to the Crisis") Hubbard begins with fundamental, obvious questions about risk management that everyone (not just risk managers!) should be asking. For example: How do you know that your risk management program is effective? Would anyone in your organization know if your risk management program didn't work? (...and how would they know - and define - that it wasn't working?). These are very simple, obvious questions, yet I have never heard them asked by management teams or even members of boards where I have served as director. Alas, there is a huge "placebo effect" in so much of what passes for risk management nowadays - perhaps that is why it is so popular.

For example, consider the following: If risk management programs really do work, then it seems logical to assume that companies in a given industry with a (self proclaimed) "highly effective" risk management program would show greater shareholder returns, less earnings volatility, and better safety and regulatory compliance records than other companies in their peer group who lack such a program. Yet there appears to be no valid evidence that current risk management practices, taken as a whole, serve to improve overall corporate performance. The evidence just isn't there.

In Part Two ("Why It's Broken"), Hubbard provides a thorough and convincing overview of the many shortcomings of modern risk management practices. As a self proclaimed "Quant," he strongly endorse quantitative analytics as the most effective approach to both measuring risk as well as the implementation of risk management programs. His approach is compelling and convincing; after all, if we can't measure accurately, how can we rely on our system of "assessing" (i.e., measuring)? It sounds pretty obvious, doesn't it? Without metrics, what tools do we have, other than generalizations, hunches, intuition, and "gut feel"? Sure, certain qualitative techniques are helpful, but qualitative risk analytics is really effective (in my view) only for the most obvious risks, and therefore no better than having no risk management program at all. Indeed, Hubbard makes a compelling argument that ineffective risk management can be worse - possibly much worse - than having no risk management program at all.

Part Two also includes concepts that Hubbard brilliantly applies to risk management practices. This includes certain characteristics of human nature, such as a proven tendency to be overconfident in our estimates (of risk, but also of other estimates), that must be acknowledged and addressed in order for risk management programs to work effectively. He also provides a practical method of adjusting or "calibrating" for such overconfidence. Similarly, there is a fascinating discussion on risk correlations and how risk events seldom materialize in isolation from one another. Consider (my own example) certain risk correlations in mortgage banking. Banks that invested in mortgage backed securities no doubt undertook some sort of risk analysis of these investments. They also had risk management systems in place for their mortgage lending business. But how many lenders tied these two risk programs together, and properly concluded that a collapse of one market would also result in the collapse of the other? Thus, it's not just a case of accurately assessing and management individual risks, but also in considering the extent to which there might be a "domino" or "cascading" effect among different risk factors.

In reading Part Two (especially Chapters 6 and 9), it occurred to me that this book should be read by anyone and everyone involved in investing or lending money.

As one might expect, Part Three of Hubbard's book ("How to Fix It") embraces a scientific and quantitative approach to improving risk management. Once you get to this point in the book, you will find it very difficult to disagree. Another important concept introduced by Hubbard is that of language and communication with respect to risk. As a potentially murky and subjective topic (if not downright Byzantine at times) risk management systems require clear and concise language and terminology to be effective. Thus, if two different managers in the same factory concur that the likelihood of a risk event materializing is "very likely," we should not assume that they both agree on the use of the term "very likely." One may feel that this means the odds are one in three, while the other feels the odds are one in ten.

Hubbard is clearly on target when he proposes that risk managers apply scientific methods to risk management. His suggestions on how to do this are fairly simple and practical. Without such methodologies, risk managers are sailing through dense fog with an unreliable compass. You might even feel that you are making great headway, but if you can't measure where you are going, you will never know if you are really making any progress.

Finally, one of the greatest benefits to me in reading this book has less to do with the specifics of risk management and a lot more to do with the way people think. Consider, for example, why your sales team frequently falls short of their sales projections, or why so many portfolio managers buy stocks near their highs and sell near their lows. Or why risk management programs are so popular, and yet seldom work. Hubbard provides a brilliant and penetrating look into the human mind in the context of business decision making as a whole - not just with respect to risk. For me, this was an excellent "upside surprise" to this book. I finished reading this book several months ago, and I still think about it all the time. It has made a lasting and beneficial impression that I will never forget.

This book is a must for every professional, undergraduate, graduate or post-graduate student dealing with risk management. Douglas Hubbard manages to combine proper mathematics with the basics of measurement and still keeping things within reach of an audience that does not necessarily has to have too much mathematical skills. Speaking of experience, I started as a PhD in Physics using Monte Carlo for simulating Magnetic forces in semiconductor interfaces. Then I transposed these methods more than 20 years ago to medical equipment, did some work in safety, environment, food hygiene and finally ended up in innovation and entrepreneurship. All of these tracks have an intensive relation with risk. I saw many of the errors (and even more) in risk management as (nearly literally) described in this book. So the level of relevance is there.

The treatment in three parts is well done, the structure is both professional and inviting to read more. The skill Douglas Hubbard apparently has in combining almost prosaic phrases with good scientific content, makes this book to be a reference book and a novel at the same time. An example for many of us (including me) that do not have this skill. I applied already formerly likewise approaches but with the reading of the book, I succeeded in leaving some very heavy (and expensive) calculation programs for the marvelous and illicit Excel sheets Douglas is posting on his website, at least for some applications. As a tutor I take the book of Douglas and leave the "heavy programs" for later on. This "step up" is amazing for students as they get gradually into the complexity of the matter.

I read some of the negative critics and think some of the people did not read the book properly. Of course when you are used to make large qualitative studies, I can imagine this book is at least a bit "annoying". But as I will always remember the quote of one of the Top Risk assessors and managers of Philips Medical Systems in Holland, "measurement is knowledge and when someone pretends to have a better risk assessment and/or risk management, let him prove to effectively be better". I love the way Douglas Hubbard takes these principles in to real life practice. I applied the risk approach as described in the book already many times and it does work amazingly well.

I strongly recommend this book to professional, tutors and students (getting) involved in risk (as well the risk assessment as the risk management). It can be applied to several domains of risk such as but not limited to: clinical trial, environmental risk, general business risk, safety of products risk, risk of medical equipment, food-safety risk, innovation & entrepreneurial risk (business plan or business case risk). That's were I already applied it with success.

Prof. Dr. Johan Braet, Antwerp University, Faculty of Applied Economics, Department of Environment and Technology, Innovation Management & Entrepreneurship, Risk Assessment (LCA) & Risk Management

By way of back ground I am an actuary (FCAS, ASA, MAAA) and a Chartered Enterprise Risk Analyst (CERA). I live in the world of risk quantification and really enjoyed and agree with Hubbard's comments on the subject. I also think he does a great job of expressing quantitative concepts in a manner that is accessible to non-quants.

One key theme that I related to was how more subjective or qualitative methods can get you into trouble. I often see this when performing modeling studies around risk for my clients. It's not uncommon for companies to take a look at the first time forecasts I perform and state "there is no way it can get that bad." They are saying this because they have already come up with a subjective estimate without really digging into the numbers. Even as time passes and the numbers do get "that bad" many are still so anchored to their original subjective assessment of the risk in question that they will continue to argue against the quantified trends.

One comment I often hear from clients is "You can't model that." My rebuttal has always been, "If you are making a decision, I assert you are modeling it. In you head you are putting together scenarios and thinks up consequences." Why not take it out of your head and try and put more rigor into your estimate? Hubbard uses the phrase "better is good" and I fully agree.

Other great points he makes include:
- Organizations are currently not consistently quantifying risk - real life examples I have seen can be as basic as different divisions using totally different interest or commodity assumptions in the decisions they make
- Qualitative risk ranking can get you into trouble
- There is a definite opportunity to take actuarial type standards to modeling insurance risk and apply them to other operational, financial and strategic risks that organizations face
- His four risk modeling fallacies (chapter 8)
- To understand risk, you need to understand the distribution of potential outcomes.

To my "You can't model that" comment above, I have picked up another of his books, "How to Measure Anything" and am really enjoying it as well.

This book provides a manifesto that should be adopted by anyone whose desk is ever touched by a risk management plan.

Douglas Hubbard opens the third part of the book, `How to Fix It', with a summary of three key improvements to risk management:
* Adopt the language and the philosophy of modelling uncertain systems;
* Be a scientist;
* Build the community as well as the organization.
I wish I'd had those on the wall of my office years ago together with the knowledge from reading the book to put them into practice. And I wish I'd had copies of the book with me to make it required reading for every board, audit committee and internal auditor I've encountered.

I've sat through a few presentations of risk management reviews or plans, each with its traffic-light-coloured matrix. They have often invited the John Cleese riposte "Can't we get you on Mastermind, Sybil? Next contestant, Sybil Fawlty from Torquay, special subject: the bleeding obvious." It was hard to avoid thinking they were rote observances for regulatory compliance boosted by the marketing puff of those contracted to do them. This book presents a refreshing alternative in stark contrast to that experience.

If you're a seasoned actuary or well versed in decision analysis, you may already know the techniques of statistics and probability Douglas Hubbard describes. There are two reasons why the book may still be of value to you. First, what makes Hubbard's approach unusual is his eclecticism. He draws on material from a wide range of sources including research from behavioural economics and experimental psychology to develop a bag of practical tools that should appeal directly to managers. Secondly, if Hubbard is right, then you'll be shocked at what has been peddled to the rest of us as risk management and you'll need tools for the urgent proselytising you should be doing to bring us up to speed and remove the flummery.

Hubbard's target audience is management. Although he covers some interesting history of approaches to risk and provides critical analysis of other writers from Frank Knight to Nassim Taleb, his focus is practical. His object is to reach people who must make decisions or steer projects. As he says in the preface:
`much of what has been done in risk management, when measured objectively, has added no value to the issue of managing risks. It may actually have made things worse.'
Much current practice, he says is `no better than astrology.' Although specialists and academic researchers know the problems, the heady quantitative texts they write don't reach the wider management audience. Hubbard's aim is to give `just enough technical information' to expose the problems and propose alternative relevant and useable tools. In that, he succeeds. The book gives enough both to guide the first steps of positive change and to point to useful directions for further study.

It's sobering to realise how very much better my own management and decision-making could have been, but exciting to contemplate the scale of improvement possible at the level of industries and national economies.

No doubt there are tweaks in presentation that might be made for future editions, but for the impact of the ideas alone, it's a five.

Doug Hubbard follows up his smash success on measurement with a penetrating look at why some of the most popular methods of risk assessment not only don't work, but are worse than no analysis at all. He argues convincingly, based on after-the-fact surveys of how various companies' decisions worked out, that "balanced scorecard" approaches typically make management feel much more confident while making the actual outcome no better, and often worse. The Analytical Hierarchy Process (AHP) fares almost as badly. Most telling is his finding, again backed by a cogent array of survey and interview data, that the purveyors of supposedly empirical methods rarely empirically study how well their methods work! He concludes by offering several improved methods to assess risk, understand what data would be most valuable to collect, correctly estimate and report one's uncertainty about predictions, and thus place risk management on a much more solid analytical footing.

He builds up to his conclusions with a competent and readable review of relevant applied probability, statistics, and simulation modeling, then a summary of what has generally been done in real decision-making in recent years. None of these sections is totally comprehensive or completely rigorous, but they don't need to be. He strikes a nice balance between readability and careful explanation of what the typical decision-making reader, somewhat knowledgeable but not highly technical, needs to know. The quantitatively naive manager will find it enlightening, if sometimes daunting; the quantitatively expert reader will find cause to quibble here and there, but no major mistakes or crucial omissions.

Warning: many of the established pooh-bahs associated with risk assessment as it's done now won't like this book. As he predicted, actuaries will continue to prefer methods firmly, even if dangerously, rooted in past experience; operations research analysts and their ilk ("war quants," he calls them) will continue to prefer increased modeling elegance over better metrics on which to base the models; economists will continue to rely on far too many unsubstantiated assumptions; and, worst of all, management consulting snake oil salesmen will continue to construct elaborate, impressive-sounding "methodologies" with lots of buzzwords and procedures and little verifiable empirical content. All of the above will try to reassert the excellence of their approaches and deprecate or marginalize Hubbard's criticisms. Believe them at your peril: Hubbard got it pretty much right. His findings and recommendations are most surely not the final, definitive answer, but they're a large promising step in the right direction. Highly recommended for anyone interested in doing risk management competently.

I work in the mineral exploration industry where managing risk is a daily aspect of life as we know that more than 95% of the time an exploration project will fail. Therefore, like others in my industry (which spends about $10 Billion a year, globally)I am very interested in any insights that I can glean from the general risk-managament literature that might help me and my business. This was the reason that I bought Douglas Hubbard's book and I was not dissappointed in finding some gems that could help me. In particular, I found the discussion about calibrating expert judgment to be very enlightening and the concept that calibration is not domain-specific to be a very interesting insight. Hubbard's general approach to the philospophy of measurement I also found very helpful. When you work in a highly uncertain domain, it is very easy for people to totally reject any form of quantitative analysis. However, Hubbard makes the very profound comment that it is actually in areas of highest uncertainty that we get the most value (ie in terms of improving our business decision making)with the least amount of additional measurement. I have recommended this book to many others in my industry and have also now purchased Hubbard's early book.

Over the first 15 years of my business career, I thought of a few key answers as unmeasurable. Doug Hubbard's books taught me that I was wrong and that what were thought of as "impossible" measurements could actually be measured. I highly recommend this book to everyone from Academia to the highest level business executives. You must learn how to measure "intangibles" and how to tell that your method of analyzing big decisions "works" (i.e. has a measurable improvement for your forecasts and decisions) Hubbard's books and webinars have given me the confidence to trust proven, quantitative methods even when, there is ostensibly a huge lack of sufficient data. He makes it easy to "Calibrate" experts to realistically assess probabilities and remove inconsistencies in judgment. I can't "Do the math" but now I don't have to rely on intuition alone, not entirely anyway. This book and his first (How To Measure Anything; Finding The Intangibles in Business) could make a huge impact on any business or individual especially those in fields that don't typically practice risk management, like IT management and even marketing and business development.

Doug Hubbard has accomplished what numerous seminars and courses on the quantitative risk management of projects attempt. Too many texts claim to define risk management, but only apply what Mr. Hubbard describes as a myopic approach. In this book he approaches the subject from a broader perspective and successfully defines a useable, common framework for a brader range of domains such as finance, engineering and Government systems acquisition. I agree with the author that scaling methods of risk analysis are seriously flawed.
What readers will found most helpful are the definitions of risk and uncertainty - two definitions to which I adhere- as well as the description and solution to the problem of current, doctrinal risk analysis. This will help in the translation of risk and uncertainty analysis imported from specific domains. I also believe Mr. Hubbard is on the right track by applying useful actuarial statistics to assist in the quantification of the uncertainties of estimates.
Overall, this an excellent book that every project manager should own and reference.

R. Covert
Technical Director, Cost and Schedule Practice, MCR, LLC
Former Chair Space Systems Cost Analysis Group Risk Subgroup

Douglas Hubbard does a great job of exposing the flaws in the vast majority of "risk management" approaches used by both companies and government agencies. He makes a strong case for increasing the level of mathematical rigor that goes into the analysis of risk - something that is often resisted by organizations only because they don't believe that many risks can be quantified. If risk management is to be taken seriously at the highest levels of the organization, and in most cases I would argue it's not, then we have to be able to answer the core question in this book - how do you know that risk management is working. The author deserves a lot of credit for framing the problem using that question.