Douglas Hubbard’s The Failure of Risk Management: Why It’s Broken and How to Fix It makes a claim that most internal auditors will find uncomfortable: that the risk assessment methods used by the majority of organisations, the scored heat maps, the high-medium-low ratings and the risk registers built on them, do not measurably reduce risk, cannot be shown to work, and in some respects perform worse than doing nothing, because they produce confidence without information. Hubbard is a management consultant and decision analyst rather than an academic, and the book is a practitioner’s argument that the alternative, calibrated probability estimates, Monte Carlo simulation and a focus on measuring the methods themselves, is neither exotic nor expensive. First published in 2009 and substantially revised in 2020, it is the book most likely to change how an audit function builds its risk assessment. This guide summarises the argument without reproducing it, defines the vocabulary, maps the method to audit practice, and ends with a worked example of a community bank’s audit function replacing its heat map.
In this guide
- The book in one page
- Who Douglas Hubbard is, and where the book came from
- The structure: three parts
- Why it is broken: the case against scoring methods
- How to fix it: the method in six steps
- The nine ideas worth keeping
- The vocabulary: fourteen terms defined for practitioners
- What it means for internal audit: from heat map to loss curve
- The criticisms, and where Hubbard overstates
- Editions, reading time and how to read it
- A one-hour discussion guide for an audit team
- Worked example: Lakeshore Bancorp replaces its heat map
- Frequently asked questions
- Related guides
The book in one page
| Question | The answer, in our words |
|---|---|
| What is the claim? | That the most popular risk management methods, qualitative scoring, risk matrices and the frameworks built around them, have never been shown to improve decisions, that there are structural reasons they cannot, and that better methods exist and are in use in fields such as insurance, engineering and finance. |
| What is broken? | Three things: the methods add no information (a “high” rating does not say how likely or how large); they introduce errors of their own (range compression, arbitrary scales, arithmetic on ordinal scores that cannot support it); and nobody measures whether they work, so failure is invisible. |
| What is the fix? | Replace scores with probabilities and ranges estimated by calibrated people, combine them with simulation to produce a distribution of possible losses, compare that distribution with the organisation’s stated tolerance, and track the method’s own performance against outcomes. |
| Is it practical? | Hubbard’s position is that a spreadsheet and a few hours of calibration training are enough to start, and that the organisations that object to quantification are usually already quantifying, badly, with numbered scores. |
| What does it not do? | It does not claim to predict individual events or to replace judgment; it claims that judgment expressed as a calibrated probability is better than judgment expressed as a colour. |
| Why read it as an auditor? | Because the annual audit risk assessment, the ERM assessment the function opines on, and the rating of every finding all use the methods the book attacks, and because the book provides a usable alternative rather than a complaint. |
| Reading time | About ten hours for the 2020 edition; the second part can be skimmed by readers already convinced, the third part cannot. |
Who Douglas Hubbard is, and where the book came from
Hubbard runs a decision-analysis consultancy and is best known for How to Measure Anything (2007), whose premise is that any quantity that matters to a decision can be measured well enough to improve the decision, usually with less effort than people assume. The Failure of Risk Management, first published by Wiley in 2009 in the aftermath of the financial crisis, applied that premise to risk: if risk is uncertainty about loss, then it is a measurable quantity, and the methods that avoid measuring it are avoiding the problem. The second edition of 2020 added material on cybersecurity, where Hubbard had co-written a companion volume, updated the research on the weaknesses of scoring methods, and responded to a decade of objections. The book sits in a lineage with the decision-analysis tradition of Stanford and the calibration research of Sarah Lichtenstein, Baruch Fischhoff and others, and with the critique of risk matrices published by the risk analyst Tony Cox, whom Hubbard draws on heavily.
The structure: three parts
| Part | What it covers | What it contributes |
|---|---|---|
| Part one: an introduction to the crisis | What risk management is for, the current state of practice, why it is hard to tell whether a method works, and a simple quantitative “straw man” model to start from | The frame: risk management is a decision method and should be judged as one |
| Part two: why it’s broken | The “four horsemen” of risk practice and how the weakest one won; the evidence against scoring methods and risk matrices; the psychology of overconfidence; the “analysis placebo”; why no one notices that the methods fail; the “risk paradox” (the biggest risks get the least rigorous analysis) | The diagnosis, with research citations an auditor can use in a report |
| Part three: how to fix it | Calibration training, decomposition, Monte Carlo simulation, loss exceedance curves, the value of information, and how to measure the method’s own performance | The method; the part that turns the book from critique into practice |
The “four horsemen” of part two are the book’s way of explaining where current practice came from. Actuaries measured risk for insurance with data and distributions; “war quants”, the operations researchers of the Second World War and after, built simulation and decision theory; economists built portfolio theory and option pricing; and management consultants built scoring frameworks that required none of the above. Hubbard’s argument is that the fourth tradition, being the cheapest to deliver and the easiest to understand, spread into general management, including the frameworks the audit profession adopted, while the first three stayed in their specialisms. The ERM guide and the Against the Gods summary cover the history he is compressing.
Why it is broken: the case against scoring methods
The second part is the one that generates arguments, and its claims are specific enough to list. Scoring methods ask people to rate likelihood and impact on ordinal scales, often one to five, and then add or multiply the scores. Hubbard’s objections, drawn from his own research and from Cox’s published analysis of risk matrices, are these.
| Objection | What goes wrong | An example an auditor will recognise |
|---|---|---|
| Range compression | A single score covers a vast range: a “4” for impact might mean anything from $1 million to $100 million, so two risks a hundredfold apart get the same rating | Two findings rated “high” where one threatens a quarter’s earnings and the other the firm |
| Arbitrary arithmetic | Ordinal scores are multiplied or added as if they were quantities; the result has no units and no meaning | A risk score of 12 compared with one of 9, with no one able to say what the difference represents |
| Inconsistent interpretation | Different assessors mean different things by the same word; “likely” is read as very different probabilities by different people in studies | The same control rated differently by two managers for reasons neither can explain |
| Rank reversal | Cox showed that matrices can rank a smaller risk above a larger one depending on where cell boundaries fall | A risk register whose order changes when the scale is redrawn |
| The analysis placebo | Structured discussion increases confidence without increasing accuracy; people feel more certain after a workshop regardless of whether the output is better | A risk workshop that everyone rates as valuable and that changes no decision |
| Overconfidence | Untrained estimators give 90 percent confidence ranges that contain the truth about half the time | Management’s point estimates of loss, presented without ranges |
| No feedback | Scores are never compared with outcomes, so a bad method is never caught | A risk assessment process running for ten years with no record of whether its “high” risks materialised more often than its “low” ones |
The strongest form of the argument is the last row. A method that cannot fail visibly cannot improve, and Hubbard’s contention is that the profession has adopted methods whose chief advantage is that they are immune to being shown wrong. The risk assessment guide describes a scoring approach, and this guide does not pretend the two are reconcilable without changes; the worked example below shows what the changes look like.
How to fix it: the method in six steps
- Define the decision. Risk analysis exists to inform a choice: how much to spend on a control, whether to accept an exposure, where to put audit hours. State the decision before assessing anything.
- Decompose the risk into events and consequences. Instead of “cyber risk: high”, name the events (ransomware on the core platform, loss of customer data) and what each would cost in a range.
- Calibrate the estimators. A few hours of training with feedback brings most people’s 90 percent ranges to containing the truth about 90 percent of the time. Hubbard treats this as the single highest-return step and provides the exercises.
- Estimate probabilities and ranges, not scores. For each event: a probability per year and a 90 percent confidence range for the loss if it occurs. Where data exists, use it; where it does not, calibrated judgment is the input, stated as such.
- Simulate. A Monte Carlo run of a few thousand iterations in a spreadsheet produces a distribution of total annual loss, from which a loss exceedance curve can be drawn: the probability of losing more than any given amount in a year.
- Compare with tolerance, and measure the method. The organisation states what probability of what loss it will accept (the risk tolerance curve); the gap between the two curves is what controls must close. Then record the estimates and compare them with outcomes, so that the method itself has a track record.
Two tools make the method practical. The first is the “one-for-one substitution”: the basic heat map can be replaced, risk by risk, by a probability and a range, so the change in process is smaller than it looks. The second is the expected value of information: a short calculation that shows which uncertainties are worth reducing before deciding, which is often not the ones people argue about. The KRI examples guide and the risk register guide show how the outputs fit into the documents an audit committee already receives.
The nine ideas worth keeping
- Risk management is a decision method, and should be judged by whether decisions improve. Satisfaction with the process is not evidence; outcomes compared with estimates are.
- If you use numbers, use real ones. A score of 4 is a number that means nothing; a probability of 20 percent and a loss range of $2 million to $8 million are numbers that can be checked, combined and compared.
- Calibration is a skill, and it is teachable. Overconfidence is the default and can be corrected in hours; an organisation that has never calibrated its estimators has never tested its inputs.
- Decomposition beats intuition. People estimate the parts of a problem better than the whole; a risk broken into events and consequences is assessed better than a risk named in two words.
- Simulation is a spreadsheet exercise. The method the book recommends was first developed at Los Alamos in the 1940s for nuclear weapons work and now runs in a few hundred cells; the objection that it is too sophisticated is an objection to the 1940s.
- Draw the loss exceedance curve. A single curve shows the whole distribution of possible loss, replaces the matrix, and can be laid over the organisation’s tolerance to show the gap at a glance.
- Measure the method. Keep the estimates, compare them with what happened, and adjust. A method with no track record is a belief.
- The biggest risks get the least analysis. The risk paradox: organisations model the price of a bond to four decimals and rate the risk of a merger “high” in a workshop. Audit effort should run the other way.
- Doing something is not better than nothing if the something is a placebo. A method that increases confidence without increasing accuracy makes decisions worse, because it removes the hesitation that uncertainty should cause.
The vocabulary: fourteen terms defined for practitioners
| Term | What Hubbard means | What it looks like in an audit |
|---|---|---|
| Risk | Uncertainty about a loss, expressed as a probability and a magnitude | The two numbers every rated risk should carry |
| Risk matrix / heat map | A grid of ordinal likelihood and impact scores | The thing to be replaced, cell by cell |
| Ordinal scale | A ranking (1 to 5) without meaningful intervals | Scores that cannot be added or multiplied |
| Calibration | The match between stated confidence and actual accuracy | Trained estimators whose 90 percent ranges hit 90 percent of the time |
| Decomposition | Breaking a risk into estimable parts | Events, frequencies and consequence ranges in place of a risk name |
| Monte Carlo simulation | Sampling from input distributions many times to produce an output distribution | A spreadsheet model of annual loss |
| Loss exceedance curve | Probability of exceeding each loss amount in a period | The chart that replaces the heat map in the committee pack |
| Risk tolerance curve | The organisation’s stated acceptable probability for each loss level | The line the loss curve must sit under; the board’s statement of appetite in usable form |
| Expected value of information | What it is worth to reduce an uncertainty before deciding | Which risks deserve a deeper audit before the plan is set |
| Analysis placebo | Increased confidence from analysis that did not increase accuracy | The workshop everyone liked and nothing changed |
| One-for-one substitution | Replacing each risk’s scores with a probability and range | The migration path from the current register |
| Range compression | A single score covering values orders of magnitude apart | Two “high” findings with incomparable consequences |
| Bayesian updating | Revising a probability as evidence arrives | Adjusting the likelihood after each incident or audit result |
| Risk paradox | The largest risks receive the least rigorous analysis | Where the audit plan’s hours actually go versus where the loss would be |
What it means for internal audit: from heat map to loss curve
The book was not written for internal audit, but three of the profession’s core activities are built on the methods it criticises: the annual risk assessment that produces the audit plan, the function’s opinion on management’s ERM, and the rating of findings. The table maps each to the change the book implies and to a proportionate first step. The Global Internal Audit Standards guide is relevant because the Standards require a documented, risk-based plan but do not prescribe the method, which leaves the function free to use a better one.
| Audit activity | Current method the book criticises | What the book implies | A proportionate first step |
|---|---|---|---|
| Annual audit risk assessment | Scoring the audit universe on likelihood and impact factors, summing to a risk score, ranking | Estimate, for each auditable unit, a probability of a material control failure and a loss range; rank by expected loss and by tail; record the estimates for later comparison | Keep the factor scores for one more cycle, but add a calibrated probability and range for the top twenty units and compare the two rankings |
| Opinion on management’s ERM | Assessing whether the ERM process follows a framework (workshops, register, heat map, reporting) | Assess whether the process has ever been shown to improve a decision: are estimates kept, compared with outcomes, and adjusted? | Ask management for the last three years of risk ratings and the losses that occurred; if no comparison exists, report that the method’s effectiveness is unmeasured |
| Rating of findings | High, medium, low by judgment, sometimes a matrix | A finding’s rating should state the probability the control failure leads to a loss and the plausible range of that loss | Add one line to the finding template: “If uncorrected, a loss of $X to $Y with a probability of about Z percent per year, estimated by…”; see the root cause guide for the causal part |
| Risk-based sampling and testing | Sample sizes from tables; coverage by judgment | Use the expected value of information: test more where reducing the uncertainty changes a decision | For the three largest planned engagements, ask what decision the result will change and size the work to that |
| Reporting to the audit committee | A heat map of the universe and a list of ratings | A loss exceedance curve for the assessed risks, overlaid on the board’s tolerance; a table of last year’s estimates against outcomes | Present both for one cycle; let the committee choose, as the audit committee playbook recommends for any change of method |
The practical constraint is skills. Calibration training takes a few hours and can be run from the book’s own exercises; building a simulation in a spreadsheet takes a day for someone comfortable with formulas; the harder change is cultural, because a register with probabilities invites challenge in a way a register of colours does not. That is the point. The professional skepticism guide describes the stance, and the writing for the board guide describes how to present a distribution to a committee that is used to a grid.
One more implication deserves its own paragraph because it is where the book bites hardest on the profession: the rating of findings. Every function rates findings, most on a three- or four-point scale, and the rating drives what the committee sees and what management must fix first. Hubbard’s method applied to a finding asks two questions a rating does not: if this control failure is not corrected, how likely is it to produce a loss in the next year, and how large could the loss be? Two findings rated high can have answers that differ by a factor of fifty, and the committee cannot see that from the colour. Adding the two numbers to the finding template costs a sentence per finding, forces the auditor to think about consequence rather than conformance, and gives the issue validation process a yardstick for which open items matter most.
The criticisms, and where Hubbard overstates
The book’s critics come from two sides. Practitioners of the frameworks it attacks argue that heat maps are communication tools, not analysis tools, that their value is in the conversation they structure, and that a board shown a loss exceedance curve will not read it; Hubbard’s reply is that a tool whose value is the conversation should be measured by whether the conversation changed anything, and that boards read curves in every other part of their business. Quantitative risk specialists argue from the other direction that Hubbard’s simulations are only as good as their inputs, that calibrated judgment about rare events is still judgment, and that Monte Carlo analysis of fat-tailed exposures can produce the same false confidence the book condemns in scoring, a point the Black Swan summary develops. Hubbard accepts most of this; his claim is comparative, that a stated probability with a track record is better than a colour without one, not that quantification is infallible.
Where he overstates is in tone and in scope. The research on scoring methods shows that they are inconsistent and that their arithmetic is meaningless; it does not show that organisations using them have worse outcomes than organisations using nothing, which is a harder claim to test and one the book asserts more than proves. And the method’s costs are larger in practice than in the book: estimators drift out of calibration, models accumulate assumptions nobody revisits, and a function of twelve people running a plan has limited hours for any of it. The proportionate reading is to adopt the method where the decision is large enough to justify it, and to be honest about the limits of the scoring approach where it remains.
Editions, reading time and how to read it
| What | Details | Best for |
|---|---|---|
| First edition | Wiley, 2009, about 280 pages | The original argument; superseded |
| Second edition | Wiley, 2020, with updated data and methods and new cases such as data breaches and natural disasters | The edition to read |
| Companion volumes | How to Measure Anything (3rd edition 2014) for the measurement method; How to Measure Anything in Cybersecurity Risk (2nd edition 2023) for the IT application | The practical toolkit |
| Online resources | The author’s site provides calibration tests and spreadsheet templates referenced in the book | Running the exercises with a team |
| Reading plan | Part one quickly, part two for the research citations, part three slowly with a spreadsheet open | An auditor who intends to use it |
| Companion reading | The Thinking, Fast and Slow summary for the psychology of overconfidence; the Superforecasting summary for calibration in practice | Context for the method |
A one-hour discussion guide for an audit team
- Ten minutes: the calibration test. Everyone answers ten trivia questions with a 90 percent confidence range before the session; score them in the room. The results, usually five or six out of ten, make the argument better than any summary.
- Fifteen minutes: translate three risks. Take three “high” risks from the current register and have the group state, for each, a probability per year and a loss range. Note how much the three differ once stated in numbers.
- Ten minutes: the feedback question. Ask whether anyone can name a risk the register rated low that materialised, or one rated high that never did, and whether anything changed as a result.
- Fifteen minutes: the first step. Choose one of the proportionate first steps from the audit map and decide who does it before the next planning cycle.
- Ten minutes: what to tell the committee. Draft the two sentences that explain to the audit committee why the function is adding probabilities to its assessment, in language the writing for the board guide would approve.
Worked example: Lakeshore Bancorp replaces its heat map
Lakeshore Bancorp is a fictional $9 billion community bank holding company with an internal audit function of twenty-two people. For eight years its audit universe of 140 auditable units had been scored on six factors, summed, and ranked; the top forty units were audited annually. In 2027 the chief audit executive ran a parallel assessment on the book’s method for the top thirty units, to see whether the ranking changed. Two auditors took the calibration exercises; the estimates were made by the auditors with the business owners present; the simulation ran in a spreadsheet.
| Auditable unit | Factor score and rank | Calibrated estimate (probability of material failure per year; loss range) | Simulated rank by expected loss | What changed |
|---|---|---|---|---|
| Wire transfer operations | 19 (rank 7) | 3 percent; $0.5 million to $25 million | Rank 2 | Moved up: the loss range’s tail was far larger than the impact score allowed |
| Consumer lending compliance | 22 (rank 2) | 15 percent; $0.2 million to $2 million | Rank 9 | Moved down: frequent but bounded; the score had compressed the range |
| Core system access management | 17 (rank 12) | 8 percent; $1 million to $40 million | Rank 1 | Moved up sharply: the factor model had no way to express an unbounded tail; see the ITGC audit guide |
| Branch cash operations | 20 (rank 5) | 20 percent; $20,000 to $300,000 | Rank 18 | Moved down: high likelihood, small loss, well controlled |
| Commercial real estate underwriting | 21 (rank 3) | 5 percent; $2 million to $60 million | Rank 3 | Unchanged in rank; the estimate made the committee’s discussion concrete for the first time |
| Vendor management | 16 (rank 15) | 6 percent; $0.5 million to $15 million | Rank 6 | Moved up: concentration in one core-processing vendor; the third-party risk guide cited |
Of the thirty units, eleven changed rank by five places or more, and two units outside the previous top forty entered the top ten. The function kept both rankings for the 2027 plan, audited the units that were high on either, and recorded every estimate so that the 2028 assessment could compare them with the year’s incidents and losses. The audit committee’s pack carried a loss exceedance curve for the thirty units beside the familiar heat map; two of the four members asked for the curve to continue and the map to go. The cost was about twelve auditor-days, most of it the calibration sessions and the estimating meetings, which the business owners described as the most useful risk conversations they had had with the function. That is the book’s argument in a sentence: the numbers are not the point; the conversation they force is.
Frequently asked questions
Is the book saying risk matrices are useless?
It is saying they have not been shown to improve decisions and that they introduce specific errors: range compression, arbitrary arithmetic, inconsistent interpretation and rank reversal. Hubbard’s word is “worse than useless” in the sense that they add confidence without information. The fair reading is that they are a weak method, not a harmless one.
Do I need statistics to apply it?
Calibration training needs none. A simple Monte Carlo simulation needs spreadsheet skills and a willingness to learn one or two functions. The book provides the exercises and templates; the method is less mathematical than most audit sampling.
How does this fit with the Global Internal Audit Standards?
The Standards require a risk-based plan built on a documented assessment and do not prescribe the method. A calibrated, simulated assessment satisfies them at least as well as a scored one, and the record of estimates against outcomes is evidence of the methodology review the Standards expect; see the Standards guide.
What about rare, catastrophic risks?
Hubbard’s method handles them better than a matrix, because a range can be wide and a probability can be small, but the inputs for rare events are still judgment. The Black Swan summary is the necessary counterweight: for unbounded tails, assess by scenario and build robustness rather than trusting a simulated percentile.
Can a small function do this?
Yes, for its largest decisions. Calibrate two people, estimate the top twenty risks in ranges, simulate in a spreadsheet, keep the record. The worked example cost twelve auditor-days for thirty units.
Where does the book sit in the reading list?
In the risk and decision-making group of the internal audit reading list, as the practical method beside Taleb’s critique and Kahneman’s psychology.
Related guides
- Internal Audit Risk Assessment: Method, Scoring and Template — the current method, and where the book’s changes attach
- Internal Audit’s Role in ERM — opining on a risk process that has never been measured
- The Risk Register: Fields, Scoring and Governance — the register with probabilities and ranges
- Key Risk Indicators: Example KRI Tables by Risk Type — indicators that feed the estimates
- The Black Swan: Summary and What It Means for Auditors — the limits of any estimate of a rare event
- Thinking, Fast and Slow: Summary and Review — why estimators are overconfident
- The Best Books for Internal Auditors — where this book sits among the others
- All book summaries — the full set of summaries for auditors
- How to Measure Anything by Douglas Hubbard: Summary and Lessons for Auditors — calibrated estimates and the value of information.
Leave a Reply