Douglas W. Hubbard’s How to Measure Anything: Finding the Value of “Intangibles” in Business is a book about a sentence internal auditors hear in closing meetings: that cannot be measured. Hubbard’s answer, developed over three editions since 2007, is that the sentence usually rests on a misunderstanding of what measurement is. A measurement, in his definition, is an observation that reduces uncertainty and expresses the reduction as a quantity; it does not have to produce an exact number, only to leave the decision-maker less uncertain than before. Anything that matters to a decision has consequences that can be observed, and so it can be measured well enough to improve the decision, usually with less data than people assume. The book then supplies a method for deciding what is worth measuring, built on the economics of information: a measurement is worth what it saves in the cost of being wrong. For internal auditors the book matters in three places. It speaks to the claims the function audits, because business cases, performance reports and risk assessments are full of benefits and exposures described as intangible. It speaks to the function’s own work, because the annual risk assessment, engagement scoping and the rating of findings are all exercises in measuring under uncertainty. And it speaks to a question audit committees ask about the function itself, which is how its value can be shown. This guide summarises the third edition (Wiley, 2014) without reproducing it, sets out its structure from the printed contents, explains its method and vocabulary, records what reviewers have found wanting, maps the method to audit practice and ends with a worked example in which a water authority’s audit function tests a business case built on intangible benefits. The site’s summary of Hubbard’s The Failure of Risk Management covers his case against risk matrices and his treatment of risk analysis; this guide does not repeat it.
In this guide
- The book in one page
- Who Douglas Hubbard is, and how the book came about
- The structure: four parts and fourteen chapters
- Part I: why the immeasurable usually is not
- Part II: the decision, the estimate and the value of information
- Part III: decomposition, small samples and Bayes
- Part IV: preferences, human judges and new instruments
- The nine ideas worth keeping
- The vocabulary: fourteen terms defined for practitioners
- What it means for internal audit: measuring what the plan depends on
- The criticisms, and what the book leaves out
- Editions, companions and how to read it
- A one-hour discussion guide for an audit team
- Worked example: Millbrook Regional Water Authority tests a business case
- Frequently asked questions
- Related guides
The book in one page
| Question | The answer, in our words |
|---|---|
| What is the claim? | That things described as intangible, such as quality, reputation, flexibility, security or the value of information itself, can be measured well enough to support a decision, and that the belief that they cannot comes from misconceptions about what measurement is. |
| What is a measurement? | An observation that reduces uncertainty and expresses the reduction as a quantity: a narrower range, a changed probability. It need not eliminate uncertainty, and a rough result can be worth a great deal when little was known before. |
| Why do people think things are immeasurable? | The book traces the belief to misunderstandings of the concept of measurement, of the object being measured and of the methods available, and then answers economic, statistical and ethical objections in turn. |
| What is the method? | Define the decision; state what is known now as calibrated ranges and probabilities; compute the value of additional information for each uncertain variable; measure the variables where that value is high; decide. Hubbard calls the full approach Applied Information Economics. |
| What should be measured? | The few variables whose uncertainty actually threatens the decision. Hubbard reports that what organisations measure and what they need to measure are usually very different, a pattern he calls the measurement inversion. |
| How much data is needed? | Less than people think. A random sample of five gives a 93.75 percent chance that the population median lies between the smallest and largest values observed, and when nothing is known about a proportion, a single random draw has a 75 percent chance of coming from the majority. |
| What did the third edition add? | By the author’s account, cases from newer clients, answers to the challenges readers had raised since 2007, calibration results from more than 900 people his firm had trained, Philip Tetlock’s forecasting findings, and more on big data, social media and personal measurement devices, with a companion workbook. |
| Why read it as an auditor? | Because audit work is measurement under uncertainty with a budget, because the book gives a defensible way to decide where audit effort reduces the most uncertainty, and because it equips the auditor to challenge the claim that a benefit or a risk is too intangible to evidence. |
Who Douglas Hubbard is, and how the book came about
Hubbard worked as a business analyst at the accounting firm Coopers and Lybrand, as Computer Weekly’s reviewer noted, and later founded Hubbard Decision Research, which he runs as president and chief executive. He is the creator of Applied Information Economics, the method at the centre of the book; his firm describes it as a synthesis of techniques from economics, actuarial science and other quantitative fields, drawing on decision theory and quantitative risk analysis, and says organisations have been using his methods since 1997. In an interview with Washington Technology after the first edition appeared, he described the method as a combination of quantitative techniques originally assembled for decisions about information technology, and recalled analysing systems at the US Environmental Protection Agency whose productivity benefits had been counted while harder benefits such as public health had been nearly left out. In our reading, the book generalises that experience: what is left out of a business case is usually left out because someone has decided it cannot be measured.
The first edition was published by Wiley in 2007, the second in 2010 and the third, the current edition, in March 2014. In an article for the INFORMS magazine Analytics in June 2014, Hubbard and Douglas Samuelson described what the third edition added: cases from newer clients, answers to the challenges readers had raised in the seven years since the first edition, findings from Tetlock’s study of more than 82,000 forecasts by 284 experts, calibration data from more than 900 people trained by Hubbard’s firm, results from more than 80 of its decision analyses, more on the interpretation of probability and statistical significance, and a discussion of how big data, social media, mobile phones and personal measurement devices undermine the excuse that there is not enough data. A companion workbook, also published by Wiley in 2014, provides exercises for classroom and self-study use, and the publisher’s description points readers to downloadable spreadsheet examples on the book’s website. Hubbard has applied the approach in other books: The Failure of Risk Management (2009; second edition 2020), How to Measure Anything in Cybersecurity Risk with Richard Seiersen (second edition 2023) and How to Measure Anything in Project Management with Alexander Budzier and Andreas Bang Leed (Wiley, November 2025), whose description names auditors and controllers among its intended readers. The author’s website lists the third edition, and we found no later one.
The structure: four parts and fourteen chapters
The third edition has a preface to the third edition, four parts of three or four chapters each and an appendix of calibration tests with their answers. The table follows the printed contents; the descriptions of what each chapter covers are drawn from the section headings in a library contents record and are ours in wording.
| Part | Chapters | What it covers | What it contributes |
|---|---|---|---|
| Part I: The Measurement Solution Exists | 1 The Challenge of Intangibles; 2 An Intuitive Measurement Habit: Eratosthenes, Enrico, and Emily; 3 The Illusion of Intangibles: Why Immeasurables Aren’t | Alleged intangibles and the proposal that measurement is about decisions; three historical and modern examples of measuring with little; the concept, object and methods of measurement, and economic, statistical and ethical objections | The case that the problem is a misconception, not a lack of tools |
| Part II: Before You Measure | 4 Clarifying the Measurement Problem; 5 Calibrated Estimates: How Much Do You Know Now?; 6 Quantifying Risk through Modeling; 7 Quantifying the Value of Information | Defining the decision and the language of uncertainty and risk; calibration exercises and their effect; Monte Carlo simulation and the risk paradox; expected opportunity loss and the value of information for ranges and partial uncertainty reduction | The economics: what you know, what you risk and what more information is worth |
| Part III: Measurement Methods | 8 The Transition: From What to Measure to How to Measure; 9 Sampling Reality: How Observing Some Things Tells Us about All Things; 10 Bayes: Adding to What You Know Now | Decomposition, secondary research, basic methods of observation, measuring just enough and considering error; random sampling, small samples, outliers, simple sample statistics, experiments and regression; Bayesian updating and benchmarking | The toolkit, kept as light on mathematics as the subject allows |
| Part IV: Beyond the Basics | 11 Preference and Attitudes: The Softer Side of Measurement; 12 The Ultimate Measurement Instrument: Human Judges; 13 New Measurement Instruments for Management; 14 A Universal Measurement Method: Applied Information Economics | Willingness to pay, risk tolerance and trade-offs; biases, linear models, Rasch models, the lens model and questionable methods; technology, the internet and prediction markets; the method assembled, with three cases | The applications, and the method as a whole |
| Appendix | Calibration tests and their answers | Practice tests for calibrating estimates | The exercise an audit team can run in an afternoon |
The final chapter’s cases, according to the contents, concern the value of a system that monitors drinking water, fuel forecasting for the US Marine Corps and the value of the ACORD standards. An early review in Computer Weekly described examples of software for monitoring water quality whose public-health benefit was given a monetary value, and of an improved fuel distribution system for US Marines. Readers who know the site’s summary of Hubbard’s later book on risk will recognise chapters 5 and 6, on calibration and simulation, which that book develops at length.
Part I: why the immeasurable usually is not
The first part sets out the claim and attacks the belief that resists it. Its second chapter uses three examples whose names form its title. Eratosthenes estimated the circumference of the Earth from the angle of the noon shadow at Alexandria, using a vertical stick and geometry rather than survey equipment. Enrico Fermi is the model for estimation by decomposition: a 2019 book-club report on the third edition records the question of how many piano tuners work in Chicago, which becomes answerable once it is broken into quantities that can each be estimated. And Emily is Emily Rosa, who in 1996, aged nine, designed a fourth-grade science project to test whether practitioners of therapeutic touch could detect a human energy field. In two series of tests, in 1996 and 1997, twenty-one practitioners, unable to see her, were asked which of their hands she was holding her hand above; across 280 trials they were right 123 times, 44 percent, close to what guessing would produce, and the study was published in the Journal of the American Medical Association on 1 April 1998. In our reading, the chapter’s point is the habit of measurement rather than any technique: each example reduced a large uncertainty with a simple design and very little data.
The third chapter is the argument proper. The section headings take the objections in turn: the concept of measurement, the object of measurement, the methods of measurement, the economic objection, a broader objection to the usefulness of statistics and the ethical objection. The underlying chain of reasoning is short. If something matters to a decision, it must have consequences that can be detected; if it can be detected, it can be detected as an amount or a range; and anything that can be detected as a range can be measured in Hubbard’s sense. Behind the chain sit four working assumptions the book recommends: the problem has probably been measured before; you have more data than you think; you need less data than you think; and useful new observations are more accessible than you think. In their 2014 article Hubbard and Samuelson named the opposite habit the uniqueness fallacy, the belief that only highly similar, if not identical, cases are informative, and pointed out that insurers price policies for people who are all, in their own way, unique. The table puts the objections in audit terms.
| Objection | The book’s line of answer, in our words | Where an auditor meets it |
|---|---|---|
| The concept: measurement means certainty | Measurement means reduced uncertainty; a range that narrows is a measurement | Management declines to quantify a benefit because the figure would not be exact |
| The object: we do not know what it is | Define the thing by its observable consequences; vagueness about the object is the usual barrier | “Culture”, “resilience” or “customer trust” in a business case with no definition |
| The method: we do not know how | Methods exist for almost every case, from sampling and experiments to structured expert judgment | An area exempted from monitoring because no one has designed a measure |
| Economic: it would cost too much | The value of information sets the budget; many measurements are cheap relative to the cost of being wrong | A control over a large exposure left untested because full testing is expensive |
| Statistical: small samples prove nothing | Small samples can reduce uncertainty sharply when little is known; significance tests answer a different question | Results dismissed because the sample was not “statistically significant” |
| Ethical: measuring people or lives is wrong | Preferring ignorance is not the more ethical position | Objections to measuring safety, conduct or customer harm in money |
Part II: the decision, the estimate and the value of information
The second part is where the book becomes a method. Its premise is that measurement exists to inform a decision, so the first task is to state the decision: what choices are available, what would make one better than another and what is uncertain. The approach Hubbard calls universal has five steps, which the 2019 book-club report summarised from the third edition and which an earlier reader’s notes on the second edition describe in the same terms. The steps are simple to state and harder to follow, because each removes an excuse: there is no measuring without a decision, no ignorance without a range and no measuring without a reason to expect the result to be worth more than its cost.
| Step | What it involves | The audit equivalent |
|---|---|---|
| 1. Define the decision | The choices, the criteria and the variables that connect them | The engagement objective: what will the audit committee or management decide differently because of this work? |
| 2. Determine what you know now | Calibrated ranges and probabilities for each uncertain variable, from people trained to be neither over- nor under-confident | The planning-stage view of the risk, stated as ranges rather than ratings |
| 3. Compute the value of additional information | For each variable, what reducing its uncertainty is worth to the decision | Which tests could change the conclusion, and which would only confirm it |
| 4. Measure where the value is high | Apply measurement methods to the few variables that matter, and only to the extent that pays | Concentrate fieldwork on the uncertainties that could change the opinion |
| 5. Decide and act | Stop measuring when more information is worth less than it costs | Conclude, report and move to the next engagement |
Calibration is the second step’s foundation. Most people, asked for a range they are 90 percent sure contains the answer, are badly overconfident, and the book treats calibration as a trainable skill rather than a talent. Lawrence Stone’s review records the evidence the third edition gives: across 927 people trained by Hubbard’s firm, about half of the 90 percent ranges on an initial ten-question test contained the true answer, and after half a day of training more than 80 percent did on a final twenty-question test. One of the chapter’s sections offers a trick worth adopting: imagine having to bet money on the range, which readers’ notes describe as comparing the bet with a lottery that pays out 90 percent of the time and adjusting the range until neither is preferred. The sixth chapter then shows how calibrated ranges are combined by Monte Carlo simulation into a distribution of outcomes; the Failure of Risk Management summary covers simulation and loss exceedance curves, and the Superforecasting summary covers calibration in forecasting, so this guide moves on to the chapter on which the method depends.
The seventh chapter, on the value of information, is in our view the book’s distinctive contribution and the part an auditor should read slowly. Its starting point is expected opportunity loss: the chance of being wrong multiplied by the cost of being wrong. Information is valuable to the extent that it reduces that expected loss, and the expected value of perfect information, the loss that would be avoided if the uncertainty disappeared entirely, is the ceiling on what any measurement is worth. Two practical results follow. The first is that most variables in a typical business case have an information value close to zero, because no plausible measurement would change the decision, while a few carry nearly all of it. The second, which Hubbard and Samuelson described in 2014 from comparing his firm’s information-value calculations with what clients had been measuring, is the measurement inversion: organisations tend to measure what is easy and familiar, and leave unmeasured the uncertainties they know least about, which are exactly the ones where measurement would pay most. The table is our illustration of how the inversion looks in an ordinary capital business case.
| Variable in the business case | Where the analysis effort had gone | Effect on the decision if wrong (our illustration) | What the book would do |
|---|---|---|---|
| Implementation cost | Three supplier quotes, a quantity surveyor and a contingency analysis | Small: the plausible range would not change the choice | Stop refining it |
| Staff time saved | A detailed time-and-motion study of one team | Small: the benefit is real but minor relative to the total | Accept the current range |
| Adoption by front-line users | An assumption of 90 percent, untested | Large: below about 60 percent the project fails to pay back | Measure it, perhaps with a pilot |
| Avoided incidents | Described as an intangible benefit and left unquantified | Large: it carries most of the claimed value | Decompose it into observable events and estimate their frequency and cost |
| Residual value of the old system | Not considered | None: no plausible value affects the choice | Ignore it |
Part III: decomposition, small samples and Bayes
Once the decision has identified what is worth measuring, the third part turns to how. The eighth chapter is a bridge, and its section headings form a checklist: identify the instrument of measurement; decompose the uncertain quantity into parts that are easier to estimate; assume you were not the first to measure it and search for existing research; try the basic methods of observation in turn, asking whether the thing leaves a trail, whether it can be observed directly or indirectly, and whether conditions can be created to make it observable; measure just enough; consider the error, systematic as well as random; and only then design the instrument. A quantity that defeats direct estimation, such as the cost of a data breach or the benefit of a new process, can often be broken into components each of which someone in the organisation can estimate within a useful range, and the decomposition alone sometimes reduces the uncertainty enough that no new measurement is needed.
The ninth chapter is about sampling, and its message is that small samples are far more informative than intuition suggests when prior uncertainty is large. Two of the book’s small-sample results stand out. The Rule of Five states that in a random sample of five from any population, there is a 93.75 percent chance that the population median lies between the smallest and largest values in the sample; the probability follows from the fact that each draw has an even chance of falling above or below the median, so the only way the median escapes the sample’s range is if all five fall on the same side. The Urn of Mystery, which Hubbard’s firm has described on its blog, imagines urns of red and green marbles in which every mixture is equally likely: a bettor who must guess each urn’s majority colour wins half the time, and one who may first draw a single marble wins 75 percent of the time. The formal version, the single sample majority rule, says that when nothing is known about a population proportion, one random observation has a 75 percent chance of coming from the majority. Hubbard and Samuelson add that the idea of statistical significance is widely misunderstood and often beside the point for a decision, because what matters is how much a result reduces uncertainty, not whether it passes a conventional test.
| Tool | What it says | What it assumes | A legitimate audit use | A misuse to avoid |
|---|---|---|---|---|
| Rule of Five | Five random observations give a 93.75 percent chance that the population median lies within their range | Random, independent draws; it bounds the median, not the mean or the extremes | A quick planning estimate of a typical value, such as processing time, exception age or hours spent on a manual task | Concluding on a control’s deviation rate or on a total that the extremes dominate |
| Single sample majority rule | With no prior knowledge of a proportion, one random draw has a 75 percent chance of coming from the majority | Maximum prior uncertainty about the proportion | Showing a sceptical stakeholder that even one observation is information | Treating one observation as evidence when the prior was already informed |
| Decomposition | Estimating parts is easier and more accurate than estimating the whole | The parts can be estimated by someone who knows them | Turning an “intangible” benefit or impact into estimable components | Adding false precision by multiplying many guessed parts |
| Bayesian updating | Start from a calibrated prior and revise it as evidence arrives | An honest prior, stated before the evidence | Combining prior audit results, management information and new testing | Choosing the prior after seeing the result |
| Measuring just enough | Measure only as much as the decision needs, and stop when more information is worth less than it costs | A defined decision and an estimate of the cost of being wrong | Sizing fieldwork to the point at which the conclusion would change | Cutting work short where the consequences of being wrong are severe |
The tenth chapter introduces Bayesian methods, starting from the observation that people already update their beliefs in a roughly Bayesian way and showing how to do it explicitly, including a section on what it calls heterogeneous benchmarking, applied to brand damage. For an audit function, in our reading, the chapter’s practical value is permission: the results of last year’s audit, the second line’s testing and management’s own monitoring are prior information, and a test designed in their light can be smaller than one designed as if nothing were known. What keeps this honest is stating the prior before the evidence is gathered. The sampling techniques guide and the attribute sampling guide cover the statistically designed samples that a control-testing conclusion still requires; the book’s small-sample tools are for estimation and planning, which is where they belong in audit work.
Part IV: preferences, human judges and new instruments
The fourth part extends the method to the things people are least willing to measure. The eleventh chapter treats preferences and attitudes: opinions and values observed through what people say and through what they trade off, with willingness to pay as the measure of value, an explicit quantification of an organisation’s risk tolerance, and methods for trading off several conflicting objectives. The twelfth chapter, on human judges, starts from the odd reasons behind human decisions and works through tools for making expert judgment consistent: simple linear models, Rasch models for standardising evaluations, and the lens model, in which a model is fitted to an expert’s own judgments of many scenarios so that the expert’s inconsistency is removed while the expert’s knowledge is kept. In their 2014 article Hubbard and Samuelson cited Paul Meehl’s review of 150 studies, reporting that statistical models clearly outperformed expert judgment in 144; Meehl’s own count included studies in which the two did equally well, but the experts almost never did better. One section of the chapter examines what its heading calls questionable methods of measurement. The Noise summary develops the same concern about inconsistent judgment, and the Thinking, Fast and Slow summary covers the psychology.
The thirteenth chapter surveys newer instruments, including tracking technology, the internet as a source of observations and prediction markets as a way of aggregating opinion, themes the third edition expanded, by the author’s account, with material on big data, social media and personal measurement devices. Stone’s review records another of the book’s examples: the Cleveland Orchestra counting standing ovations, an observable measure for something that sounds unmeasurable. The fourteenth chapter assembles the pieces into Applied Information Economics and illustrates it with the three cases named above. For an auditor, the part’s main lesson is that the softest-looking measures, such as satisfaction, culture or the quality of judgment, have observable correlates that can be tracked, and that a measure built from structured expert judgment can be tested for consistency in the same way as any other control.
The nine ideas worth keeping
- Measurement is uncertainty reduction. A result that narrows a range is a measurement, even if it is not an exact number; demanding certainty is a way of avoiding measurement.
- If it matters, it is observable. Anything that affects a decision has consequences, and consequences can be observed as amounts or ranges. Define the “intangible” by what would be different if it changed.
- Start from the decision. A measurement with no decision attached has no value to compute; the first question for any metric is what would be done differently depending on its result.
- State what you know now, as a range. Calibrated estimates are the starting point, and calibration is a trainable skill: in the book’s data, half a day lifted the hit rate of 90 percent ranges from about half to more than 80 percent.
- Measure where information has value. Expected opportunity loss, the chance of being wrong times the cost of being wrong, says which uncertainties are worth reducing. Most variables are not.
- Beware the measurement inversion. Organisations measure what is easy and familiar and neglect what they know least about, which is usually where measurement pays.
- Small samples go a long way. Five random observations bound a median with 93.75 percent confidence; one random draw identifies a majority 75 percent of the time when nothing is known.
- Decompose before you measure. Breaking a quantity into estimable parts often reduces uncertainty without any new data, and shows which part needs measuring.
- Make expert judgment consistent. Models of experts’ own judgments, structured scales and simple linear models remove inconsistency while keeping knowledge.
The vocabulary: fourteen terms defined for practitioners
| Term | What Hubbard means | What it looks like in an audit |
|---|---|---|
| Measurement | An observation that reduces uncertainty, with the reduction expressed as a quantity | Evidence that narrows the range of a conclusion, even if it does not settle it |
| Intangible | Something believed to be immeasurable, usually because it has not been defined | A benefit, risk or control objective stated without observable consequences |
| Uniqueness fallacy | The belief that only highly similar, if not identical, past cases are informative | “Our situation is different” offered as a reason not to use benchmark or historical data |
| Decomposition | Breaking an uncertain quantity into parts that can be estimated | Turning “reputational impact” into events, frequencies and costs |
| Calibrated estimate | A range or probability from someone whose stated confidence matches their actual hit rate | Planning estimates from auditors who have taken a calibration test |
| 90 percent confidence interval | A range the estimator believes has a 90 percent chance of containing the true value | The form in which an impact or likelihood estimate should be recorded |
| Expected opportunity loss | The chance of being wrong multiplied by the cost of being wrong | The exposure a decision carries before any further audit work |
| Expected value of perfect information | The loss avoided if uncertainty were eliminated; the ceiling on what measurement is worth | The upper bound on what an engagement or a test is worth to the decision it informs |
| Measurement inversion | The tendency to measure what is easy rather than what matters most | Audit hours spent re-testing stable, well-evidenced areas while uncertain ones go unexamined |
| Rule of Five | A 93.75 percent chance that the population median lies within the range of five random observations | A quick, honest estimate of a typical value at planning stage |
| Single sample majority rule | With no prior knowledge, one random draw has a 75 percent chance of coming from the majority | The case for looking at even one example before forming a view |
| Lens model | A statistical model fitted to an expert’s own judgments to remove their inconsistency | A model of how reviewers actually rate findings, used to calibrate the rating scale |
| Applied Information Economics | Hubbard’s method combining decision modelling, calibration, information value and measurement | A structured way to scope an audit of a decision or a business case |
| McNamara fallacy | Measuring only what is easy and then treating what cannot easily be measured as unimportant | A dashboard that reports volumes because outcomes were never defined |
What it means for internal audit: measuring what the plan depends on
The book was written for managers and analysts, but internal audit is a measurement business with a fixed budget, and the method fits it closely. The Global Internal Audit Standards, published in January 2024 and effective from 9 January 2025, require the internal audit plan to be based on a documented assessment of the organisation’s strategies, objectives and risks (Standard 9.4), require engagement planning to rest on an understanding of the activity drawn from reliable, relevant and sufficient information (Standard 13.2), and require each finding’s significance to be evaluated by considering the likelihood and impact of the risk (Standard 14.3). Each leaves the method to the methodologies the chief audit executive must establish under Standard 9.3, which leaves room for Hubbard’s ranges and information values. The table maps the method to the main activities; the risk assessment playbook, the scoping memo guide and the finding severity guide carry the surrounding procedures.
| Audit activity | What the book implies | Procedure it changes |
|---|---|---|
| Annual risk assessment and audit plan | Rank auditable areas by the value of the uncertainty an audit would reduce, not only by inherent risk scores | For the largest candidate engagements, record what the committee would decide differently depending on the result; drop or shrink those where the answer is nothing |
| Engagement scoping | Spend effort where it could change the conclusion; measure just enough | In the scoping memo, name the two or three uncertainties that could change the opinion and size the testing to them |
| Auditing business cases and capital projects | Intangible benefits can be decomposed and measured; the measurement inversion is common | Test whether the case states the decision, expresses uncertain benefits as ranges and measured the variables with the highest information value; see the capital projects audit guide |
| Auditing performance measures and KPIs | A measure without a decision attached is a vanity measure; the McNamara fallacy is a design flaw | For each reported KPI, ask what decision it informs and what change in it would trigger action; see the KPI design guide |
| Auditing culture and other soft areas | Soft does not mean unobservable; behaviour leaves trails | Define the cultural attribute by observable behaviour before testing it; see the culture audit guide |
| Analytics and estimation | Small random samples are powerful for estimating typical values when uncertainty is high | Use the Rule of Five for planning estimates; keep statistically designed samples for control-testing conclusions, as the sample sizes guide explains |
| Rating findings | Impact is a quantity with a range, not a colour | Add a calibrated range for the potential impact to the finding template; the impact writing guide shows how to state it |
| Measuring the function’s own value | The value of assurance is the value of the information it provides to decisions | Under Standard 12.2, which requires performance objectives and a methodology to measure progress, track decisions changed by audit work alongside activity measures; see the self-assessment guide |
Two cautions keep the map honest. The first is that information value is a tool for deciding where to look, not a reason to stop looking at areas where a failure would be catastrophic but the current evidence happens to be reassuring; the book’s own logic gives high information value to uncertain variables with large consequences, and an audit function should be careful not to read low uncertainty into areas it has simply not examined. The second is that the book’s small-sample results concern estimation, and an audit opinion on whether a control operated effectively is a different question, usually about the rate of deviations in a population, for which the function’s sampling methodology remains the authority. Used for planning, scoping, challenging business cases and expressing impact, the method strengthens audit judgment; used to justify thin testing of controls, it would not.
The criticisms, and what the book leaves out
The reviews we found are largely favourable, and their reservations are practical rather than fundamental. Lawrence Stone, chief scientist at Metron and a member of the US National Academy of Engineering, reviewed the third edition for ISIF Perspectives on Information Fusion in July 2024 and admired its gradual, low-mathematics case for statistical decision-making, and his caveats were practical: the book runs to about 400 pages because of its discussion, anecdotes and examples, which he thought suited its audience of managers; calibration takes half a day, which may be difficult or time-consuming but is probably worth trying where the decision matters; the Rule of Five requires random samples, each an independent draw; and in his own field, tests during real operations are often hard or impossible to run, so he advises blind tests on held-back recorded data. The 2019 book-club report, written by university technology staff, found the first two parts the most useful for decision-makers and the statistical third part less relevant unless one is designing measurements, and one member observed that the therapeutic touch experiment answered whether practitioners could sense a field, not whether patients benefited. An earlier review of the second edition, in the New York Journal of Books, warned that the book asks for sustained intellectual effort and that its later technical sections would be hard going for some readers.
Three further limits matter for an auditor, and they are our reading rather than the reviewers’. The first is the evidence base. Much of the support for the method, the calibration results and the decision analyses, comes from Hubbard’s own firm, as his 2014 article made clear, and a reader should weigh it as practitioner evidence rather than independent research. The second is behaviour. The book is about measurement to inform a decision, and Hubbard himself addresses the McNamara fallacy of valuing only what is measured; but an audit function should add the caution from performance-management practice that a measure tied to pay or targets changes the behaviour it measures, so that a well-designed measure can degrade once it becomes a target. The third is governance. The method makes assumptions explicit, which is its strength, but explicit assumptions can still be chosen to support a decision already made; the auditor’s question about any model built this way is who set the ranges, whether the estimators were calibrated, and whether the ranges were recorded before the result was known. The Black Swan summary is the counterweight for exposures whose tails no calibrated range can capture.
Editions, companions and how to read it
| What | Details | Best for |
|---|---|---|
| First edition | Wiley, 2007 | Historical interest only |
| Second edition | Wiley, April 2010 | Superseded |
| Third edition | Wiley, March 2014; 432 pages in the Google Books record; the current edition | The edition to read |
| Workbook | How to Measure Anything Workbook, Wiley, 2014 | Teams that want to practise the exercises |
| Online material | Downloadable spreadsheet examples on the book’s website, as the publisher’s description notes | Rebuilding the calculations |
| Companions by the same author | The Failure of Risk Management (2009; second edition 2020); How to Measure Anything in Cybersecurity Risk with Richard Seiersen (second edition 2023); How to Measure Anything in Project Management with Alexander Budzier and Andreas Bang Leed (2025) | Applying the method to risk, security and projects |
| Reading plan | Part I quickly; Part II slowly, especially the chapter on the value of information; the eighth and ninth chapters; the calibration appendix as an exercise; the rest as needed | An auditor who wants the method first |
Read it with a current business case or risk assessment open. The test of the book is whether, after the chapter on the value of information, the reader can say which two or three uncertainties in that document actually matter to the decision it supports, and whether anyone has measured them. The Failure of Risk Management summary is the natural next step for readers who want the risk-analysis side.
A one-hour discussion guide for an audit team
- Ten minutes: calibrate. Everyone answers ten general-knowledge questions with 90 percent ranges before the session. Score them in the room; the book’s data suggests that untrained estimators get only about half of such ranges right.
- Fifteen minutes: find an intangible. Take a recent business case, performance report or finding that described a benefit or impact as intangible. Define it by its observable consequences and decompose it into parts someone could estimate.
- Ten minutes: the decision. For the next engagement in the plan, write down what the audit committee or management would decide differently depending on the result. If the answer is nothing, discuss why the engagement is in the plan.
- Fifteen minutes: the inversion. List where the last three engagements spent most of their hours and where the greatest uncertainty about the conclusion lay. Note any engagement where the two did not match.
- Ten minutes: one change. Choose one practice to adopt, such as calibrated ranges in the finding template or a value-of-information question in the scoping memo, and name an owner; the QAIP playbook shows where to record it.
Worked example: Millbrook Regional Water Authority tests a business case
Millbrook Regional Water Authority is a fictional statutory water body serving 1.4 million people, with an audit and risk committee and an internal audit function of four: the chief audit executive and three auditors. In the 2027 plan the committee asked the function for an advisory review of a business case before it went to the board: a $38 million smart-network programme of pressure sensors, acoustic leak loggers and smart meters across the authority’s 140 district metered areas. The case claimed an 18 percent reduction in leakage and listed three further benefits as intangible: customer trust, resilience and regulatory standing. Together the intangibles were said to account for most of the programme’s value, but none was quantified. The chief audit executive agreed the scope with management on the basis that the function would test the case, not rewrite it, and assigned two auditors for six weeks.
| Test | What the function found | Result |
|---|---|---|
| Is the decision defined? | The case asked the board to approve the full programme; two alternatives, a phased rollout starting with the oldest districts and continued active leak detection without new technology, had been discussed by the asset planning team but not costed | Recommendation: present the three options with costs and ranges; management added both alternatives to the paper |
| Are the intangibles defined? | In a workshop with the customer, operations and regulation teams, “customer trust” became the volume of complaints about low pressure and interruptions; “resilience” became hours of supply interruption per property per year; “regulatory standing” became the probability of formal enforcement action on leakage performance in the next three years | Three intangibles became three measurable quantities, each with a current baseline from existing records |
| Are estimates ranges from calibrated people? | Every figure in the case was a single point. After a half-day calibration session, the asset planning team’s 90 percent range for leakage reduction was 6 to 22 percent, against the case’s 18 percent | Recommendation: ranges throughout the case; the point estimates removed |
| Where is the information value? | Most of the analysis effort had gone into implementation cost, with three supplier quotes and a quantity surveyor’s review, but a simple model showed that no plausible cost outcome changed the choice; nearly all the value of further information lay in the leakage reduction achievable in the older cast-iron districts | Observation: a measurement inversion; further work on cost stopped and effort moved to leakage |
| Can the key uncertainty be measured cheaply? | The board’s question for the first phase was whether a typical older district would achieve at least an 8 percent reduction, the level at which the asset planning model showed the first phase paying back. Management installed loggers for three months in five districts chosen at random from the 52 older cast-iron districts; the reductions were 9, 11, 12, 15 and 19 percent | By the Rule of Five, a 93.75 percent chance that the median older district’s reduction lies between 9 and 19 percent, above the break-even point; the function’s report noted that the result bounds the median, not the total saving, which the first phase would measure, and says nothing about the newer network |
| Does the plan keep measuring? | The case had no arrangement for checking the claimed benefits after installation | Recommendation: benefits tracking against the three defined quantities, reported to the committee annually |
The board approved a first phase of $14 million, covering 40 of the older districts and the central monitoring platform, with the second phase to be decided on measured results from the first. The decision was the board’s, and the function’s report said plainly that it expressed no view on whether the programme should proceed; what it gave the board was a case in which the claimed benefits could be checked, the uncertainty was visible and the evidence on the variable that mattered had been gathered before the money was committed rather than after. The pilot cost a small fraction of the first phase, and the function recorded in its report that the result settled the question the board needed answered, whether a typical older district was likely to clear the 8 percent threshold, without settling questions it did not need answered, such as the exact figure.
The chief audit executive then applied the same discipline to the function’s own plan. For each engagement in the draft 2027 plan, the team wrote down what decision the result could change and how uncertain the function was about the conclusion. Two engagements, a fourth successive payroll audit in an area with clean results and stable systems, and a review of a policy that the board had already decided to replace, could change no decision; they were dropped, and the days went to the operational technology security review and the chemicals procurement audit, where the function’s uncertainty was greatest. The audit and risk committee chair’s comment at the next meeting was that it was the first plan the committee had seen that explained why each audit was worth doing, which is the book’s argument applied to the function that audits everyone else. The guide to internal audit in energy and utilities and the guide to internal audit in government describe the setting.
Frequently asked questions
Can anything really be measured?
In Hubbard’s sense, anything that matters to a decision can be measured, because measurement means reducing uncertainty rather than reaching certainty, and anything that matters has observable consequences. That does not mean every measurement is worth making: the value of information decides which ones are.
How does this book differ from The Failure of Risk Management?
How to Measure Anything is the general method: defining the decision, calibrated estimates, the value of information and measurement techniques. The Failure of Risk Management applies the method to risk and makes the case against risk matrices and scoring; the site has a separate summary of it.
Do I need statistics to use it?
Less than the subject suggests. Calibration needs none, the value-of-information logic needs arithmetic, and the Rule of Five needs only a random sample of five. Reviewers note that the later technical chapters are harder going, and the companion workbook and online spreadsheets help.
Can the Rule of Five replace audit sampling?
No. It estimates where the median of a population lies and requires random, independent draws. It is useful for planning estimates, such as typical processing times, but conclusions on whether a control operated effectively need a sample designed for that purpose; see the sampling techniques guide.
What is the measurement inversion?
Hubbard’s name for the pattern his firm found when it compared the information value of clients’ uncertain variables with what they had been measuring: organisations measure what is easy and familiar and neglect the uncertainties they know least about, which are usually the ones where measurement would pay most.
Where does the book sit in an auditor’s reading?
With the books on risk and judgment, beside The Failure of Risk Management, Superforecasting and Noise; the internal audit reading list organises the site’s recommendations by theme and career stage.
Related guides
- The Failure of Risk Management by Douglas Hubbard: Summary for Auditors — the method applied to risk assessment
- The Annual Internal Audit Risk Assessment: A Step-by-Step Playbook — where information value enters the plan
- How to Write an Engagement Scoping Memo (With a Worked Example) — scoping to the uncertainties that matter
- How to Audit Capital Projects and Construction: Cost, Contract, and Schedule Controls — testing business cases and benefits
- Internal Audit Sampling Techniques: Statistical, Non-Statistical and Risk-Based Methods — when a designed sample is still required
- Superforecasting by Tetlock and Gardner: Book Summary and Review — calibration and measured prediction
- The Best Books for Internal Auditors — where this book sits among the others
- All book summaries — the full set of summaries for auditors
- Thinking in Bets by Annie Duke: Summary and Lessons for Auditors — judging decisions apart from their outcomes.
Leave a Reply