Daniel Kahneman, Olivier Sibony and Cass R. Sunstein’s Noise: A Flaw in Human Judgment (2021) is about the component of judgment error that, the authors argue, individuals and organisations mostly neglect. Bias is the shared, predictable tilt that pushes many judgments the same way; noise is the scatter, the unwanted variability among judgments that ought to be the same. Two underwriters quote different premiums for the same risk, two judges give different sentences for the same offence, and the same professional can judge the same case differently on another day. The authors argue that this variability is much larger than the people in charge expect, that it adds to error in the same way bias does, that it can be measured without knowing the right answer, and that a set of procedures they call decision hygiene reduces it. For internal auditors the book matters twice over. Much of what an audit function tests is professional judgment, such as loan risk ratings, claims reserves, impairment reviews and control self-assessments, and the book supplies a method, the noise audit, for testing whether that judgment is consistent. And internal audit is itself a judgment business: it rates risks to build its plan, rates findings in its reports and reaches conclusions on whole control environments. This guide summarises the book without reproducing it, sets out its taxonomy of noise and the evidence behind it, separates what the book reports from what the underlying studies found, records the published criticisms, maps the method to audit practice and ends with a worked example in which a bank’s audit function runs a noise audit on its loan risk ratings and on its own finding ratings.
In this guide
- The book in one page
- Who the authors are, and how the book came about
- The structure: six parts and twenty-eight chapters
- Bias, noise and the error equation
- The anatomy of noise: level, pattern and occasion
- The evidence, and how much weight it bears
- Decision hygiene: the remedies and their limits
- The nine ideas worth keeping
- The vocabulary: fourteen terms defined for practitioners
- What it means for internal audit: judgment as an auditable process
- The criticisms, and what the book leaves to the reader
- Editions, reading time and how to read it
- A one-hour discussion guide for an audit team
- Worked example: Lakeshore Bancorp audits its loan ratings and its own
- Frequently asked questions
- Related guides
The book in one page
| Question | The answer, in our words |
|---|---|
| What is noise? | Unwanted variability in judgments that should be identical: different professionals reaching different conclusions on the same case, or one professional reaching different conclusions on different occasions. Bias is the shared error, the tendency of judgments to miss in the same direction; noise is the unshared scatter. |
| Why does it matter? | Because overall error, measured as mean squared error, is the sum of bias squared and noise squared. Noise and bias add to error in the same way, so reducing noise reduces error even when the average judgment stays where it was. |
| How much is there? | More than the people in charge expect. At the insurer whose underwriters the authors tested, executives expected two underwriters’ quotes for the same case to differ by about 10 percent; the book reports a median difference of 55 percent, and 43 percent for claims adjusters. In sentencing the authors report an average sentence of seven years and an average gap of more than three and a half years between two randomly chosen judges. |
| Where does it come from? | Level noise (some judges are harsher than others across the board), pattern noise (judges respond differently to particular cases, often the largest component in the authors’ account) and occasion noise (the same judge varies with mood, fatigue and circumstance). |
| What is the remedy? | Measure first, with a noise audit; then apply decision hygiene: control the sequence in which information reaches the judge, collect independent judgments and aggregate them, use guidelines and shared scales anchored on real cases, break complex judgments into separately assessed parts, and use rules or algorithms where they fit. |
| What are the limits? | The last part of the book argues that noise reduction has costs, that people value a hearing from a person rather than a formula, and that the choice between rules and open-ended standards is a trade-off. Zero noise is not the goal. |
| Who wrote it? | A psychologist and Nobel laureate in economics (Kahneman), a professor of strategy and former McKinsey senior partner (Sibony), and a Harvard law professor who ran the White House regulatory office (Sunstein). |
| Why read it as an auditor? | Because consistency of judgment is testable, because an audit programme can test that a judgment-based control operated without ever testing whether the judges agree, and because the function’s own ratings of risks and findings are judgments of exactly the kind the book shows to be noisy. |
Who the authors are, and how the book came about
Kahneman, who died in March 2024 at the age of 90, was a psychologist whose work with Amos Tversky on judgment and decision-making under uncertainty helped to found behavioural economics and earned him the 2002 Nobel Prize in Economic Sciences, shared with the economist Vernon Smith. He joined Princeton in 1993 and became emeritus professor of psychology and public affairs there; his Thinking, Fast and Slow (2011) is summarised in the Thinking, Fast and Slow guide. Sibony is professor of strategy at HEC Paris and an associate fellow of Oxford’s Saïd Business School, after about twenty-five years at McKinsey and Company in France and the United States, where he was a senior partner; one of his earlier books, You’re About to Make a Terrible Mistake!, is about how biases distort decision-making. Sunstein is the Robert Walmsley University Professor at Harvard, co-author with Richard Thaler of Nudge (2008) and founder and director of Harvard Law School’s Program on Behavioral Economics and Public Policy, and he was Administrator of the White House Office of Information and Regulatory Affairs from 2009 to 2012. The combination is visible in the book: a psychologist’s theory of error, a strategist’s interest in how organisations decide, and a lawyer’s preoccupation with rules, discretion and fairness.
The idea reached a management audience before it became a book. In 2016 Kahneman and three colleagues, Andrew Rosenfield and Tom Blaser of the consultancy TGG Group and Linnea Gandhi, published an article in the Harvard Business Review on the hidden cost of inconsistent decisions, proposing the noise audit, in which members of a professional unit evaluate the same set of cases and the spread of their answers measures the noise. In a podcast interview in May 2021 Kahneman described the starting point as a question put to a large insurance company: did its underwriters agree with one another? Several dozen of them were given the same case descriptions and asked to set a premium. He said the book grew from the follow-on question of how organisations could reduce noise, on which he, Sibony and Dan Lovallo had begun working together; that at his age he could no longer run the experiments himself, so a book was the remaining route; and that he and Sibony, who was in Paris, worked over video for an hour or two a day through the pandemic. One of the remedies the book develops, the mediating assessments protocol, had already appeared in a 2019 MIT Sloan Management Review article by Kahneman, Lovallo and Sibony on structuring strategic decisions. The book was published in May 2021, by Little, Brown Spark in the United States and William Collins in the United Kingdom.
The structure: six parts and twenty-eight chapters
The book has an introduction, six parts of between three and eight chapters, a review and conclusion, a short epilogue and three appendices. The table follows the printed contents; the descriptions of what each part covers are ours, and where we name a topic within a chapter it is because a reviewer’s chapter notes place it there.
| Section | Chapters | What it covers | What it contributes |
|---|---|---|---|
| Introduction: Two Kinds of Error | None | The distinction between bias and noise | The frame for the rest of the book |
| Part I: Finding Noise | 1 Crime and Noisy Punishment; 2 A Noisy System; 3 Singular Decisions | Sentencing disparity and the work of the federal judge Marvin Frankel; the noise audit at an insurance company; why one-off decisions are noisy too | The evidence that noise exists and is large |
| Part II: Your Mind Is a Measuring Instrument | 4 Matters of Judgment; 5 Measuring Error; 6 The Analysis of Noise; 7 Occasion Noise; 8 How Groups Amplify Noise | Judgment treated as measurement; the error equation; the division of system noise into level and pattern noise; variability within one person and the averaging behind the wisdom of crowds; how cascades and polarisation make groups noisier | The theory and the arithmetic |
| Part III: Noise in Predictive Judgments | 9 Judgments and Models; 10 Noiseless Rules; 11 Objective Ignorance; 12 The Valley of the Normal | Human judgment compared with models and rules, and the limits that unknowable information puts on any prediction | Why simple rules so often match or beat experts |
| Part IV: How Noise Happens | 13 Heuristics, Biases, and Noise; 14 The Matching Operation; 15 Scales; 16 Patterns; 17 The Sources of Noise | How noise relates to the heuristics and biases of Kahneman’s earlier work, how rating scales add noise, and how the components combine | The causes |
| Part V: Improving Judgments | 18 Better Judges for Better Judgments; 19 Debiasing and Decision Hygiene; 20 Sequencing Information in Forensic Science; 21 Selection and Aggregation in Forecasting; 22 Guidelines in Medicine; 23 Defining the Scale in Performance Ratings; 24 Structure in Hiring; 25 The Mediating Assessments Protocol | What makes a better judge, the definition of decision hygiene, then one technique per domain | The method |
| Part VI: Optimal Noise | 26 The Costs of Noise Reduction; 27 Dignity; 28 Rules or Standards? | When reducing noise is not worth it, and what is lost when judgment is replaced by procedure | The limits |
| Review and Conclusion; Epilogue | Taking Noise Seriously; A Less Noisy World | The argument restated as a programme | The summary to reread |
| Appendices | A How to Conduct a Noise Audit; B A Checklist for a Decision Observer; C Correcting Predictions | Practical tools and a technical note | The pages an audit team will use |
Bias, noise and the error equation
The book’s starting point, signalled by the title of its second part, is that judgment is a form of measurement in which the mind is the instrument, and that measurement error comes in two kinds. Bias is shared error: when judgments of the same thing tend to miss in the same direction, the miss they have in common is bias. Noise is the variability of the errors: the extent to which judgments that should agree do not, measured statistically as a standard deviation. The authors illustrate the difference with shots at a target, in which a biased group’s shots cluster away from the centre while a noisy group’s are scattered. The distinction matters because the remedies differ. Correcting a bias requires knowing its direction; reducing noise does not require knowing its cause, any more than washing hands requires identifying the germ, which is the analogy behind the term decision hygiene.
The arithmetic is what the authors build on, and the chapter on measuring error is where they set it out. If error is measured as the mean of the squared errors, the standard measure in statistics, it divides into two parts: the square of the bias plus the square of the noise. Because the two are independent they add like the sides of a right-angled triangle, so a given amount of noise does as much damage as the same amount of bias, and cutting either cuts overall error. The consequence that most readers find counter-intuitive, and that Michael Blastland singled out in his review in Prospect, is that pulling scattered judgments toward their average reduces overall error even if the average itself is wrong. The authors apply the equation to predictive judgments, where a true value will eventually be known; they say it does not strictly apply to evaluative judgments such as a sentence, where there is no true value to miss, though Blastland noted that sentencing is the book’s own leading example.
Two further claims complete the frame. The first is that noise matters for one-off decisions as well as repeated ones, the subject of the chapter on singular decisions, and that in repeated decisions errors add up rather than cancelling out: a policy priced too high and another priced too low are two errors, not none. The second is that noise is invisible to the people who produce it. Each professional sees only his or her own judgments, each judgment feels well reasoned, and nobody compares the judgments of different people on the same case unless the organisation arranges it. That is why the authors insist that the first step is always to measure, and why the measurement surprises the executives who commission it. For an auditor the second claim is the important one, because it describes a control gap: a process that relies on judgment and has never tested whether its judges agree has no evidence that it is consistent. The professional skepticism guide describes the individual auditor’s discipline; the book adds the system-level test.
The anatomy of noise: level, pattern and occasion
The book’s analysis of noise is a decomposition, introduced in the chapter on the analysis of noise and extended in the chapters that follow. The variability across a set of judges assessing the same cases, which the authors call system noise, has two parts. Level noise is the difference between judges’ average judgments: one judge is more severe than another across all cases. Pattern noise is what remains when level is removed: judges respond differently to particular cases, so that, for instance, one is harsher than colleagues on fraud and more lenient on drugs while another is the reverse. Pattern noise in turn has two sources. Stable pattern noise reflects a judge’s lasting idiosyncrasies, the personal weighting of the features of a case that the authors associate with individual values and tastes. Occasion noise is variability within one judge: the same person reaching a different conclusion on the same case on a different day, under the influence of mood, fatigue, weather or timing. In the authors’ account pattern noise is often the largest component.
| Component | What it is | In the book’s terms | What it looks like in audit work |
|---|---|---|---|
| System noise | Total unwanted variability among judges assessing the same cases | Different sentences for the same offender from different judges | Different risk grades for the same loan file from different credit officers |
| Level noise | Differences between judges’ average levels | A judge harsher than colleagues across the board | An audit manager who rates every finding a notch higher than peers do |
| Pattern noise | Judges’ different responses to particular cases | Judges weighting the features of a case differently | One auditor rating access findings high and change findings low while a colleague does the reverse |
| Stable pattern noise | The lasting part of pattern noise | A judge’s enduring idiosyncrasies, values and tastes | A reviewer’s consistent personal emphasis, visible across years of reports |
| Occasion noise | Variability within one judge over time | The same judge, a different mood, day or time | The same auditor rating the same finding differently at year-end and in spring |
| Group amplification | Discussion that magnifies rather than averages differences | First speakers, information cascades and polarisation | A rating meeting in which the most senior person speaks first |
The chapter on groups adds a twist that audit managers will recognise. Averaging independent judgments reduces noise, which is the statistical basis of the wisdom of crowds; but a group that discusses before its members have formed independent views does not average, because early speakers, cascades of agreement and polarisation pull the group toward whichever view was voiced first. The remedy is to collect judgments separately before discussion, a point that applies directly to rating meetings, risk assessment workshops and the closing of a review. The part on how noise happens supplies the psychology, connecting noise to the heuristics and biases that Kahneman studied for decades and to rating scales themselves, because two people can agree about a case and still choose different words or numbers for it. The LSE reviewer illustrates the point with an income of $10 million a year, which different people might call wealthy, affluent or super-rich.
The evidence, and how much weight it bears
The book’s case rests on a run of studies across professions. The table sets out the main ones as the book or its reviewers report them and, alongside, what the research in each area that we were able to check found. The figures should be read as the authors’ reports of others’ work and of their own noise audits, not as a single body of evidence.
| Domain | What the book reports | Research we checked | A note for the reader |
|---|---|---|---|
| Insurance underwriting and claims | Executives expected about a 10 percent gap between two professionals’ figures for the same case; the median gap was 55 percent for underwriters and 43 percent for claims adjusters | The authors’ own noise audit at an insurer, described by Kahneman as several dozen underwriters pricing the same case descriptions | One company, not named; the measure is the gap between two randomly chosen professionals, relative to their average |
| Criminal sentencing | An average sentence of seven years and an average gap of more than three and a half years between two randomly chosen judges; ranges of one to ten years for a heroin dealer and five to eighteen years for a bank robber | The Federal Judicial Center’s 1974 Second Circuit study, in which fifty judges sentenced the same thirty cases, found a five-to-eighteen-year range for its bank robbery case and disagreement about whether to imprison at all in sixteen of twenty cases | The book’s first chapter begins with the federal judge Marvin Frankel, whose 1973 book Criminal Sentences: Law Without Order was a landmark study of sentencing disparity |
| Sentencing after the guidelines became advisory | Disparity increased after the Supreme Court’s 2005 decision in United States v. Booker made the federal guidelines advisory | A 2014 study by Crystal Yang of nearly 400,000 defendants found that interjudge disparities had doubled | Consistent with the book’s argument that rules reduce noise |
| Mammography | Radiologists’ false-negative rates ranged from zero to more than 50 percent | A 1996 study had 108 radiologists from 50 accredited centres read the same 79 screening mammograms; their sensitivity ranged across at least 40 percentage points | A verifiable judgment, so noise here is also error against the truth |
| Psychiatric diagnosis | Two psychiatrists independently assessing 426 patients agreed on the diagnosis only about half the time | Reported from the book by Blastland | Diagnostic criteria and structured interviews are the book’s remedy in medicine |
| Fingerprint examination | Examiners do not always reach the same conclusion on the same prints when they see them again | A 2012 study retested 72 latent print examiners after about seven months: 89.1 percent of individualisation decisions and 90.1 percent of exclusions were repeated | Occasion noise among trained examiners |
| Performance ratings | Most of the variance in ratings, 70 to 80 percent in the studies the book draws on, reflects the rater rather than the person rated | As reported from the book by Theodore Kinni in strategy+business | The basis for the chapter on defining the scale |
| Occasion noise | Mood, weather, time of day, hunger and even football can shift judgments | Studies of particular settings; Blastland notes that some studies the book cites, including one about judges, have been challenged | In our view the part of the evidence to weigh most cautiously; see the criticisms below |
Two cautions apply. First, Blastland noted that some studies the book cites, including one about judges, have been challenged by other experts, and the statistician Andrew Gelman, endorsing a review by Gaurav Sood, argued that the book’s recommendations rest on an uncritical reading of a behavioural-science literature with noise problems of its own. Our advice is to place the least weight on the individual occasion-noise studies and the most on an organisation’s own measurement. Second, the most striking figures, the insurer’s, come from one organisation that the authors do not name. Neither caution touches the central claim, because the central claim is easy to test. Any organisation can run a noise audit on its own judgments, and that result, not the book’s figures, is the evidence that matters. That is the stance an audit function should take when it brings the book into its methodology: treat the published studies as reasons to look, and the organisation’s own noise audit as the finding.
Decision hygiene: the remedies and their limits
Part V turns from diagnosis to treatment. The authors call their remedies decision hygiene because, like hand-washing, the procedures prevent errors without the user needing to know which error they prevent. The part begins with the judges themselves: the chapter on better judges distinguishes experts whose accuracy can be verified from those the authors call respect-experts, whose authority rests on the regard of their peers. The next chapter defines decision hygiene and sets it beside debiasing, which targets a known error. The remaining chapters each pair a domain with a technique, and two ideas recur throughout: independence, meaning judgments made separately before they are combined, and structure, meaning complex judgments broken into parts that are assessed one at a time on defined scales before an overall view is formed. The mediating assessments protocol, which closes the part, combines both. As set out in the 2019 article that preceded the book, it has three elements: decide in advance which assessments matter, make each one independently and on the evidence for that assessment alone, and hold back the overall judgment until every assessment is on the table.
| Technique | Where the book develops it | What it does | The audit version |
|---|---|---|---|
| Noise audit | Chapter 2 and Appendix A | Gives the same cases to several judges and measures the spread of their answers | An inter-rater test of loan grades, claims reserves, impairment conclusions or the function’s own finding ratings |
| Sequencing information | Chapter 20, forensic science | Gives examiners only the information they need, when they need it, to prevent premature conclusions | Form the test conclusion before reading management’s explanation; let the reviewer see the evidence before the in-charge’s conclusion |
| Selection and aggregation | Chapter 21, forecasting | Chooses better judges and averages independent judgments; averaging a hundred independent judgments cuts noise by about 90 percent | Independent first ratings by two or three auditors, then averaged or reconciled |
| Guidelines | Chapter 22, medicine | Diagnostic criteria and structured assessments make clinicians agree more often | Rating criteria with worked examples in the methodology manual |
| Anchored scales | Chapter 23, performance ratings | Shared scales anchored on real cases, so that raters mean the same thing by the same grade | A reference library of past findings at each severity level |
| Structure | Chapter 24, hiring | Breaks the overall judgment into separately assessed components, as in a structured interview | Separate assessments of impact and likelihood before the overall finding rating |
| Mediating assessments protocol | Chapter 25 | Assessments defined in advance, made independently, overall judgment delayed | Engagement conclusions built up from component assessments rather than formed first and justified afterwards |
| Decision observer | Appendix B | A checklist for a person assigned to watch a decision process for signs of error | A quality reviewer who attends the rating meeting rather than reading its minutes |
Algorithms and rules remove noise altogether, because a formula gives the same output for the same input every time, and the part of the book on predictive judgment argues that simple rules often match or beat experts. The authors are careful about the limits. Algorithms are not always practical or affordable, they can carry bias from the data used to build them, and they can violate the expectation that a person facing a serious decision will be heard by another person. Part VI is the book’s own qualification of its argument: noise reduction costs money and time, some methods introduce errors of their own or invite gaming, rigid procedures can demoralise the professionals they bind and stifle innovation, and the book’s final chapter weighs rules against standards. The authors do not argue for eliminating noise in every case; they argue for measuring it and deciding deliberately how much to tolerate. The model risk audit guide covers the assurance side of the algorithmic remedy, which replaces noise with model risk.
The nine ideas worth keeping
- Noise is a separate error from bias. It has its own causes and its own remedies, and an organisation that works only on bias has addressed half of the error equation.
- Noise and bias add to error in the same way. Mean squared error is bias squared plus noise squared; reducing scatter reduces error even if the average does not move.
- Noise can be measured without knowing the right answer. Give the same cases to several judges and measure the spread. No outcome data are needed, which is why the noise audit is available to any audit function.
- Expect more noise than you think. The insurer’s executives expected a 10 percent gap and the book reports 55 percent. The people who run a judgment process are badly placed to estimate its noise, because they never see the comparison.
- Pattern noise is the hidden part. Differences in level are easy to spot and to calibrate; differences in how judges weigh particular features of particular cases are not, and they are often the largest component.
- Groups amplify noise unless judgments are made independently first. Discussion after independent judgment averages; discussion before it cascades.
- Simple rules often match or beat unaided judgment. Consistency is worth more than intuition suggests, which is the case for scoring models, checklists and defined criteria.
- Structure the judgment before forming it. Define the component assessments in advance, make them independently and delay the overall view; the mediating assessments protocol is a method for any complex judgment, including an engagement conclusion.
- Some noise is worth keeping. Noise reduction has costs, rules can harm dignity and morale, and the right level of noise is a decision, not a default.
The vocabulary: fourteen terms defined for practitioners
| Term | What the authors mean | What it looks like in an audit |
|---|---|---|
| Noise | Unwanted variability in judgments that should be identical | Inconsistent ratings, reserves or conclusions on comparable cases |
| Bias | Shared error: judgments that miss in the same direction | A rating standard that is consistently too lenient or too harsh |
| Error equation | Mean squared error equals bias squared plus noise squared | The reason a consistency test is worth running even when accuracy cannot be tested |
| System noise | Variability among the judges of a system on the same cases | The spread across credit officers, claims handlers or auditors |
| Level noise | Differences between judges’ average levels | One team grading consistently more favourably than another |
| Pattern noise | Judges’ different reactions to particular cases, split into a stable part and occasion noise | Disagreement concentrated in particular types of case |
| Occasion noise | Variability within one judge from one occasion to another | The same rater changing a grade when the file comes back unrecognised |
| Noise audit | A test in which several judges assess the same cases and the spread is measured | An inter-rater consistency test; see the worked example |
| Decision hygiene | Procedures that prevent judgment errors without identifying them | Independent first ratings, sequencing, anchored scales, structured criteria |
| Mediating assessments protocol | Assessments defined in advance, made independently, with the overall judgment delayed | Component scoring before an overall rating or engagement conclusion |
| Respect-expert | An expert whose authority rests on peers’ regard rather than on verified accuracy | A specialist whose judgments have never been checked against outcomes |
| Decision observer | A person who watches a decision process with a checklist for signs of error | A quality reviewer attending a rating or risk assessment meeting |
| Objective ignorance | The limit on prediction set by what cannot be known from the information available | The share of credit losses that no rating process could have foreseen |
| Singular decision | A decision made once rather than repeatedly, still subject to noise | A one-off acquisition, restructuring or crisis decision |
What it means for internal audit: judgment as an auditable process
Audit programmes often treat a judgment-based control as a single act: the reviewer reviewed, the grade was assigned, the reserve was approved, and the auditor checks that the act occurred and was evidenced. The book supplies a second question that is rarely asked: if a different, equally qualified person had made the same judgment on the same facts, would the answer have been the same? Where it would not, the control’s output depends on who happened to perform it, and the organisation’s reported numbers carry an error that no single test of operation will reveal. The table maps the book’s method onto the places in an audit universe where judgment does the work. The management review controls guide, the credit risk guide and the finding severity ratings guide carry the surrounding procedures.
| Audit activity | What the book implies | Procedure it changes |
|---|---|---|
| Auditing judgment-based processes such as credit grading, claims reserving, impairment and underwriting | Consistency is measurable and, in the book’s evidence, worse than management expects | Add a noise audit: give the same anonymised cases to several qualified people and measure the spread; the underwriting audit guide and the claims audit guide describe the processes to test |
| Credit risk review at a bank | Independent review of ratings reduces noise only if the reviewers are consistent themselves | Under the 2020 interagency guidance on credit risk review, lending staff assign ratings and an independent review validates them, and internal audit audits the review rather than performing it; test the reviewers’ own consistency as part of that audit |
| Management review controls | The reviewer’s judgment is the control, and its noise is the control’s precision | For a sample of reviews, have a second qualified reviewer assess the same items blind and compare conclusions |
| Rating the function’s own findings | Finding ratings are judgments subject to level, pattern and occasion noise | GIAS Standard 14.3 requires each finding’s significance to be assessed and findings ranked; run a noise audit on the function’s ratings and anchor the scale with reference findings; see the GIAS Domain V guide |
| The annual risk assessment | Scoring auditable entities in a workshop invites group amplification | Collect independent scores before the meeting, discuss differences, then score again; the risk assessment playbook gives the method |
| Engagement supervision and quality assurance | Sequencing and the decision observer reduce noise in review | Standard 12.3 requires the chief audit executive’s methodologies to address supervision and quality assurance; build blind re-performance of a sample of conclusions into the QAIP |
| Models and scoring rules | Algorithms remove noise but can embed bias | Where a model replaces judgment, the audit question becomes model risk; the model risk management audit guide covers the tests |
| Recruiting and appraising auditors | Hiring and performance ratings are two of the domains the book examines, and it finds both noisy | Use structured interviews with separately scored components and anchored appraisal scales, as GIAS Standard 10.2 on human resources management would support; see the interviewing guide for the fieldwork equivalent |
Two cautions keep the map honest. A noise audit measures consistency, not correctness: a team of credit officers can agree perfectly on a grading standard that is too lenient, and agreement then hides a bias rather than revealing a control. The audit must therefore pair the consistency test with whatever test of accuracy the domain allows, such as back-testing grades against later defaults or reserves against settled claims. And noise reduction is a management decision, not an audit one. The function’s role is to measure and report the noise, show what it costs and test the remedies management chooses; it should not design the rating scale it will later audit. For models, SR 11-7 (2011) gave internal audit the parallel role of assessing the overall effectiveness of the model risk management framework; the SR 11-7 guide covers it.
The criticisms, and what the book leaves to the reader
The book was widely reviewed and its reception mixed admiration for the argument with complaints about the execution. Three lines of criticism matter for an auditor. The first concerns novelty and evidence. Blastland argued in Prospect that noise in this sense was already fashionable jargon and that the phenomenon has long been known as subjectivity; the economist Andreas Ortmann, writing on Medium in June 2021, argued that the error equation and its noise term were set out in earlier work, including a 2015 paper by Brighton and Gigerenzer, that the book does not credit, and found the book too long for its message; and Gelman, as noted above, criticised its reliance on weak studies. The second concerns the concept. If there is no true value, as in many evaluative judgments, it is not obvious that scatter is error, and Blastland pointed out that converging on a wrong figure can be worse than keeping a spread when the costs of error are asymmetric, and suggested, tentatively, that forcing psychiatrists to agree could harm patients if the shared diagnosis were wrong. The third concerns machines. Vasant Dhar, reviewing the book in the Los Angeles Review of Books in August 2021, argued that it does not resolve how work should be divided between people and algorithms, and the reviewer for the LSE Review of Books noted that algorithms can produce prejudiced outcomes of their own and cited Gigerenzer’s view that heuristics need not produce bias or noise.
The criticisms sharpen rather than undermine the book’s use in audit. The asymmetric-cost point is exactly why a noise audit must be paired with an accuracy test, and why an audit report should never recommend consistency for its own sake. The evidence point is why the function’s own measurement should carry the argument. And the book is written for a general readership, so an auditor applying it has to add the audit design: when measured noise becomes a finding, how to report it to an audit committee in terms of money and risk rather than standard deviations, and which line should own the remedy. Those are audit design questions, and the worked example below shows one set of answers. Readers who want the individual psychology the book builds on will find it in the Thinking, Fast and Slow summary, and the forecasting chapter’s debt to Philip Tetlock is explored in the Superforecasting summary.
Editions, reading time and how to read it
| What | Details | Best for |
|---|---|---|
| US hardback | Little, Brown Spark, on sale 18 May 2021; ISBN 9780316451406 | The first edition |
| UK edition | William Collins, 2021 | UK readers |
| US trade paperback | Little, Brown Spark, 31 May 2022; the publisher lists 464 pages and describes no added material | The edition most readers will find |
| Other formats | A large-print hardback and an ebook from the same publisher | Accessibility and travel |
| How long it is | In the US hardback the main text and appendices end before page 395, and acknowledgments, notes and index take up the remaining sixty or so pages | Planning: several sittings rather than one |
| Reading plan | The introduction, Part I and the chapters on measuring and analysing noise; then Part V, the review and conclusion and Appendix A; Parts III, IV and VI as interest allows | An auditor who wants the method first |
| Companion reading | The Thinking, Fast and Slow summary for the psychology; the Superforecasting summary for aggregation and calibration | Placing the argument |
Read it with one question in mind: where in my organisation do two qualified people make the same kind of judgment on the same kind of case, and has anyone ever compared their answers? The answer is usually a list, and the list is an audit plan. Appendix A on conducting a noise audit is short and practical, and the audit team that reads only one part closely should read that one, with the chapter on the insurer as its case study.
A one-hour discussion guide for an audit team
- Ten minutes: the prediction. Before anything else, each person writes down how often two auditors in the function would give the same rating to the same finding. Collect the guesses; they are the equivalent of the insurer’s executives’ 10 percent.
- Fifteen minutes: a small noise audit. Hand out three anonymised findings from past reports, with the ratings removed. Everyone rates them independently on the function’s scale. Reveal the spread and compare it with the predictions.
- Ten minutes: decompose. Ask whether the disagreement is level (some people rate everything higher) or pattern (people disagree about particular findings), and what in each finding produced the pattern.
- Ten minutes: hygiene. Choose two techniques from the decision hygiene table that the function could adopt next quarter, such as independent first ratings or a reference library of rated findings, and name an owner.
- Fifteen minutes: the plan. List the organisation’s judgment-heavy processes, such as credit grading, reserving, impairment, vendor risk tiering and control self-assessment, and decide which deserves a noise audit in the 2027 plan; the RCSA guide covers one of the most common candidates.
Worked example: Lakeshore Bancorp audits its loan ratings and its own
Lakeshore Bancorp is a fictional US bank with $9 billion of assets, a commercial loan book that carries most of its credit risk and an internal audit function of twenty-two. Its loan risk ratings use a nine-grade scale on which grades 1 to 6 are pass grades, 7 is special mention, 8 substandard and 9 doubtful. Lending officers assign the grades and an independent credit risk review function, reporting to the board’s risk committee, validates them. In the 2027 plan the function scheduled an audit of credit risk review and, as a framing exercise before writing the programme, ran a noise audit. It took twenty-four commercial credit files from the previous year, removed the assigned grades and asked ten people to grade each one independently: six lending officers and the four credit risk reviewers. Six of the files went back to the same raters eight weeks later, presented as new cases. Before the results were opened, the chief credit officer predicted that two raters would typically differ by half a grade.
| Test | What the function found | What changed |
|---|---|---|
| How far apart are two raters? | Two randomly chosen raters’ grades for the same file differed by 1.3 grades on average, more than two and a half times the chief credit officer’s prediction | Finding: a repeat noise audit each year, with the result reported by credit risk review to the risk committee |
| Do differences cross the line that matters? | On 7 of the 24 files at least one rater gave a pass grade and at least one a criticised grade; those files carried $186 million of commitments, so criticised-asset totals and allowance inputs depended on who graded them | Finding: files within one grade of special mention to be graded independently by the lending officer and a reviewer before any discussion, with disagreements escalated |
| Is it level or pattern? | One lending team graded on average 0.8 of a grade more favourably than the panel; level differences explained about a third of the variance and pattern differences the rest, concentrated in files that relied on guarantor support or commercial real estate values | Observation: guarantor and real estate criteria rewritten with worked reference files at each grade |
| Do raters agree with themselves? | Of the sixty re-gradings (six files, ten raters), seventeen differed from the rater’s own first grade, four of them by two grades | Supports the second-rating requirement above; occasion noise reported to the committee in plain terms |
| Are reviewers more consistent than lenders? | The reviewers differed from each other by 0.9 of a grade on average against 1.5 among the lending officers, but their grades were on average half a grade harsher | Observation: consistency is not accuracy; a three-year back-test of grades against later downgrades and defaults added to the programme |
| Are the function’s own ratings consistent? | Eighteen auditors rated twelve anonymised 2026 findings on the four-level scale; two auditors gave the same rating 49 percent of the time, five findings drew ratings across three levels, and the IT audit team rated about half a level higher than colleagues | Finding against the function itself: a reference library of rated findings, separate impact and likelihood assessments, independent ratings by the in-charge and the manager before review, and a repeat audit in 2028 |
The audit of credit risk review that followed produced two findings and two observations, and the exercise changed its emphasis. The original programme had tested whether reviews were completed on schedule and whether downgrades were approved; the revised one tested whether the review produced consistent grades, whether the grades that mattered most were the ones most protected from noise, and whether anyone was measuring accuracy as well as agreement. The reviewers were the most consistent raters and also the harshest, and whether that reflected rigour in the reviewers or leniency in the lenders was a question only the back-test could answer. The audit report said so rather than pretending the noise audit had settled it, which is the discipline the criticisms of the book recommend.
The chief audit executive took the function’s own results to the audit committee in the same meeting as the credit results, on the reasoning that a function asking the bank to measure its judges should measure its own. The committee chair’s first question was why the IT team rated higher, and the answer, that its findings were written against a different set of reference cases, was itself a finding. The function adopted the changes, mapped them to its methodology under GIAS Standards 9.3 and 14.3, and recorded the 49 percent figure as the baseline for the 2028 repeat. The finding severity ratings guide describes the calibrated matrix the function moved to, and the self-assessment guide describes where the result sits in the function’s quality programme.
Frequently asked questions
What is the difference between noise and bias?
Bias is shared error: judgments that miss in the same direction. Noise is the variability of judgments that should agree. A team of credit officers who all grade too leniently is biased; a team whose grades for the same file differ widely is noisy. Both add to overall error in the same way, and they need different remedies.
Do you need to know the right answer to measure noise?
No. A noise audit gives the same cases to several qualified people and measures the spread of their answers, which requires no outcome data. That is what makes it practical for an audit function. Measuring bias does require a standard of accuracy, so a complete test pairs the noise audit with an accuracy test such as a back-test.
How does Noise relate to Thinking, Fast and Slow?
Thinking, Fast and Slow is about the systematic errors of individual thinking; Noise is about the variability across and within the people making the same judgment, though its fourth part connects that variability to the heuristics and biases of the earlier book. The Thinking, Fast and Slow summary covers the first.
Is the book’s evidence reliable?
The studies on sentencing, medicine and fingerprint examination rest on published research, and the insurance figures come from the authors’ own noise audit at one company. Critics have pointed out that some of the studies the book cites have been challenged, and have questioned its reliance on a contested literature. The practical answer for an auditor is to run a noise audit in the organisation and rely on that.
Should judgment be replaced by algorithms?
The authors argue that rules and algorithms reduce noise and often match or beat experts, but that they are not always practical, can carry bias from their data and can deny people a hearing from a person; their last part argues that some noise is worth keeping. Where a model does replace judgment, the risk moves to the model, which the model risk audit guide covers.
Where does the book sit in the reading list?
Beside Thinking, Fast and Slow and Superforecasting, as a book on judgment; the internal audit reading list organises the books it recommends by theme and career stage.
Related guides
- Finding Severity Ratings: The Calibrated Matrix — anchoring the function’s own ratings
- Credit Risk: Ratings, PD/LGD/EAD, CECL, Capital and How Internal Audit Tests It — the rating process a noise audit tests
- Management Review Controls: Documenting and Testing the Hardest Controls in SOX — judgment as a control
- Model Risk Audit: How Internal Audit Validates the Models Management Relies On — when rules replace judgment
- Thinking, Fast and Slow by Daniel Kahneman: Summary for Auditors — the psychology the book builds on
- Superforecasting by Tetlock and Gardner: Book Summary and Review — aggregation and measured judgment
- The Best Books for Internal Auditors — where this book sits among the others
- All book summaries — the full set of summaries for auditors
- Thinking in Bets by Annie Duke: Summary and Lessons for Auditors — judging decisions apart from their outcomes.
- How to Measure Anything by Douglas Hubbard: Summary and Lessons for Auditors — calibrated estimates and the value of information.
Leave a Reply