Superforecasting: The Art and Science of Prediction, by the psychologist Philip Tetlock and the journalist Dan Gardner, reports on a very large study of forecasting accuracy: a four-year tournament sponsored by the United States intelligence community in which tens of thousands of volunteers answered hundreds of questions about world events, had their answers scored, and were compared with each other and with professional analysts. A small group of ordinary people, the superforecasters of the title, turned out to be consistently and measurably better than everyone else, including analysts with access to classified information, and the book is about what they did differently. For an internal auditor the subject is closer to home than it looks: an audit opinion is a forecast about whether controls will hold, a risk assessment is a set of forecasts about losses, and a remediation date is a forecast about management. This guide summarises the book without reproducing it, sets out the habits that produced accuracy, maps them to audit practice, and ends with a worked example of a hospital system’s audit function keeping score on its own predictions.
In this guide
- The book in one page
- Who Tetlock and Gardner are, and where the book came from
- The tournament: how accuracy was measured
- The structure: twelve chapters
- What superforecasters do: the ten habits
- The eight ideas worth keeping
- The vocabulary: fourteen terms defined for practitioners
- What it means for internal audit: three forecasts the function already makes
- Three cases from the book, read as audit situations
- The criticisms, and the limits of the method
- Editions, reading time and how to read it
- A one-hour discussion guide for an audit team
- Worked example: Ashgrove Health System keeps score
- Frequently asked questions
- Related guides
The book in one page
| Question | The answer, in our words |
|---|---|
| What is the finding? | That forecasting accuracy is a real, measurable and improvable skill; that a minority of people are much better at it than experts and than chance; and that what they do can be described, taught and partly reproduced by others. |
| Where does the evidence come from? | The Good Judgment Project, the entry Tetlock led with Barbara Mellers in a forecasting tournament run from 2011 to 2015 by the intelligence community’s research agency, in which thousands of volunteers made probabilistic forecasts on geopolitical questions that were scored against what happened. |
| Who are the superforecasters? | The top two percent of the project’s volunteers: retirees, engineers, a film-maker, people with no special access, who beat the field by wide margins year after year and improved when put in teams. |
| What do they do? | They break questions into parts, start from base rates, update in small steps as evidence arrives, express probabilities in fine gradations, seek out disagreement, treat beliefs as hypotheses to be tested, and keep score. |
| What does the book argue against? | The hedgehog style of the confident expert with one big idea, the media culture that rewards it, and vague forecasts that can never be scored, which the book treats as the main reason prediction has not improved. |
| Why read it as an auditor? | Because the function’s central products are forecasts stated without probabilities and never scored, and because the habits that made the superforecasters accurate are habits an audit team can adopt in a quarter. |
| Reading time | About nine hours for the paperback; the appendix of ten commandments is the summary the authors wrote themselves. |
Who Tetlock and Gardner are, and where the book came from
Tetlock is a professor at the University of Pennsylvania whose earlier book, Expert Political Judgment (2005), reported a twenty-year study in which hundreds of experts made tens of thousands of forecasts about political and economic events and were, on average, barely better than chance, with the most famous and most confident experts the worst. That study popularised the fox-and-hedgehog distinction the book is known for, borrowed from Isaiah Berlin, and the much-quoted line comparing the average expert to a dart-throwing chimpanzee, which Tetlock says was widely misread as showing that expert forecasts are useless. The Good Judgment Project followed, and Superforecasting, written with Gardner, a Canadian journalist and author of two earlier books on risk and prediction, was published by Crown in September 2015. The project became a commercial forecasting service, and its methods are now used on the forecasting platforms that grew after the book. The authors write for a general reader, with case studies and portraits of individual forecasters, and the research is in the notes.
The tournament: how accuracy was measured
The research agency’s tournament asked questions with a definite answer by a definite date: will a named leader leave office by a given month, will a commodity price exceed a level, will a country test a weapon. Forecasters gave a probability, could revise it daily until the question closed, and were scored with the Brier score, which rewards assigning high probabilities to things that happen and low probabilities to things that do not, and punishes confidence that proves wrong more than hedging that proves unnecessary. Over four years the Good Judgment Project answered about five hundred questions, and the scoring made three things visible that ordinary commentary hides: who was accurate, whether accuracy persisted, and what the accurate people had in common. The project won the tournament by a wide margin, and its top forecasters outperformed professional analysts with classified access on the same questions.
| Design feature | What it did | Why it matters to an auditor |
|---|---|---|
| Questions with resolvable answers | Made forecasts scorable | A control opinion phrased so that it can be checked a year later is scorable; “adequate” is not |
| Probabilities, not yes/no | Allowed degrees of confidence to be measured | A finding’s likelihood of leading to loss, stated as a probability, can be compared with outcomes |
| Brier scoring | Made a confident miss cost far more than a cautious one | The same asymmetry an audit committee applies to an assurance that failed |
| Daily updating allowed | Measured how people responded to new evidence | Follow-up on open issues is updating; the question is whether the function changes its view in step with the evidence |
| Teams and training arms | Tested whether accuracy could be improved deliberately | One hour of training improved accuracy about ten percent; teams improved it further |
| Persistence across years | Separated skill from luck | A function that keeps score for three years can tell whether its judgments are good or merely confident |
The structure: twelve chapters
| Chapter | What it covers | What it contributes |
|---|---|---|
| 1. An optimistic skeptic | Why prediction has a bad name and why the authors think it can improve | The frame |
| 2. Illusions of knowledge | The history of medicine as a field that learned to test itself; the psychology of overconfidence | The case that feedback is what improves a profession |
| 3. Keeping score | The earlier expert study, the Brier score, why vague forecasts cannot be judged | The method |
| 4. Superforecasters | Who they are and how they were identified | The evidence that the skill exists |
| 5. Supersmart? | Intelligence and knowledge help but do not explain the results | The skill is a way of thinking, not a credential |
| 6. Superquants? | They use numbers and fine gradations of probability, but little mathematics | Granularity over sophistication |
| 7. Supernewsjunkies? | Updating: small, frequent, evidence-driven revisions | The core habit |
| 8. Perpetual beta | Grit, self-correction and treating one’s own beliefs as revisable | The temperament |
| 9. Superteams | Why diverse teams that argue well beat individuals | The organisational lesson |
| 10. The leader’s dilemma | Reconciling decisiveness with doubt, with the Prussian and later German army as the example | How to run an organisation on probabilistic thinking |
| 11. Are they really so super? | Taleb’s Black Swan objection and the limits of forecasting | The honest boundary |
| 12. What’s next? | A call for forecasting tournaments in public life | The programme |
| Appendix | Ten commandments for aspiring superforecasters | The summary to keep |
What superforecasters do: the ten habits
The appendix states the method as ten commandments, which the authors immediately tell the reader not to treat as commandments. In our words, and in the order they matter for audit work:
- Triage. Spend effort on questions where effort can improve the answer: not the trivially predictable, not the unknowable, but the middle. An audit plan is a triage of this kind.
- Break big questions into small ones. “Will this control environment hold?” becomes a set of answerable parts: has the key control operated, has the system changed, has the owner changed, what does the exception history say. The book’s term is Fermi-ising, after the physicist Enrico Fermi, famous for rough estimates built from first principles; the classic exercise of the kind is the number of piano tuners in Chicago.
- Start from the outside view. Before considering the particulars, ask how often things like this happen: the base rate. What fraction of remediation plans of this kind are completed on time? Then adjust for the specifics, the inside view, and not the other way round.
- Update often and in small steps. New evidence should move the estimate a little; dramatic swings are usually overreactions, and refusal to move is usually ego. Bayesian in spirit, not in formula.
- Look for the clashing forces. Every question has arguments on both sides; the forecaster’s job is to weigh them rather than to pick a side and defend it.
- Distinguish as many degrees of doubt as the problem allows. Superforecasters used the full scale and meant the difference between 60 and 65 percent; the rest of the field was less fine-grained. Rounding the superforecasters’ forecasts made them less accurate, so the fine distinctions carried real information.
- Balance confidence and humility. Be as confident as the evidence warrants and no more; the Brier score punishes both the coward and the zealot.
- Learn from errors, including the lucky right answers. Post-mortems on misses and on hits that were right for the wrong reason.
- Bring out the best in others. Teams that disagree constructively, share information and challenge each other outperformed individuals by a wide margin; the book’s warning case is the groupthink behind the Bay of Pigs.
- Practice. The skill improved with feedback and deteriorated without it; it is a habit, not an insight.
The eight ideas worth keeping
- Keep score or stop claiming judgment. A profession that never checks its forecasts against outcomes cannot know whether its judgment is good; the book’s history of medicine is the model for one that learned to check.
- Foxes beat hedgehogs. People who draw on many sources and hold views loosely outperform people with one big framework, and the latter are the ones who sound authoritative on television and in board papers.
- The outside view first. Base rates are the single most under-used input in professional judgment, and starting from them corrects the most common bias, which is treating every case as unique.
- Granularity is accuracy. The habit of distinguishing 55 from 65 percent is not false precision; in the data it tracked real differences in outcomes.
- Updating is the skill. The superforecasters’ first estimates were already more accurate than other forecasters’, and they then revised them more often than others did, usually in small steps and in the right direction.
- Teams that argue well win. Diversity of view plus a norm of challenge produced the best scores; the lesson for a review process is that the second reviewer must be expected to disagree.
- Overconfidence is the default and training reduces it. An hour of instruction on base rates and calibration improved accuracy measurably; the Hubbard summary describes the same training in a risk context.
- Know the boundary. The method works for questions resolving within months to a couple of years in domains with feedback; it does not forecast the next decade or the next Black Swan, and the authors say so.
The vocabulary: fourteen terms defined for practitioners
| Term | What it means in the book | What it looks like in an audit |
|---|---|---|
| Brier score | A measure of forecast accuracy: the squared differences between the probabilities given and what happened, summed over the possible outcomes and averaged; on the book’s scale 0 is perfect, a constant 50 percent scores 0.5 and 2 is the worst | The way to score rated findings and control opinions against what happened |
| Calibration | Whether things rated 70 percent likely happen about 70 percent of the time | The test of whether the function’s ratings mean what they say |
| Resolution | Whether the forecaster distinguishes between cases, rather than giving everything the base rate | Whether “high” and “low” ratings actually separate outcomes |
| Fox / hedgehog | Many-sourced, tentative thinkers versus one-framework, confident ones | The reviewer who asks what else could explain this versus the one who knows |
| Outside view | The base rate for the class of event | How often remediation plans of this type finish on time |
| Inside view | The particulars of this case | This owner, this system, this quarter |
| Fermi-ising | Decomposing a question into estimable parts | Breaking “is the control environment adequate” into testable pieces |
| Granularity | Using fine distinctions of probability | Replacing “likely” with a number |
| Bayesian updating | Revising beliefs incrementally with evidence | Changing a rating as follow-up evidence arrives |
| Perpetual beta | Treating one’s own method as always under revision | A methodology that changes when the scores say it should |
| Dragonfly eye | Synthesising many perspectives, like the insect’s compound eye | Consulting the business, the second line and the data before concluding |
| Superteam | A trained team with norms of challenge | A review process that expects disagreement |
| Scope insensitivity | Failing to adjust an estimate when the question’s scale changes | Rating a $1 million and a $50 million exposure the same |
| Hindsight bias | Believing after the fact that one knew all along | Post-mortems that find the warning obvious; the root cause guide guards against it |
What it means for internal audit: three forecasts the function already makes
An audit function makes forecasts every week without calling them that. The table takes the three most important, applies the book’s method, and names a proportionate practice. The professional skepticism guide and the professional judgment guide describe the surrounding discipline.
| Forecast the function makes | How it is usually stated | The book’s method applied | Proportionate practice |
|---|---|---|---|
| The control opinion | “Controls are adequate / need improvement” | A probability that a material control failure will occur in the area within twelve months, stated and recorded | Add the probability to the engagement memo; score it against incidents a year later |
| The finding rating | High, medium, low | A probability of loss and a range, as the Hubbard summary also proposes | One sentence in the finding template; a calibration review of the function’s ratings every year |
| The remediation date | Management’s committed date, accepted or challenged by judgment | The outside view first: what share of plans of this type, owner and complexity finish on time? Then adjust | Keep the base rate from the issue tracking data; state a probability of on-time completion beside each date |
| The risk assessment | Scores and ranks | Decomposed estimates with base rates, updated as the year proceeds, scored at year-end | Covered in the risk assessment guide; the addition is the scorekeeping |
| Reliance on management’s forecasts | Taken as given or challenged on instinct | Ask for management’s track record: how did last year’s forecasts of this kind turn out? | A standard request in every review of budgets, projections and plans |
| The review process | Supervisor signs off | A second view expected to disagree, with the disagreement recorded | The review notes guide‘s challenge step, made explicit |
The cost of the method is low and the discomfort is real, and both are worth stating to the team before the first scorecard rather than after it. Stating a probability invites being wrong in a way “adequate” never does, and the first year of scoring will show that the function’s “high” findings did not lead to losses more often than its “medium” ones, or that its remediation-date judgments were optimistic by months. That is the finding. A function that has read the book treats the first scorecard as the start of the improvement the book documents rather than as an embarrassment to be filed.
Three cases from the book, read as audit situations
The book’s narrative chapters contain three episodes that translate directly into audit situations, and they are worth retelling in that form because the method is easier to adopt when it is attached to a recognisable scene.
The first is the intelligence estimate behind the 2011 raid that killed Osama bin Laden. Analysts briefed the president with widely differing probabilities, by the accounts the book cites, that the man in the compound was the target, and the president concluded that it was a coin toss. The authors use the scene to show a decision-maker, faced with a spread, collapsing it to 50 percent rather than weighing the sources. An audit committee receiving three ratings of the same risk from the first line, the second line and internal audit is in the same position, and the function’s job is to explain why its number differs and what evidence would move it, not to add a fourth opinion.
The second is the history of medicine in chapter two, which the authors tell as the story of a profession that for centuries judged its treatments by the confidence of its practitioners and only in the twentieth century accepted controlled trials. The parallel is uncomfortable and exact: an audit methodology is a treatment, its effectiveness is testable against outcomes, and the profession has mostly preferred the testimony of its practitioners. The Global Internal Audit Standards require quality assessment, and a function that scores its own judgments has the strongest possible evidence to bring to one.
The third is the account of the superteams in chapter nine, in which groups of trained forecasters who were encouraged to disagree outperformed individuals working alone, and the superteams also beat prediction markets. The audit version is the review of a workpaper or a report. A review that exists to approve produces approval; a review in which the second reader is expected to find the strongest objection produces accuracy, and the book’s data suggest the difference is large. The escalation ladder guide describes what to do when the disagreement cannot be resolved at the desk, which is the condition under which the method works best.
The criticisms, and the limits of the method
The most serious criticism is the one the book itself raises in chapter eleven: forecasting tournaments ask questions that resolve within a year or two, in domains where there is news to update on, and the superforecasters’ advantage shrinks as horizons lengthen. Taleb’s objection, that the events that matter most are the ones no tournament question anticipates, stands, and the authors concede it; their reply is that most decisions depend on the forecastable middle, not the tails, and that improving the middle is worth doing. A second criticism is selection: the superforecasters were volunteers who chose to spend hours a week on the task, and the habits that made them accurate may be hard to sustain in people with other jobs, a point that matters for an audit function trying to adopt them. A third is that the book’s portraits of individual forecasters, engaging as they are, can make the method look like a personality type rather than a set of practices, when the training results show the practices transfer.
For an auditor the limit to state plainly is this: the method improves judgments about near-term, feedback-rich questions, which is most of audit work, and says nothing useful about the catastrophic tail, for which the Black Swan summary is the right companion. Used within that boundary it is the best available account of how to be right more often.
Editions, reading time and how to read it
| What | Details | Best for |
|---|---|---|
| Hardback | Crown, September 2015, about 340 pages | The first edition |
| Paperback | Broadway Books, 2016 | The edition most readers own |
| Audiobook | Unabridged, about nine and a half hours | Works well as narrative; the appendix needs the page |
| The appendix | Ten commandments for aspiring superforecasters, a few pages long | The team handout |
| Reading plan | Chapters 3, 7 and 9 and the appendix for the method; the rest for the evidence | An auditor with one evening |
| Companion reading | Expert Political Judgment for the earlier study; the Thinking, Fast and Slow summary for the biases; the Hubbard summary for calibration in risk work | Placing the method |
A one-hour discussion guide for an audit team
- Ten minutes: three forecasts. Each person writes down a probability for three questions that will resolve within a quarter: will a named remediation close on time, will a named system change go live by its date, will a named KRI breach its threshold. Seal them; open them in three months.
- Fifteen minutes: the base rate. Pull the last two years of remediation commitments from the issue tracker and compute the share completed on time, by rating and by owner function. This is the outside view the team has never had.
- Ten minutes: granularity. Take five findings rated “high” and ask each person for a probability of loss within a year. Compare the spread; discuss what the rating was hiding.
- Fifteen minutes: the superteam. Agree one norm for workpaper review that makes disagreement expected rather than tolerated, and one for the planning meeting.
- Ten minutes: the scorecard. Decide what the function will score this year, who keeps the record and when it is reviewed; the interview questions guide is a reminder that the skill is also something to hire for.
Worked example: Ashgrove Health System keeps score
Ashgrove Health System is a fictional non-profit of five hospitals with $2.1 billion in revenue and an internal audit function of twelve. In 2027 the function’s director adopted the book’s method for two things it already did: rating findings and accepting remediation dates. Every finding issued in the first half of the year carried a stated probability of loss within twelve months; every remediation date carried the function’s probability of on-time completion, estimated after looking up the base rate from three years of issue tracking. The estimates were sealed in the tracker and scored at year-end.
| What was scored | The function’s estimates | What happened | What the scorecard showed |
|---|---|---|---|
| Probability of loss within a year, 41 findings | Averaged 35 percent for “high” findings, 15 percent for “medium” | Nine findings were followed by an incident or loss in the area within the year | Calibration was reasonable for high (5 of 14) and poor for medium (4 of 27 against an expected 4): the ratings were separating outcomes, which the function had never been able to show |
| On-time remediation, 58 commitments | Base rate from the tracker: 46 percent on time overall; 31 percent for IT owners; 68 percent for finance owners. Estimates adjusted from these | Twenty-nine of 58 finished on time | The adjusted estimates averaged 52 percent; the outcome was 50 percent; the function’s pre-method practice of accepting dates had implied near-certainty |
| Three sealed quarterly forecasts per auditor | Various | Scored each quarter | Brier scores improved from the first quarter to the fourth for ten of twelve auditors; the two who did not improve had skipped the calibration session |
| Management’s capital-budget forecasts (asked for track record) | Not the function’s estimate: management’s own | Three prior years showed average overruns of 18 percent | The function’s review of the 2028 capital plan cited the record and asked for ranges; the finance committee adopted ranges |
The year’s scorecard went to the audit committee with two conclusions: that the function’s ratings carried information, which was welcome news nobody had tested before, and that its acceptance of remediation dates had been systematically optimistic, which changed how dates were negotiated. The cost was an hour of calibration training, one extra sentence per finding and a lookup per remediation date. The committee chair’s remark, minuted, was that it was the first time the function had reported on its own accuracy, and that she would like it every year. That is the book’s ambition for every profession that makes judgments for a living, applied to one that is paid to.
Frequently asked questions
Is superforecasting just intelligence or expertise?
The study found both help but neither explains the result. Superforecasters were above average in intelligence and general knowledge but not exceptional; what distinguished them was the way they worked, particularly decomposition, base rates, granular probabilities and frequent updating, and those practices improved with training in people who did not have them.
How does this fit with professional skepticism?
Closely. The book’s fox temperament, holding beliefs as hypotheses and seeking disconfirming evidence, is professional scepticism described by a psychologist, and the professional skepticism guide covers the audit version.
Can a small audit function keep score?
Yes; the worked example is a function of twelve. The minimum is one probability per finding, one per remediation date and a year-end comparison with outcomes.
What about forecasts that never resolve, like “is the culture sound”?
The book’s answer is to turn them into questions that do resolve: will a named indicator move, will a named event occur. Anything that cannot be made resolvable cannot be scored and should not be dressed up as a forecast.
Does it conflict with The Black Swan?
They describe different domains. Tetlock’s method improves forecasts of near-term, feedback-rich events; Taleb’s argument is about rare, extreme ones the method cannot reach. Chapter eleven of Superforecasting addresses the conflict directly, and the Black Swan summary gives the other side.
Where does the book sit in the reading list?
In the decision-making group of the internal audit reading list, with Thinking, Fast and Slow for the biases and Hubbard for the risk application.
Related guides
- Professional Skepticism: The Auditor’s Stance — the fox temperament in audit terms
- Professional Judgment in Audit — how judgments are formed and documented
- Audit Issue Tracking: From Spreadsheet to System — the data that gives the base rates
- Internal Audit Risk Assessment: Method, Scoring and Template — the assessment as a set of forecasts
- The Failure of Risk Management: Summary for Auditors — calibration applied to risk assessment
- Thinking, Fast and Slow: Summary and Review — the biases the method corrects
- The Best Books for Internal Auditors — where this book sits among the others
- All book summaries — the full set of summaries for auditors
- Noise by Kahneman, Sibony and Sunstein: Summary and Lessons for Auditors — why judgments that should agree do not
- Thinking in Bets by Annie Duke: Summary and Lessons for Auditors — judging decisions apart from their outcomes.
Leave a Reply