Why Your 90% Confidence Is Probably 70%

The Gap Is Bigger Than You Think

Ask someone to give you a range they are 90% confident contains the right answer—say, the year a famous building was completed, or next quarter's revenue—and something predictable happens. They are right roughly 50 to 70% of the time, not 90%.

This is one of the most replicated findings in judgment research, documented across decades by Lichtenstein, Fischhoff, Alpert, Raiffa, and others. It is not a quirk of naive subjects. Experts show the same pattern. The label for it is overprecision: your stated confidence exceeds your actual accuracy, and your intervals are too narrow to reflect genuine uncertainty.

The practical consequence is that when you say I'm almost certain, you are probably describing something closer to a coin flip with good odds. When you say I'm fairly confident, you may be describing something close to a guess. The words feel accurate from the inside. The record says otherwise.

Why This Happens

The core problem is that confidence is generated from the inside—from how coherent your reasoning feels, how easily examples come to mind, how much you know about the topic. None of those signals reliably track whether you will be right.

Philip Tetlock's long-running research on expert forecasters found that domain experts, people paid to have informed opinions, performed barely better than chance on political and economic predictions, and were systematically overconfident. The experts who did well shared one habit: they tracked their accuracy over time and updated their sense of how reliable their own judgment was.

The mechanism matters here. Overconfidence is not cured by trying harder to be humble or by reminding yourself that you might be wrong. Those are intentions. What changes behavior is feedback with memory—seeing, repeatedly, that your 90% calls land at 70%, until that gap becomes part of how you think about your own estimates.

What Scoring Does

A proper scoring rule is a mathematical tool that rewards honest probability reporting. The most common one used in forecasting is the Brier score: you take the difference between your stated probability and the outcome (0 or 1), square it, and average across predictions. Lower is better. A forecaster who says 90% on things that happen 70% of the time will score worse than one who says 70%—the math penalizes both overconfidence and underconfidence.

The important property is that you cannot game a proper scoring rule by inflating or deflating your stated confidence. Your best strategy is always to report what you actually believe. This makes it a useful mirror: over enough predictions, your score tells you whether your stated probabilities match reality.

For a more detailed treatment of how Brier scores work and how to calculate your own, see Brier Scores Explained.

How to Start Closing the Gap

The method is simple, though it requires patience because you need enough predictions to see a pattern.

Keep a prediction log. Write down your forecast as a probability, not a verbal label. I think this project will finish on time is not a forecast. I think there is a 75% chance this project finishes by the deadline is one. The number forces you to commit.

Record the outcome. When the event resolves, note what happened. This sounds obvious, but most people never do it. Without outcomes, you have no feedback loop.

Group your predictions by stated confidence. After thirty or forty predictions, look at everything you called 80–90% confident. What fraction came true? If it is 60%, you now have a concrete, personal calibration gap to work with—not a general warning about human psychology, but a measurement of your own judgment.

Adjust your stated probabilities. If your 80% calls land at 60%, start saying 60% when you feel 80% confident. This feels wrong at first. It should. The discomfort is the gap becoming visible.

One additional tool: before making a forecast, look at the base rate for similar events. If you are estimating whether a software project will ship on time, the relevant question is not just how does this project look? but what fraction of similar projects ship on time? That outside view is a check on the inside-view confidence that tends to inflate your estimates. Start With the Base Rate covers this in detail.

What Improvement Looks Like

Calibration research by Lichtenstein and Fischhoff found that people who receive repeated feedback on their probability estimates do improve over time. The improvement is not dramatic after a single session—it accumulates. Tetlock's superforecasters were not people with unusual intelligence or access to better information. They were people who had made enough tracked predictions to develop an accurate internal model of their own reliability.

The goal is not to become less confident. It is to become accurately confident—to say 85% when you will be right 85% of the time, and to know the difference between that and the feeling of certainty that currently reads as 90%.

If you want to extend this practice to decisions with longer time horizons, the same calibration discipline applies to planning estimates. The planning fallacy—the tendency to underestimate how long and costly things will be—is overconfidence applied to your own future actions. How to Make Better Predictions addresses how to build these habits more systematically.

The One Thing to Take Away

Your confidence intervals are probably too narrow. The fix is not to think harder about uncertainty in the abstract—it is to score yourself on enough specific predictions that the gap between your stated confidence and your actual accuracy becomes a number you have seen with your own data. Once you have seen it, you cannot unsee it, and your estimates will start to shift.