Writing Questions That Can Actually Be Resolved

A forecast without a scoreable resolution condition is not a forecast. It is an opinion with a date attached.

This distinction matters because the entire point of forecasting practice is feedback. You make a prediction, the outcome arrives, you compare the two, and your calibration improves. If the resolution condition is vague, the feedback loop breaks. You can always find a reading of the outcome that confirms what you believed, which means you learn nothing.

The problem almost always originates at the question-writing stage, not the scoring stage.

What makes a resolution condition unambiguous

A resolution condition is unambiguous when a stranger, given only the question text and a factual record of what happened, would reach the same resolution as you — without needing to ask what you meant.

That is the test. Not whether you know what you meant. Whether someone with no access to your intentions can resolve it correctly from the words alone.

Three things tend to make resolution conditions fail this test:

1. The key term is undefined.

Will the project succeed by Q3? fails immediately. Success is not defined. Q3 of which year? Succeed by whose standard? The question looks concrete because it has a deadline, but the resolution hinge — success — is entirely in the asker's head.

Fix: replace the evaluative term with an observable event. Will the project ship to external users before October 1, 2025? Now a stranger can resolve it by checking a release date.

2. The resolution depends on a judgment call made after the outcome is known.

This is subtler and more dangerous. Will inflation fall significantly in 2025? seems like it could be resolved with data. But significantly will be interpreted in light of whatever actually happens. If inflation drops 0.4 points, was that significant? The answer will be shaped by whether the forecaster wants it to count.

Post-hoc judgment is the enemy of calibration. The resolution condition must be specified precisely enough that it could have been coded into a script before any outcomes were observed.

Fix: use a specific threshold from a named source. Will the US CPI 12-month change, as reported by the BLS, fall below 3.0% in any month of 2025? A script could resolve that automatically.

3. The resolution source is unspecified or unreliable.

Will the economy be in recession? has a hidden problem even if you define recession precisely: who declares it? The NBER's recession dating committee operates with a lag of many months. If your question resolves December 31, 2025, but the NBER hasn't ruled yet, what happens?

Fix: name the source and handle the edge case. Will the NBER Business Cycle Dating Committee announce a recession start date falling within 2025, with the announcement made before December 31, 2026? Longer, but resolvable.

The pre-mortems approach to question writing

Before you finalize a question, run it through two scenarios mentally:

Scenario A: The thing you're forecasting clearly happens. Can you resolve the question Yes without any ambiguity? If you find yourself thinking well, it depends on how you look at it, the condition is underspecified.

Scenario B: The thing clearly does not happen. Can you resolve it No with equal confidence? Sometimes questions are written so that a Yes is easy to identify but a No is murky — or vice versa.

If either scenario produces hesitation, rewrite the condition before you assign a probability.

This matters especially because ambiguous questions tend to resolve in favor of the forecaster's prior belief. If you thought something would happen and it sort of did, you'll call it Yes. If you thought it wouldn't and it sort of didn't, you'll call it No. The question becomes a mirror for your existing views rather than a test of them. Your Brier score will look fine while your calibration quietly degrades.

A worked example

Original question: Will our team's morale improve after the reorg?

Problems: morale is undefined, improve is undefined, the time window is undefined, and there is no resolution source.

Revision 1: Will the average score on the monthly team survey's morale question be higher in the month after the reorg than in the month before?

Better. Now there's a source (the survey), a metric (average score on a specific question), and a comparison window. A stranger could resolve this.

Revision 2: Will the average score on Q4 of the monthly Lattice team survey be higher in the calendar month following the reorg announcement than in the calendar month prior, using the scores as exported from Lattice on the last day of each month?

This is more verbose, but it closes the remaining gaps: which survey platform, which question number, which export date. Whether you need this level of specificity depends on how much is at stake and how likely edge cases are. For a low-stakes personal forecast, Revision 1 is probably fine. For a forecast you're sharing with others or tracking over time, Revision 2 prevents arguments.

The timing rule

One principle cuts across all of this: the resolution condition must be fully specified before you assign a probability.

This sounds obvious. It is routinely violated. The common failure mode is writing a vague question, assigning a probability, watching the outcome unfold, and then deciding what the question really meant based on what happened. This is not forecasting. It is storytelling after the fact.

If you find yourself wanting to refine the resolution condition after you've seen early results, that is a signal to void the forecast entirely and start over with a clean question. Partial information has already contaminated your judgment about what the condition should say.

This connects to a broader point about calibration: most overconfidence doesn't come from bad probability estimates alone. It comes from questions that were quietly adjusted to match outcomes, making the forecaster's record look better than it is.

What good questions enable

When you write questions that resolve cleanly, a few things become possible that weren't before.

You can aggregate forecasts from multiple people without arguing about what the question meant. You can compare your performance across questions over time. You can identify whether you're systematically overconfident in a particular domain. And you can use tools like Brier scores to track improvement, because the scores will actually reflect your accuracy rather than your ability to reinterpret outcomes.

None of that works if the resolution condition was ambiguous. The question is the foundation. Get it wrong and everything built on top of it is unreliable.

Write the resolution condition first. Assign the probability second. Resolve it strictly, using only what the condition says. That sequence is the practice.