Brier Scores Explained

A Brier score measures how accurate your probabilistic predictions are. It
rewards both correctness and honest confidence.

The formula

For a single prediction, the Brier score is:

(predicted_probability - actual_outcome)^2

The outcome is 1 if the event happened and 0 if it did not. Scores range
from 0 to 1, and lower is better. A perfect, fully confident, correct
prediction scores 0. Saying 50% on everything scores 0.25 no matter what
happens.

Why squaring matters

Squaring the error punishes confident mistakes far more than cautious ones.
Being 99% sure and wrong is much costlier than being 60% sure and wrong —
exactly the incentive you want.

Reading your average

Your average Brier score across many predictions tells you whether your
confidence is trustworthy. Track it over time: a falling average means your
calibration is improving, even if individual calls still miss.