In quantitative research, just as in life, we rarely deal with certainties. Instead, we deal with probabilities. But how do you measure if a probabilistic forecast is actually “good”?
Introducing: the Brier Score.
Unlike simple accuracy (which only asks if you were right or wrong), the Brier Score asks: How confident were you in the right answer?
The Mathematics of Certainty
The Brier Score is essentially the Mean Squared Error (MSE) of a probability forecast:
\(f_t\): The forecast probability (e.g., \(0.7\) for a 70% chance of a price increase).
\(o_t\): The actual outcome (\(1\) if the price increased, \(0\) if it didn’t).
Interpreting the Result
0.0: Perfect forecast (you assigned 100% probability to everything that happened).
0.25: The “Naive” score (assigning 50% to everything—essentially a coin flip).
1.0: Worst possible forecast (you were 100% sure the opposite would happen).
Example
Let’s consider a practical model for the Brier Score - Bitcoin (BTC) prices.
First, let’s retrieve one year’s worth of BTC data. Then, for each day, we will create a label for whether the next day’s closing price is higher than today’s.
This is a teaching example: we will score an illustrative momentum rule, not test an investment strategy.
import yfinance as yfimport numpy as npimport pandas as pd# Download one year of daily BTC closes.btc = yf.download('BTC-USD', period='1y', interval='1d', progress=False)close = btc['Close']ifisinstance(close, pd.DataFrame): close = close.iloc[:, 0]df = pd.DataFrame({'close': close})# Outcome: 1 when tomorrow closes above today; the final day has no known outcome.df['outcome'] = (df['close'].shift(-1) > df['close']).astype(float)df.loc[df.index[-1], 'outcome'] = np.nan# Input: whether today's close rose from yesterday's close.df['today_up'] = (df['close'] > df['close'].shift(1)).astype(float)df.loc[df.index[0], 'today_up'] = np.nandf = df.dropna(subset=['outcome', 'today_up']).copy()assert df['outcome'].isin([0.0, 1.0]).all()assert df['today_up'].isin([0.0, 1.0]).all()print(f"BTC data through {df.index.max().date()}; scored days: {len(df):,}")
BTC data through 2026-09-03; scored days: 364
Scenario 1: The Naive Baseline
If we have no view on tomorrow’s direction, we can assign a constant 50% probability of an increase.
The Brier score gives us a better way to judge probability forecasts than simply counting correct guesses. In this one-year BTC example, the illustrative momentum rule scored slightly worse than a neutral 50% forecast, so its confidence did not improve the forecast on this sample.
That does not mean momentum can never matter in markets; it means that a rule should earn our confidence through measured results, not intuition alone.