Quantifying Forecast Accuracy: The Brier Score

Published

September 4, 2026

In quantitative research, just as in life, we rarely deal with certainties. Instead, we deal with probabilities. But how do you measure if a probabilistic forecast is actually “good”?

Introducing: the Brier Score.

Unlike simple accuracy (which only asks if you were right or wrong), the Brier Score asks: How confident were you in the right answer?

The Mathematics of Certainty

The Brier Score is essentially the Mean Squared Error (MSE) of a probability forecast:

\[\text{BS} = \frac{1}{N} \sum_{t=1}^{N} (f_t - o_t)^2\]

  • \(f_t\): The forecast probability (e.g., \(0.7\) for a 70% chance of a price increase).
  • \(o_t\): The actual outcome (\(1\) if the price increased, \(0\) if it didn’t).

Interpreting the Result

  • 0.0: Perfect forecast (you assigned 100% probability to everything that happened).
  • 0.25: The “Naive” score (assigning 50% to everything—essentially a coin flip).
  • 1.0: Worst possible forecast (you were 100% sure the opposite would happen).

Example

Let’s consider a practical model for the Brier Score - Bitcoin (BTC) prices.

First, let’s retrieve one year’s worth of BTC data. Then, for each day, we will create a label for whether the next day’s closing price is higher than today’s.

This is a teaching example: we will score an illustrative momentum rule, not test an investment strategy.

import yfinance as yf
import numpy as np
import pandas as pd

# Download one year of daily BTC closes.
btc = yf.download('BTC-USD', period='1y', interval='1d', progress=False)
close = btc['Close']
if isinstance(close, pd.DataFrame):
    close = close.iloc[:, 0]

df = pd.DataFrame({'close': close})
# Outcome: 1 when tomorrow closes above today; the final day has no known outcome.
df['outcome'] = (df['close'].shift(-1) > df['close']).astype(float)
df.loc[df.index[-1], 'outcome'] = np.nan

# Input: whether today's close rose from yesterday's close.
df['today_up'] = (df['close'] > df['close'].shift(1)).astype(float)
df.loc[df.index[0], 'today_up'] = np.nan
df = df.dropna(subset=['outcome', 'today_up']).copy()
assert df['outcome'].isin([0.0, 1.0]).all()
assert df['today_up'].isin([0.0, 1.0]).all()
print(f"BTC data through {df.index.max().date()}; scored days: {len(df):,}")
BTC data through 2026-09-03; scored days: 364

Scenario 1: The Naive Baseline

If we have no view on tomorrow’s direction, we can assign a constant 50% probability of an increase.

df['naive_prob'] = 0.5
naive_bs = np.mean((df['naive_prob'] - df['outcome'])**2)
print(f"Naive Brier Score: {naive_bs:.4f}")
Naive Brier Score: 0.2500

Scenario 2: A Simple Momentum Rule

Let’s construct an illustrative momentum rule. These 60% and 40% probabilities are chosen for demonstration, not estimated from the data.

  • If BTC rose today, forecast a 60% chance that it rises tomorrow.
  • If BTC fell today, forecast a 40% chance that it rises tomorrow.
df['momentum_prob'] = np.where(df['today_up'] == 1, 0.6, 0.4)
momentum_bs = np.mean((df['momentum_prob'] - df['outcome'])**2)
print(f"Momentum Brier Score: {momentum_bs:.4f}")
Momentum Brier Score: 0.2556
results = pd.DataFrame({
    'Forecast': ['Neutral 50%', 'Illustrative momentum'],
    'Brier score': [naive_bs, momentum_bs],
})
print(results.to_string(index=False, formatters={'Brier score': '{:.4f}'.format}))
             Forecast Brier score
          Neutral 50%      0.2500
Illustrative momentum      0.2556

Takeaway

The Brier score gives us a better way to judge probability forecasts than simply counting correct guesses. In this one-year BTC example, the illustrative momentum rule scored slightly worse than a neutral 50% forecast, so its confidence did not improve the forecast on this sample.

That does not mean momentum can never matter in markets; it means that a rule should earn our confidence through measured results, not intuition alone.