TL;DR / Key Takeaways

Brier score is mean squared error for probabilistic predictions. Lower is better. 0.25 is a coin flip.
Always compare your model's Brier score against the base rate. Beating a coin flip is not the bar. Beating the historical frequency is.
My Weather Bot v2.1 scored 0.2858. Simply guessing the historical base rate scored 0.2439. My model was statistically worse than making no prediction at all.
A confident wrong model is worse than an honest "I don't know." Brier score punishes overconfidence directly.

Why Accuracy Is the Wrong Metric

If you build a model that predicts "yes" or "no" and you measure it with accuracy, you will fool yourself.

Here is why. Say 70% of your events resolve "yes." A model that predicts "yes" every single time gets 70% accuracy. That model has zero intelligence. It learned nothing. It cannot lose because it never takes a position that could be wrong. And by the standard metric, it looks pretty good.

Prediction markets are probabilistic. You are not saying yes or no. You are saying "I think this resolves yes with 73% probability." That is a fundamentally different claim and it needs a fundamentally different measurement.

Brier score is that measurement.

What the Brier Score Actually Is

The formula is simple. For each prediction, you take the squared difference between your predicted probability and the actual outcome. The outcome is either 1 (happened) or 0 (did not happen). You average those squared errors across all predictions.

BS = (1/n) * sum((p_i - o_i)^2)

Where:

p_i is your predicted probability for event i (a number between 0 and 1)
o_i is the actual outcome for event i (1 or 0)
n is the total number of predictions

That is it. No black box. No hyperparameters. No secret sauce.

The score ranges from 0 to 1. Lower is better. A perfect model scores 0. A model that predicts 1.0 probability on every event that resolves false scores 1.0. A model that always predicts 0.5 (pure uncertainty) scores 0.25.

That 0.25 number is important. It is the theoretical floor for a model that knows nothing. It is also your first bar to clear.

Interpreting the Scale

| Score | What It Means | |---|---| | 0.00 | Perfect. Every prediction was exactly right. | | 0.00 - 0.10 | Excellent. Professional weather forecast territory. | | 0.10 - 0.20 | Good. The model is genuinely adding signal. | | 0.20 - 0.25 | Marginal. You are near the noise floor. | | 0.25 | Coin flip. You predicted 50% on everything. | | 0.25+ | You are now worse than saying "I don't know." | | 1.00 | You predicted maximum confidence on every wrong outcome. Impressive in the worst way. |

The coin flip at 0.25 is a fixed benchmark. But the more important benchmark is your base rate.

The Base Rate Comparison

The base rate is the historical frequency of the event resolving "yes." If 62% of "high temperature above 85F" contracts resolve yes, then your base rate probability is 0.62.

A model that predicts 0.62 for every single contract scores:

BS_base = (1/n) * sum((0.62 - o_i)^2)

This is the real bar. Not 0.25. Not 0.0. The question is: does your model beat a know-nothing predictor that just memorized the historical frequency?

If your model cannot beat the base rate, it is not a model. It is a liability. It adds complexity, computational cost, and false confidence with no actual signal.

Python Implementation

Here is a clean implementation you can drop into any project. No fancy libraries required, though I will show the sklearn version too.

import numpy as np
from typing import Sequence


def brier_score(
    predictions: Sequence[float],
    outcomes: Sequence[int]
) -> float:
    """
    Calculate the Brier score for a set of probabilistic predictions.

    Args:
        predictions: Predicted probabilities, each in [0.0, 1.0]
        outcomes: Actual outcomes, each 0 or 1

    Returns:
        Brier score (lower is better, 0.25 = coin flip)
    """
    p = np.array(predictions, dtype=float)
    o = np.array(outcomes, dtype=float)

    if len(p) != len(o):
        raise ValueError(
            f"Length mismatch: {len(p)} predictions, {len(o)} outcomes"
        )

    if np.any((p < 0) | (p > 1)):
        raise ValueError("All predictions must be between 0.0 and 1.0")

    if not np.all(np.isin(o, [0, 1])):
        raise ValueError("All outcomes must be 0 or 1")

    return float(np.mean((p - o) ** 2))


def base_rate_brier_score(outcomes: Sequence[int]) -> float:
    """
    Calculate the Brier score for a naive base-rate predictor.
    This is your minimum bar. If your model doesn't beat this,
    your model has no skill.

    Args:
        outcomes: Actual outcomes, each 0 or 1

    Returns:
        Brier score for predicting the base rate on every event
    """
    o = np.array(outcomes, dtype=float)
    base_rate = np.mean(o)
    base_predictions = np.full_like(o, base_rate)
    return float(np.mean((base_predictions - o) ** 2))


def evaluate_model(
    predictions: Sequence[float],
    outcomes: Sequence[int],
    model_name: str = "Model"
) -> dict:
    """
    Full evaluation: model score, base rate score, and verdict.
    """
    model_bs = brier_score(predictions, outcomes)
    base_bs = base_rate_brier_score(outcomes)
    o = np.array(outcomes, dtype=float)
    base_rate = float(np.mean(o))
    has_skill = model_bs < base_bs

    return {
        "model_name": model_name,
        "n_predictions": len(predictions),
        "base_rate": round(base_rate, 4),
        "model_brier_score": round(model_bs, 4),
        "base_rate_brier_score": round(base_bs, 4),
        "delta": round(model_bs - base_bs, 4),
        "has_skill": has_skill,
        "verdict": "SKILL" if has_skill else "NO SKILL"
    }

Run it on a dataset:

# Simulated predictions and outcomes
# Replace with your actual model output and trade results

predictions = [0.72, 0.55, 0.91, 0.48, 0.63, 0.38, 0.80, 0.44]
outcomes    = [1,    1,    0,    0,    1,    0,    1,    1   ]

result = evaluate_model(predictions, outcomes, model_name="Weather Bot v2.1")

print(f"Model:          {result['model_name']}")
print(f"Predictions:    {result['n_predictions']}")
print(f"Base rate:      {result['base_rate']:.4f}")
print(f"Model BS:       {result['model_brier_score']:.4f}")
print(f"Base rate BS:   {result['base_rate_brier_score']:.4f}")
print(f"Delta:          {result['delta']:+.4f}")
print(f"Verdict:        {result['verdict']}")

If you would rather use sklearn:

from sklearn.metrics import brier_score_loss

# sklearn's brier_score_loss uses the same formula
model_bs = brier_score_loss(outcomes, predictions)

Both give the same result. I wrote the manual version because I want to understand what is being calculated, not just call a function and trust it.

How to Load Your Trade History

If you are logging trades to SQLite the way my bot does, you can pull predictions and outcomes directly from the database:

import sqlite3
import pandas as pd


def load_trade_history(db_path: str) -> pd.DataFrame:
    """
    Load completed trades with model predictions and outcomes.
    Assumes a trades table with columns:
      - predicted_probability: float, model's confidence at trade time
      - contract_side: 'yes' or 'no'
      - settled_outcome: 'win' or 'loss'
    """
    conn = sqlite3.connect(db_path)

    query = """
        SELECT
            predicted_probability,
            contract_side,
            settled_outcome
        FROM trades
        WHERE settled_outcome IS NOT NULL
        ORDER BY created_at ASC
    """

    df = pd.read_sql_query(query, conn)
    conn.close()

    # Normalize: prediction is always the probability of winning
    # If you bought 'no' at 0.65, that means you thought 'no' resolves
    # with 65% probability, so outcome 1 = 'no' resolves
    df["outcome"] = (df["settled_outcome"] == "win").astype(int)

    return df


df = load_trade_history("trades.db")

result = evaluate_model(
    predictions=df["predicted_probability"].tolist(),
    outcomes=df["outcome"].tolist(),
    model_name="My Bot"
)

print(result)

Adjust the column names to match your schema. The logic is the same regardless of how you store it.

What I Found When I Actually Ran This

I ran the Weather Bot for four months. 112 completed trades. Net loss of roughly $23, which sounds small enough to ignore. So I ran the Brier score calculation instead of ignoring it.

My model scored 0.2858.

The base rate scored 0.2439.

Delta: +0.0419. My model was worse than guessing the historical frequency on every single trade.

Here is the part that stings. The model was not producing 0.5 on everything. It was producing near-certainties. Predictions in the 0.90-0.98 range on contracts that resolved correctly about 60% of the time. That is the worst possible outcome for Brier score. High confidence, mediocre accuracy. Every overconfident wrong prediction adds a large squared error. That is exactly what happened.

The market itself, priced by thousands of participants with actual skin in the game, was pricing those same contracts at 60 cents. My model was insisting it knew better. It did not.

Why Overconfidence Is Especially Punished

Look at the squared error for a single prediction:

You predict 0.95, event resolves 1 (correct): squared error = (0.95 - 1)^2 = 0.0025
You predict 0.95, event resolves 0 (wrong): squared error = (0.95 - 0)^2 = 0.9025

That asymmetry is the point. A model that is wrong with high confidence takes a catastrophic hit on every such prediction. A model that says "I am 50% sure" takes a maximum of 0.25 on any single prediction regardless of outcome.

Brier score does not just penalize being wrong. It penalizes being confidently wrong. That is the right behavior for any system where overconfidence has real consequences, including prediction market trading.

If you are building a model for binary markets and it is producing a lot of predictions above 0.85, run the Brier score before you trust it. Those high-confidence predictions need to be right at a matching rate or they are destroying your score.

Calibration vs. Skill

Brier score is actually the sum of two things: calibration and resolution. You can decompose it, but the simple version is enough for most practical purposes.

Calibration measures whether your probabilities match observed frequencies. A calibrated model that predicts 0.70 should be right 70% of the time on that subset of predictions. My old ensemble was systematically overconfident, producing probabilities well above what the outcome rate justified.

Resolution measures whether your predictions actually discriminate between outcomes. A model with high resolution predicts high for events that happen and low for events that do not.

A good model needs both. My v2.1 model had poor calibration (overconfident) and poor resolution (didn't actually discriminate). It just confidently predicted the same general direction as the base rate.

The fix was not a cleverer model. NOAA's National Blend of Models publishes professionally calibrated forecasts for exactly the weather stations Kalshi uses for settlement. They employ people whose entire job is calibration. I was hand-rolling a worse version of that. The rebuilt bot uses NOAA's already-calibrated output and is currently in paper-trading mode undergoing validation.

A Practical Checklist Before You Trust Your Model

Log every prediction with a timestamp, the predicted probability, and the eventual outcome. If you are not logging this, you cannot measure anything.
Run Brier score after at least 50 predictions. Below 30 the variance is too high to draw conclusions.
Calculate base rate Brier score first. That is your actual minimum bar.
Look at your high-confidence predictions (above 0.80) separately. What is the outcome rate in that subset? If it is not above 0.80, your model is overconfident and your Brier score is being dragged down.
Plot predicted probability vs. actual frequency in bins. A calibration plot. A well-calibrated model produces a roughly diagonal line. An overconfident model curves below the diagonal. That plot told me more than any individual metric.
Run this audit before you put real money on it. Not after.

The Brier score is a single number but it is an honest one. It does not care about your intentions. It does not care how sophisticated your model architecture is. It just measures whether your probability estimates correspond to reality.

If they do not, you have work to do. Find out now, not after 112 trades.

The uncomfortable truth about probabilistic modeling is that being wrong is not the problem. Being confidently wrong at scale is the problem. Brier score measures exactly that. Run it on your model before you trust it with anything that matters.

Brier Score Explained: How to Measure If Your Probabilistic Model Actually Works