< Back to Blog

Probability Calibration for Binary Markets: Why 98% Confidence Should Scare You

TL;DR / Key Takeaways

  • A well-calibrated model's stated probabilities match observed win rates. If it says 90%, it should win roughly 90% of the time.
  • Our original Weather Bot said 98% confidence on markets that resolved correctly about 60% of the time. That's not a good model. That's an overconfident disaster with decent luck.
  • Reliability diagrams are the fastest way to see miscalibration. If your curve sits below the diagonal, you're overconfident. Full stop.
  • In binary prediction markets, the 40-60% zone is where overconfidence kills you. The market already knows it's a coin flip. Your model pretending otherwise costs money.

The Number That Should Have Been a Warning

When I audited 112 completed trades from the original Weather Bot, one figure stood out before I even ran the Brier score.

The model was regularly producing probabilities above 95%. Some trades came in at 98%.

I want to be precise about what that means. A 98% probability is a claim that, across 100 markets where you assign that confidence, you should be wrong twice. Not 20 times. Not 40 times. Twice.

We were not wrong twice.

The bot was buying contracts at 60-cent prices that won 60% of the time. The market had those priced correctly. Our model had them at 95%+. That's not edge. That's a model confusing noise for signal and outputting a number that sounds authoritative.

The Brier score confirmed what the raw numbers already showed: 0.2858 versus 0.2439 for simply guessing the historical base rate. Our model was statistically worse than making no prediction at all.

Extreme confidence was the symptom. Miscalibration was the disease.

What Calibration Actually Means

Calibration is the relationship between predicted probability and observed frequency.

A calibrated model that says "70% chance of X" should see X happen about 70% of the time across all predictions in that probability bucket. Not 90% of the time. Not 50% of the time. Seventy.

This sounds obvious. It almost never holds in practice, especially when you build your own forecasting stack.

For binary markets specifically, where the outcome is exactly one of two states, calibration is everything. You're not predicting a direction with unlimited upside. You're predicting whether a contract worth $1 at settlement resolves YES or NO. The math leaves nowhere to hide.

The formal relationship:

Expected Calibration Error (ECE) = Σ (|bin_count| / n) * |accuracy(bin) - confidence(bin)|

For each probability bucket (say, 0.60-0.70), you're computing the gap between what the model claimed and what actually happened, weighted by how many predictions fell in that bucket. Sum those weighted gaps and you have a single number measuring how wrong your confidence is.

Low ECE means the model knows what it doesn't know. High ECE means the model is lying to itself.

The Reliability Diagram

The fastest diagnostic is a reliability diagram. It takes your model's predicted probabilities, bins them, computes the actual win rate per bin, and plots predicted versus observed.

A perfectly calibrated model is a 45-degree line from (0,0) to (1,1). Points below the diagonal mean overconfidence. Points above mean underconfidence.

Here's the code. It's not complicated. That's the point.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve

def plot_reliability_diagram(y_true, y_prob, n_bins=10, title="Reliability Diagram"):
    """
    y_true: array of binary outcomes (0 or 1)
    y_prob: array of predicted probabilities
    n_bins: number of probability buckets
    """
    fraction_of_positives, mean_predicted_value = calibration_curve(
        y_true, y_prob, n_bins=n_bins, strategy="uniform"
    )

    fig, ax = plt.subplots(figsize=(7, 6))

    # Perfect calibration reference line
    ax.plot([0, 1], [0, 1], "k--", label="Perfect calibration", linewidth=1.5)

    # Your model's calibration curve
    ax.plot(
        mean_predicted_value,
        fraction_of_positives,
        "s-",
        color="#e05c5c",
        label="Model",
        linewidth=2,
        markersize=8
    )

    # Shade the overconfidence region
    ax.fill_between(
        [0, 1], [0, 1], [0, 0],
        alpha=0.05, color="red", label="Overconfidence zone"
    )

    ax.set_xlabel("Mean Predicted Probability", fontsize=12)
    ax.set_ylabel("Fraction of Positives (Observed)", fontsize=12)
    ax.set_title(title, fontsize=13)
    ax.legend(loc="upper left")
    ax.set_xlim(0, 1)
    ax.set_ylim(0, 1)

    plt.tight_layout()
    plt.savefig("reliability_diagram.png", dpi=150)
    plt.show()

    return fraction_of_positives, mean_predicted_value

Call it like this after you've accumulated enough resolved trades:

import pandas as pd

# Pull from your trade database
df = pd.read_csv("resolved_trades.csv")  # columns: predicted_prob, outcome

plot_reliability_diagram(
    y_true=df["outcome"].values,        # 1 if YES resolved, 0 if NO
    y_prob=df["predicted_prob"].values,
    n_bins=10,
    title="Weather Bot v2.1 Calibration (112 trades)"
)

If your curve bends toward the bottom-right, every point below the diagonal is money you lost because your model was more confident than reality justified.

Computing Brier Score

The Brier score is the mean squared error between predicted probability and actual outcome. For binary outcomes:

Brier = (1/n) * Σ (predicted_prob_i - outcome_i)²

Lower is better. A Brier score of 0.25 is exactly what you get from predicting 0.5 on every market. If your model can't beat 0.25, it's not doing anything useful.

def brier_score(y_true, y_prob):
    """
    Returns Brier score (lower = better).
    0.0 = perfect, 0.25 = uninformative (always predict 0.5), 1.0 = perfectly wrong
    """
    y_true = np.array(y_true, dtype=float)
    y_prob = np.array(y_prob, dtype=float)
    return np.mean((y_prob - y_true) ** 2)

def brier_skill_score(y_true, y_prob):
    """
    Brier Skill Score: positive means better than climatology baseline.
    Negative means worse than just guessing the base rate.
    """
    base_rate = np.mean(y_true)
    brier_ref = np.mean((base_rate - y_true) ** 2)
    brier_model = brier_score(y_true, y_prob)
    return 1 - (brier_model / brier_ref)

# Example
outcomes = df["outcome"].values
probs = df["predicted_prob"].values

bs = brier_score(outcomes, probs)
bss = brier_skill_score(outcomes, probs)

print(f"Brier Score:       {bs:.4f}")
print(f"Brier Skill Score: {bss:.4f}  (negative = worse than guessing)")

Our original v2.1 model: Brier 0.2858, Brier Skill Score negative. We were operating a prediction engine that destroyed information rather than producing it.

Why the 40-60% Zone Is the Danger Zone

Here's something that took me longer to internalize than it should have.

In a liquid prediction market, prices near 50 cents are not an opportunity. They're a warning.

Kalshi prices reflect aggregate market belief. A contract trading at 50 cents means the market, in aggregate, thinks this is a coin flip. If your model says 85% on a 50-cent contract, one of two things is true:

  1. Your model has genuine private information the market doesn't have.
  2. Your model is miscalibrated and you're about to overpay for a bet you don't actually have edge on.

Option 1 is almost never true when your data sources are public NOAA feeds. Option 2 is always worth checking before you place the trade.

This is what I call the overconfidence cliff. A model that's overconfident in the 80-95% bucket loses money slowly. A model that's overconfident in the 40-60% bucket loses money fast, because it sizes positions like it has conviction in coin-flip markets.

The math is unforgiving. If you pay 80 cents for a contract that wins 60% of the time, you collect $1.00 when you win and lose $0.80 when you don't. Expected value: (0.60 * $0.20) - (0.40 * $0.80) = $0.12 - $0.32 = -$0.20. You lose 20 cents per contract in expectation, with complete confidence that you know what you're doing.

Detecting Underdispersion

Overconfidence in ensemble models often shows up as underdispersion. The ensemble members mostly agree, the spread is narrow, and the output probability gets pushed toward the extremes.

Our original Weather Bot had 164 ensemble members across 4 sources. When they agreed, the model output probabilities near 0 or 1. The problem is that "agreement among ensemble members" is not the same as "correctness." Correlated models that share the same biases will agree confidently and be wrong together.

Here's a quick check for underdispersion:

def check_dispersion(y_true, y_prob, n_bins=10):
    """
    For a well-calibrated model, the spread of predicted probabilities
    should roughly match the variance in outcomes.
    
    Underdispersion: model probabilities cluster near 0 and 1.
    Overdispersion: model probabilities cluster near 0.5.
    """
    pred_variance = np.var(y_prob)
    obs_base_rate = np.mean(y_true)
    
    # For binary outcomes, max variance is p*(1-p)
    max_variance = obs_base_rate * (1 - obs_base_rate)
    dispersion_ratio = pred_variance / max_variance
    
    print(f"Predicted probability variance: {pred_variance:.4f}")
    print(f"Max variance at base rate:      {max_variance:.4f}")
    print(f"Dispersion ratio:               {dispersion_ratio:.2f}")
    
    if dispersion_ratio > 1.2:
        print("Likely OVERDISPERSED — model hedges too much, probabilities pile near 0.5")
    elif dispersion_ratio < 0.8:
        print("Likely UNDERDISPERSED — model overconfident, probabilities pile near 0 and 1")
    else:
        print("Dispersion looks reasonable")

check_dispersion(outcomes, probs)

We measured 4.2x underdispersion in the v2.1 model. The ensemble was pushing probabilities to the extremes at more than four times the rate that outcomes justified. Every 95% prediction should have been something closer to 65%.

The Fix Is Not a Cleverer Model

Here's the uncomfortable conclusion from the post-mortem.

The fix was not a better calibration layer. It was not Platt scaling or isotonic regression applied on top of the existing ensemble. Those are legitimate techniques, but they require enough in-sample data to fit a calibration curve, and they can't fix a model that lacks predictive skill in the first place.

The fix was replacing the custom ensemble with NOAA's National Blend of Models.

NOAA publishes calibrated, bias-corrected temperature forecasts for free. Professionally calibrated. Covering exactly the weather stations that Kalshi uses for contract settlement. We were hand-rolling a worse version of a public good and paying for it with real money.

If you're building a probabilistic model and the calibrated public data already exists, use it. Your custom ensemble is not smarter than the agency that has been doing this since before you were writing Python.

That's not a discouraging conclusion. It's a practical one. Know what you're actually adding versus what already exists.

When 95%+ Should Be a Red Flag, Not a Green One

I now treat any model output above 90% as a request for a second opinion, not a signal to act.

In practice, that means:

  • Pull the raw forecast data and check the uncertainty range directly, not just the point estimate.
  • Check the market price. If the contract is trading at 55 cents and your model says 95%, the market is almost certainly right and you are almost certainly wrong.
  • Look at the ensemble spread. If all members agree, ask whether they share the same biases rather than whether they have the same information.
  • Check the historical hit rate for predictions in that confidence bucket. If your 90-95% predictions win 65% of the time, your 90-95% bucket is actually a 65% bucket. Price accordingly.

The last point is just reliability diagram logic applied to a single bucket. But it's the one that costs the most when you ignore it.

A Note on Sample Size

112 trades sounds like a reasonable audit sample. It's not enough to split into fine-grained calibration buckets.

For a 10-bucket reliability diagram to be statistically meaningful, you want at least 100 predictions per bucket. That's 1,000 trades minimum, and realistically 2,000+ for the tail buckets near 0 and 1 to have any data in them at all.

At our trading rate, that's a year or more of live data. Paper trading gives you prediction volume without capital at risk, which is why the rebuilt bot is in paper-trading mode now.

Don't try to fit a calibration curve on 50 data points. You'll overfit the curve and underfit the lesson.

Practical Checklist

Before you deploy any probabilistic model against binary markets:

  1. Run a reliability diagram against resolved historical predictions. If you don't have 200+ resolved predictions, you don't have enough to assess calibration.
  2. Compute Brier score AND Brier Skill Score. If BSS is negative, stop.
  3. Check dispersion. If predicted probabilities cluster near 0 and 1, check for shared bias across ensemble members.
  4. Cap acting on predictions above 90% until you have calibration data in that bucket specifically.
  5. Cross-check model output against market price. A 20-point gap should require an explicit reason, not a default to trusting the model.
  6. Log everything, including predictions that didn't trigger a trade. You need full coverage to audit calibration later.

The whole point of building a probabilistic system is that the numbers mean something. If your 80% doesn't win 80% of the time, the number is decoration. And in a binary market, decoration costs money.

Miscalibration is not a model failure you can see from the win rate alone. A 60% win rate looks fine until you realize your model was calling those same trades at 95% confidence. The reliability diagram shows you the gap. The Brier score quantifies it. Neither one is complicated to compute. The hard part is being willing to run them when your model feels like it's working.

Build to be proven wrong. The numbers will tell you the truth if you let them.