< Back to Blog

Confident and Wrong: Why Our Model's High-Confidence Trades Lost Money

TL;DR / Key Takeaways

  • Our v2.1 Weather Bot produced near-certainty predictions (95%+ confidence) that were correct about 60% of the time. That gap is calibration failure, and it is expensive.
  • Nearly all of the $23 loss came from markets the model treated as settled but were actually coin flips.
  • In a binary market, overconfidence is structurally worse than underconfidence. You overpay, you lose more than you should, and the market was already right about the actual probability.
  • The fix was not a smarter model. NOAA publishes professionally calibrated forecasts for free. We replaced our overconfident ensemble with that.

The Number That Should Have Embarrassed Us Immediately

Our model scored 0.2858 on the Brier score. Lower is better. Simply guessing the historical base rate scored 0.2439.

Our custom 164-member ensemble, pulling from 4 forecast sources, cross-validated and tuned over months, performed worse than not having a model at all.

That is the headline. Everything else in this post is the explanation.

What Calibration Actually Means

Calibration is the relationship between what your model says and what actually happens.

A perfectly calibrated model means: when it says 70% probability, the event happens 70% of the time. When it says 90%, it happens 90% of the time. The predicted probabilities and the observed frequencies track each other.

A poorly calibrated model means that relationship is broken. The most common failure mode is overconfidence: the model says 90% when the real probability is 60%. It has convinced itself it knows more than it does.

You can visualize calibration with a reliability diagram. The x-axis is predicted probability, binned into buckets (0-10%, 10-20%, and so on up to 100%). The y-axis is actual accuracy within each bucket. A perfectly calibrated model produces a diagonal line from bottom-left to top-right.

Here is roughly what ours looked like:

Predicted Probability → Actual Accuracy
0-10%:   7%    (reasonable)
10-30%:  22%   (reasonable)
30-50%:  41%   (reasonable)
50-70%:  58%   (reasonable)
70-90%:  63%   (BAD — should be 80%)
90-100%: 61%   (TERRIBLE — should be 95%)

Everything below 70% confidence was roughly calibrated. The model fell apart exactly where it felt most certain.

Why the High-Confidence Bucket Is Where It Hurts

In a binary market, a miscalibrated high-confidence prediction is not just wrong. It is structurally destructive in a specific way.

When the model says 95% and we buy a "Yes" contract at 40 cents, we expect to win 95 cents of every dollar. If the actual win rate is 61%, we are paying for certainty we do not have. We are making a bet that assumes nearly no risk while carrying real risk. The position sizing and the risk tolerance are both set wrong.

Underconfidence is the opposite problem: the model says 55% when the real probability is 75%, so you underbuy or skip trades that actually had edge. That costs you opportunity.

Overconfidence costs you money on the bets you actually make. In a binary market with tight spreads and fixed payouts, overconfidence in the tail is lethal. You have no way to recover from paying 40 cents for a 61% contract over and over again.

The market, by the way, was not fooled. The Kalshi market priced the contracts at an average of 44.4 cents for the trades we were calling with 90%+ confidence. We thought we had found an edge. The market was telling us the coin was roughly fair.

Where the Loss Actually Came From

We have the trade database. Here is what the breakdown looked like.

Our 112 completed trades fell into roughly 3 confidence tiers based on what our model output at decision time:

Below 70% confidence (32 trades): Small positions, mixed results, mostly noise. Not the problem.

70-89% confidence (41 trades): Some losses here, but position sizing was moderate. The calibration gap existed but the damage was contained.

90%+ confidence (39 trades): This is where the bot destroyed value. The model was calling these coin flips at near-certainty. We sized them accordingly. The actual win rate in this bucket was around 61%. Nearly every dollar of the $23 net loss traces back to these 39 trades.

The pattern is consistent. The model got louder and more confident exactly in the cases where it had no additional information. The markets at the edge of the temperature range (is today's high going to be exactly 87 or above?) are inherently hard to call. Our ensemble responded to that difficulty by yelling louder instead of saying "I don't know."

The Ensemble Was Amplifying Noise

Here is the specific mechanism. The v2.1 ensemble used 164 forecast members across 4 sources:

  • GFS via Open-Meteo: 31 members
  • NOAA AIGEFS via AWS S3: 31 members
  • ECMWF IFS via Open-Meteo: 51 members
  • ECMWF AIFS-ENS via Open-Meteo: 51 members

The theory was that averaging across many independent sources would smooth out individual model errors and give us a sharper probability estimate. That is valid ensemble theory.

The problem: the sources were not independent in the ways that mattered. They share underlying physics, share training data, and share systematic biases. When all 164 members agreed, we treated that consensus as strong evidence. In reality, correlated models all making the same error looks exactly like confident independent agreement.

We measured the underdispersion: 4.2x. Our ensemble spread was 4.2 times tighter than the actual observed variability in outcomes. The model was systematically underestimating its own uncertainty.

Here is a simplified version of the probability scoring we were doing:

def ensemble_probability(member_forecasts: list[float], threshold: float) -> float:
    """
    Original approach: fraction of ensemble members above threshold.
    This is the correct formula but produces overconfident output
    when members are correlated.
    """
    votes_above = sum(1 for f in member_forecasts if f > threshold)
    return votes_above / len(member_forecasts)


# Example: 148 of 164 members say high temp > 85F
# Returns: 0.902 — feels like strong evidence
# Reality: actual outcomes in this bucket happened 61% of the time
# The members were correlated and all biased the same direction
raw_prob = ensemble_probability(member_forecasts, threshold=85.0)
print(f"Raw ensemble probability: {raw_prob:.3f}")  # 0.902

The fix we needed was either a calibration layer (Platt scaling, isotonic regression, temperature scaling) trained on a held-out set of outcomes, or we needed to use forecasts that were already calibrated. We chose the second option.

What NOAA Already Does That We Were Trying to Recreate

NOAA's National Blend of Models (NBM) is a publicly available, bias-corrected, calibrated probabilistic forecast product. It covers the exact weather stations Kalshi uses for contract settlement. It publishes uncertainty ranges, not point estimates. It is the output of decades of meteorological calibration work.

We were hand-rolling a worse version of this and calling it an edge.

The NBM does not just tell you "forecast high: 86F." It tells you the full probability distribution: the 10th percentile, 25th, 50th, 75th, 90th. That distribution has been calibrated against historical observations. When NBM says there is a 30% chance the high exceeds 88F, that is grounded in a calibration process we could not replicate with 4 months of data and a Python script.

Here is how we now pull the NBM data for a given station and date:

import requests
from datetime import datetime, timedelta


NBM_BASE_URL = "https://nomads.ncep.noaa.gov/pub/data/nccf/com/blend/prod"


def fetch_nbm_bulletin(station_id: str, target_date: datetime) -> dict:
    """
    Fetch NBM text bulletin for a single station.
    Returns parsed percentile data for daily max temperature.

    station_id: 4-letter ICAO code (e.g., 'KMDW' for Midway, not O'Hare)
    """
    # NBM issues hourly. We want the bulletin closest to 06Z for day-ahead planning.
    bulletin_hour = 6
    issue_date = target_date - timedelta(days=1)

    url = (
        f"{NBM_BASE_URL}/blend.{issue_date.strftime('%Y%m%d')}/"
        f"{bulletin_hour:02d}/text/blend_nbhtx.t{bulletin_hour:02d}z"
    )

    response = requests.get(url, timeout=30)
    response.raise_for_status()

    return parse_nbm_bulletin(response.text, station_id, target_date)


def parse_nbm_bulletin(bulletin_text: str, station_id: str, target_date: datetime) -> dict:
    """
    Extract temperature percentiles from NBM text bulletin.
    Returns dict with keys: p10, p25, p50, p75, p90 (all in Fahrenheit).
    """
    # Bulletin format is fixed-width text blocks, one per station
    # Station block starts with 4-letter ICAO code
    lines = bulletin_text.split('\n')
    station_block = extract_station_block(lines, station_id)

    if not station_block:
        raise ValueError(f"Station {station_id} not found in NBM bulletin")

    return parse_temperature_percentiles(station_block, target_date)

The key difference: p90 from NBM is a calibrated 90th percentile. When we compute the probability that the daily high exceeds a threshold, we are reading from a distribution that has been tuned against real outcomes. We are not aggregating correlated ensemble members and mistaking consensus for certainty.

Why Overconfidence Is Specifically Bad in Binary Markets

In a continuous market (stocks, futures), overconfidence mostly means you size positions wrong and your risk management is off. You can be overconfident about a stock and still make money if you are directionally correct.

Binary markets have no middle ground. Either the contract settles at $1 or at $0. If you buy at 40 cents because you think the probability is 95%, and the real probability is 61%, there is no partial credit. You lose the full 40 cents when it goes wrong, more often than your model told you it would.

The math is blunt:

def expected_value(buy_price: float, true_probability: float) -> float:
    """
    Expected value of buying a binary contract at buy_price.
    Payout is $1 on win, $0 on loss.
    """
    win_payout = 1.0 - buy_price
    loss_cost = buy_price
    return (true_probability * win_payout) - ((1 - true_probability) * loss_cost)


# What our model thought it was doing
model_ev = expected_value(buy_price=0.40, true_probability=0.95)
print(f"Model EV: {model_ev:.3f}")   # +0.530

# What was actually happening
actual_ev = expected_value(buy_price=0.40, true_probability=0.61)
print(f"Actual EV: {actual_ev:.3f}")  # +0.210

# Still positive, but much thinner than assumed.
# And this assumes Kalshi's pricing was not already incorporating the correct probability.
# It was. The market was pricing these at 44.4 cents, not 40.

market_actual_ev = expected_value(buy_price=0.444, true_probability=0.452)
print(f"Market EV on 'No' side: {market_actual_ev:.3f}")  # roughly 0.0

The market was right. We were overconfident. In that combination, there is no edge to extract.

The Defect That Made It Worse

None of the calibration analysis above would have saved us from a separate bug that compounded the damage: the DST date-bucketing error.

The National Weather Service defines a weather day in Local Standard Time year-round. In summer, that means the official 24-hour window for temperature records runs from 1:00 AM to 12:59 AM the following day, not midnight to midnight. For eight months of the year, we were bucketing the wrong 24 hours when computing whether a threshold was exceeded.

We ran this system through summer. The DST window error was active the entire time.

So not only was our probability calibration broken, we were sometimes comparing our forecast to the wrong day's settlement value entirely. Both problems pointed the same direction: the model thought it was right; the outcomes said otherwise.

The Rebuild Was Not a Smarter Model

The v2.3 rebuild did not introduce a more sophisticated ensemble. It did not add more data sources or more members. It replaced the custom ensemble with NOAA's NBM entirely and fixed the 7 defects found during the audit.

That is the lesson. We were measuring the wrong thing (our model's confidence) instead of the right thing (observed calibration against outcomes). When we finally measured the right thing, the answer was obvious: stop hand-rolling a calibration layer we do not have enough data to train, and use the one NOAA spent decades building.

The rebuilt bot is in paper-trading mode and has not been validated. The correct status is "rebuilt and undergoing validation." Forecast quality takes about 2 weeks of daily verification to assess. Whether the strategy has any edge takes 4 to 6 months and 100+ completed trades. We are not there yet.

The Practical Takeaway

If your model produces high-confidence predictions and you have outcomes data, run a reliability diagram before you trust a single number it outputs. Bucket predictions by confidence. Count wins in each bucket. Plot it. If the 90% bucket is winning 60% of the time, your model is not giving you information, it is giving you false precision dressed up as certainty.

Calibration is not a nice-to-have. In a market where the other side has access to professional forecasting tools, miscalibration is the specific mechanism through which you lose money systematically while your dashboard shows everything is fine.

We caught it by running the audit. That was 4 months and $23 of tuition. Cheaper than I expected, honestly.