TL;DR / Key Takeaways

Our custom 164-member ensemble scored 0.2858 on Brier score. Guessing the base rate scores 0.2439. We were statistically worse than making no prediction at all.
NOAA's National Blend of Models (NBM) is already professionally calibrated, bias-corrected, and published daily for exactly the weather stations Kalshi settles on.
We were hand-rolling an inferior version of a public good we didn't know existed.
The lesson is not specific to weather: before you build a forecasting model, check if a federal agency already maintains a better one and gives it away for free.

The Uncomfortable Number

After four months and 112 completed trades, the Weather Bot had lost roughly $23. That number alone is not the story. $23 is noise. The story is what the audit found underneath it.

I pulled every trade from the database and ran a Brier score against actual outcomes. The Brier score is a proper scoring rule for probabilistic forecasts. Lower is better. A perfect forecaster scores 0. Randomly guessing the historical base rate for each market scores 0.2439 for our trade sample.

Our custom ensemble scored 0.2858.

We were not just failing to add value. We were destroying it. A bot that ignored our model entirely and just guessed the base rate would have made better predictions than the bot we spent months building. That is the kind of number that ends a conversation about tweaking hyperparameters.

The fix was not going to be a cleverer ensemble. Something more fundamental was wrong.

What the Old Architecture Looked Like

The v2.1 Weather Bot ran a 164-member ensemble built from 4 sources:

GFS via Open-Meteo (31 members)
NOAA AIGEFS via AWS S3 (31 members)
ECMWF IFS via Open-Meteo (51 members)
ECMWF AIFS-ENS via Open-Meteo (51 members)

The idea was reasonable on paper. Aggregate multiple independent forecasting models, weight by historical skill, and produce a calibrated probability distribution over the next day's high temperature. Then compare that distribution against Kalshi market pricing and trade when we saw an edge.

The problem was the phrase "calibrated probability distribution." We wrote that code ourselves. And we were not very good at it.

The ensemble produced extreme probabilities constantly. Near-certainties of 95%+ that turned out to be right about 60% of the time. That is called overconfidence, and in a proper scoring rule it is punished hard. Every time the model screamed 95% and the coin came up tails, we absorbed a full loss.

# v2.1 probability output — this is what bad calibration looks like
{
    "station": "KMDW",
    "date": "2026-04-15",
    "high_temp_forecast": 71.3,
    "prob_above_70": 0.94,   # model is very sure
    "ensemble_members": 164,
    "calibration": "homemade"  # here is the problem
}

Homemade calibration. That phrase should have been a warning sign from the start.

The Sunk Cost Problem

By the time I ran the Brier score audit, I had been building that ensemble for months. The multi-source architecture was genuinely interesting engineering. Pulling GRIB2 files from S3 using byte-range requests on the index file, parsing ensemble member spreads, writing the weighting logic. I was proud of it.

That pride is exactly the problem.

When you have invested months in a system, there is a gravitational pull toward "let's tune it more" rather than "let's ask whether the whole approach is wrong." I had to force myself to look at the Brier score and sit with what it meant instead of immediately opening a ticket to fix the calibration weights.

What it meant was: the approach was wrong. Not the implementation. The approach.

We were trying to build a probabilistic temperature forecasting system good enough to find edges against a liquid prediction market full of people who also know what they're doing. And we were trying to do it with homemade calibration on top of raw ensemble output.

NOAA has been doing this professionally for decades.

What NOAA's NBM Actually Is

The National Blend of Models is NOAA's operational consensus forecast. It ingests dozens of deterministic and ensemble models, applies bias correction derived from years of observed station data, and produces calibrated probabilistic forecasts at specific weather stations across the US.

Critically: it publishes those forecasts for exactly the weather stations Kalshi uses for settlement.

It is updated multiple times per day. It is free. It is available via a public HTTPS endpoint. The data is in GRIB2 format with a companion index file that lets you do byte-range fetches for specific variables without downloading the entire bulletin.

The bulletin covers 4,228 stations. Kalshi's temperature markets settle on a small subset of major airport stations. Every single one of them is in the NBM bulletin.

I found this out about three weeks into the post-mortem. My first reaction was something like exhausted relief. We had been hand-rolling a worse version of this for months.

The Integration

Pulling NBM data is not complicated once you understand the file structure. NOAA publishes a .idx index file alongside each GRIB2 bulletin. The index file maps each variable and level to a byte range in the main file. You fetch the index, find the byte range for your variable, and do a ranged HTTP GET for just those bytes.

import requests
import re

NBM_BASE_URL = "https://nomads.ncep.noaa.gov/pub/data/nccf/com/blend/prod"

def fetch_nbm_index(cycle_date: str, cycle_hour: str) -> str:
    """
    Fetch the .idx file for a given NBM cycle.
    cycle_date: "20260814"
    cycle_hour: "00", "06", "12", or "18"
    """
    url = (
        f"{NBM_BASE_URL}/blend.{cycle_date}/{cycle_hour}/core/"
        f"blend.t{cycle_hour}z.core.f024.co.idx"
    )
    response = requests.get(url, timeout=30)
    response.raise_for_status()
    return response.text


def get_byte_range(index_text: str, variable: str, level: str) -> tuple[int, int]:
    """
    Parse the .idx file and return the byte range for a specific variable.
    Returns (start_byte, end_byte). end_byte of -1 means read to EOF.
    """
    lines = index_text.strip().split("\n")
    for i, line in enumerate(lines):
        parts = line.split(":")
        if len(parts) < 5:
            continue
        var_name = parts[3]
        level_name = parts[4]
        if var_name == variable and level_name == level:
            start_byte = int(parts[1])
            if i + 1 < len(lines):
                next_parts = lines[i + 1].split(":")
                end_byte = int(next_parts[1]) - 1
            else:
                end_byte = -1
            return start_byte, end_byte
    raise ValueError(f"Variable {variable}:{level} not found in NBM index")

The variable we care about for daily high temperature is TMAX at 2 m above ground for the 24-hour forecast window. NBM publishes both the point forecast and quantile forecasts, so you can reconstruct a probability distribution without doing your own calibration.

def fetch_nbm_variable(
    cycle_date: str,
    cycle_hour: str,
    variable: str,
    level: str
) -> bytes:
    """
    Fetch the raw GRIB2 bytes for a specific variable from the NBM bulletin.
    """
    index_text = fetch_nbm_index(cycle_date, cycle_hour)
    start_byte, end_byte = get_byte_range(index_text, variable, level)

    url = (
        f"{NBM_BASE_URL}/blend.{cycle_date}/{cycle_hour}/core/"
        f"blend.t{cycle_hour}z.core.f024.co.grib2"
    )
    headers = {}
    if end_byte == -1:
        headers["Range"] = f"bytes={start_byte}-"
    else:
        headers["Range"] = f"bytes={start_byte}-{end_byte}"

    response = requests.get(url, headers=headers, timeout=60)
    response.raise_for_status()
    return response.content

Parsing the GRIB2 bytes requires cfgrib or pygrib. We use cfgrib because it integrates cleanly with xarray, which is how the rest of the pipeline handles gridded data.

The station mapping step was separately verified. We pulled every Kalshi temperature market from the exchange's metadata API and confirmed the settlement station for each one. Chicago settles on Midway (KMDW), not O'Hare. Houston settles on Hobby (KHOU), not IAH. These are documented elsewhere in the codebase, but the NBM integration required us to verify that every settlement station appeared in the bulletin. They all do.

Before vs. After

Here is the honest comparison of the two architectures:

v2.1 (retired)

| Component | Detail | |---|---| | Forecast sources | Open-Meteo GFS, NOAA AIGEFS, ECMWF IFS, ECMWF AIFS-ENS | | Ensemble members | 164 | | Calibration | Custom, homemade | | Bias correction | None | | Output | Overconfident probabilities (95%+ on coin-flip markets) | | Brier score | 0.2858 (worse than base rate) |

v2.5 (current, paper trading)

| Component | Detail | |---|---| | Forecast sources | NOAA NBM (primary), NOAA AIGEFS (secondary confirmation) | | Calibration | NOAA's operational calibration, derived from decades of observed data | | Bias correction | Built into NBM | | Output | Quantile forecasts from which we derive probabilities | | Brier score | Not yet measured — undergoing validation |

The architecture got simpler, not more complex. That is usually a sign you found the right abstraction.

What We Stopped Building

The ensemble weighting code is gone. About 800 lines of Python that computed running skill scores for each model source, adjusted weights dynamically, and produced a combined probability estimate. It was the most technically interesting part of the old system.

It was also the part that was producing Brier 0.2858.

The probability calibration module is gone too. The piece of code that was supposed to take raw ensemble probabilities and map them to calibrated outputs. We wrote it. It did not work. NOAA has been solving this problem with actual meteorologists and actual historical station data for decades. Our weekend version was not going to win.

What remains is cleaner: fetch the NBM bulletin, extract the relevant variable for the relevant station, derive a probability that the 24-hour high exceeds the Kalshi strike, compare against market pricing. The hard forecasting work is done upstream by professionals with better data and better tools.

The Actual Lesson

Check what already exists before you build.

That sounds obvious. Every senior engineer has heard it a hundred times. I still violated it because I was excited about the problem and I jumped to implementation before I understood the landscape of existing solutions.

The NBM is not obscure. It is NOAA's flagship operational consensus product. It powers commercial weather apps. It is the output of a multi-decade investment in numerical weather prediction. And it is free, because it was built with public money for public use.

I did not know it existed at the right level of detail until the post-mortem forced me to ask the question: "What does a professional forecaster actually use for exactly this problem?"

The answer was: the thing NOAA publishes every 6 hours to a public HTTPS endpoint.

There is a version of this lesson that applies everywhere. Before you build a custom NLP classifier, check whether a fine-tuned public model already solves your problem. Before you build a custom time-series anomaly detector, check whether the statistics literature has a standard method that outperforms ad hoc approaches. Before you build a probability calibration system for ensemble weather forecasts, check whether NOAA already does this and gives it away.

Most of the time, the professional solution exists. It is just not where you were looking.

Where We Are Now

The Weather Bot is rebuilt and undergoing validation. It is in paper-trading mode. We have not measured its Brier score against live outcomes yet because we have not accumulated enough samples. That takes time. The bot scans 291 weather markets per cycle and makes conservative trade decisions, which means completed trades accumulate slowly.

The validation plan is straightforward. Run the forecasts daily, record outcomes, compute Brier score every two weeks. If the forecast quality looks reasonable after a month, we start looking at whether the strategy has edge against market pricing. That assessment needs 100+ completed trades, which is four to six months at current rates.

We are not going to rush that timeline. The v2.1 post-mortem happened because we ran the bot live for four months without rigorous ongoing measurement. We are not doing that again.

The code is available if you want to look at the NBM integration yourself. The pattern generalizes to any application that needs calibrated probabilistic temperature forecasts at specific US weather stations. NOAA built something genuinely useful and made it free. Use it.

The sunk cost fallacy is real and it is expensive. I spent months on an ensemble that a Brier score audit killed in an afternoon. The right response to that is not embarrassment. It is to publish the number, fix the architecture, and not make the same mistake twice.

We published the number. 0.2858. Worse than guessing. Now it is fixed.

Why We Replaced Our Custom Ensemble with NOAA's National Blend of Models