< Back to Blog

We Turned Down Google's AI Weather Model

TL;DR / Key Takeaways

  • Google's WeatherNext 2 is genuinely impressive. Free, drop-in compatible, beats ECMWF on 97% of benchmarks.
  • We measured a 5.2°F cold bias on daily high temperature across our test stations. That is the exact number Kalshi contracts settle on.
  • Benchmark accuracy and contract-settlement accuracy are two different things. The industry conflates them constantly.
  • WeatherNext's spread calibration was actually better than our current model. That part stays on the roadmap. We're not anti-AI. We're anti-assumption.

The Pitch Was Perfect

Google released WeatherNext 2 earlier this year. The benchmarks are real: 97% of test cases beat ECMWF, which has been the gold standard in numerical weather prediction for decades. It runs on a transformer architecture trained on 40 years of ERA5 reanalysis data. It is free. The API is clean. The documentation is good.

Every data engineering instinct I have said: adopt this immediately.

I spent one afternoon testing it instead.

We declined to ship it.

What We Actually Measure

Before I explain the rejection, I need to explain what we care about.

Kalshi temperature contracts settle on one number: the official daily high temperature at a specific airport, measured in whole degrees Fahrenheit, reported by NOAA. That's it. Not the mean temperature. Not the overnight low. Not a 24-hour average. The daily high.

Chicago contracts settle on Midway. Houston on Hobby. Not O'Hare, not Bush Intercontinental. We learned that the hard way and verified every station against Kalshi's own metadata API. But that's a different post.

The point is: our entire edge, if we have one, lives or dies on the accuracy of one number, at one station, for one 24-hour window. A model that is excellent at predicting 500hPa geopotential height anomalies over the North Atlantic is completely irrelevant to us if it is biased on daily surface maximum temperature in the continental US.

Most weather model benchmarks do not test what we need.

The Test

I pulled WeatherNext 2 forecasts for 16 Kalshi settlement stations over a two-week period and compared them against the NOAA observed highs. Same stations, same dates, same observation source we use for verification.

Then I compared against the models we already have in the pipeline: GFS, ECMWF IFS, ICON, and GEM.

Here is what I found on daily high temperature, measured as mean forecast error (model minus observed, in °F):

| Model | Mean Error (°F) | Notes | |---|---|---| | GFS | +0.3 | Slight warm bias | | ECMWF IFS | -0.2 | Near-neutral | | ICON | -0.4 | Slight cold bias | | GEM | -0.1 | Near-neutral | | WeatherNext 2 | -5.2 | Cold bias on daily highs |

Not -0.5°F. Not -1.2°F. Minus 5.2 degrees Fahrenheit, cold, on the exact temperature type our contracts settle on.

For context: Kalshi temperature contracts typically span 2-3 degree ranges. A 5.2°F cold bias doesn't introduce noise into our signal. It systematically points to the wrong contract on almost every trade.

Why This Happens

WeatherNext 2 is trained and benchmarked primarily on 500hPa geopotential height, wind fields, and mean 2-meter temperature. These are the traditional NWP verification metrics. They are meaningful for general forecast skill.

Daily maximum 2-meter temperature is a different animal. It depends on the planetary boundary layer, local surface energy balance, urban heat island effects, and the specific observing conditions at individual ASOS stations. A model that is extraordinary at synoptic-scale pattern recognition can still be systematically biased on the specific surface variable that NOAA uses to settle contracts.

WeatherNext was built for global forecast skill. We need local daily-high accuracy.

Those are different problems.

The Code

Here is the test harness I used. Nothing fancy. If you're pulling WeatherNext 2 outputs via their API, you're getting 0.25-degree gridded data. You need to extract the nearest grid point to your station lat/lon and pull the TMAX field.

import requests
import numpy as np
import pandas as pd
from datetime import datetime, timedelta

STATION_COORDS = {
    "KMDW": (41.786, -87.752),   # Chicago Midway
    "KHOU": (29.645, -95.279),   # Houston Hobby
    "KBWI": (39.175, -76.668),   # Baltimore
    "KDFW": (32.898, -97.038),   # Dallas Fort Worth
    # ... full list in stations.py
}

def get_weathernext_tmax(station_id: str, target_date: datetime) -> float | None:
    """
    Pull daily high temperature forecast from WeatherNext 2 gridded output.
    Returns degrees Fahrenheit or None if unavailable.
    """
    lat, lon = STATION_COORDS[station_id]

    # WeatherNext serves 6-hourly fields; daily max requires aggregation
    # over the NOAA LST window: 0700 LST to 0659 LST next day
    # DST offset matters here. Get it wrong and you're bucketing the wrong day.
    utc_offset = get_utc_offset_for_station(station_id, target_date)
    window_start_utc = target_date.replace(hour=7) - timedelta(hours=utc_offset)
    window_end_utc = window_start_utc + timedelta(hours=24)

    params = {
        "latitude": lat,
        "longitude": lon,
        "start": window_start_utc.isoformat(),
        "end": window_end_utc.isoformat(),
        "variables": "temperature_2m",
    }

    resp = requests.get(
        "https://api.weathernext.google.com/v1/forecast/point",
        params=params,
        timeout=30,
    )

    if resp.status_code != 200:
        return None

    data = resp.json()
    temps_c = [entry["temperature_2m"] for entry in data["hourly"]]

    if not temps_c:
        return None

    tmax_c = max(temps_c)
    tmax_f = (tmax_c * 9 / 5) + 32
    return round(tmax_f, 1)


def compute_bias(forecasts: list[float], observations: list[float]) -> float:
    """Mean error: positive = warm bias, negative = cold bias."""
    errors = [f - o for f, o in zip(forecasts, observations)]
    return round(np.mean(errors), 2)

The DST window comment on line 15 is not optional boilerplate. That is the bug that invalidated eight months of our v2.1 trades. NOAA's official weather day for temperature records runs from 0700 Local Standard Time to 0659 LST the next morning, year-round. In summer, that means your "Tuesday high" observation actually includes early Wednesday morning UTC. Bucket the wrong 24 hours and you're comparing against the wrong observation. I wrote about this in the v2.1 post-mortem. It bit us before. It will bite you too if you're not deliberate about it.

What the Benchmark Numbers Don't Tell You

The 97% figure is from WeatherNext's own paper, comparing against ECMWF IFS ensemble mean on synoptic-scale variables over held-out periods. That is a legitimate result. I am not disputing it.

But "97% of benchmarks" is a portfolio metric. It tells you how the model performs on average across a wide range of verification targets. It does not tell you that the model is unbiased on the specific variable at the specific locations for the specific time window that determines whether your contract pays.

This is the failure mode I see constantly in data engineering. Someone runs a model against a standard benchmark suite, gets good numbers, and ships it. The benchmark was never designed to answer the operational question you actually have.

In corporate life, this is how you end up with a fraud detection model that scores 98.7% AUC in validation and misses every novel fraud pattern that shows up six months after deployment. The benchmark was real. The question it answered was not the one you needed answered.

We ran the benchmark that matters to us: mean error on daily high temperature at our settlement stations. WeatherNext failed that benchmark by a margin that makes it unusable for our application.

The Part We Are Keeping

Here is where I want to be precise, because "we rejected it" is not the whole story.

WeatherNext 2's probabilistic spread calibration was better than our current implementation. When I looked at the distribution of its ensemble forecasts versus observed outcomes, the spread was tighter and better-calibrated than the NOAA AIGEFS ensemble we use as a secondary source.

That matters. A well-calibrated spread means the model's uncertainty estimates are trustworthy. When it says it's 80% confident, it's right about 80% of the time. When it says it's uncertain, the outcome is genuinely uncertain. That is exactly what you want for a probability-scoring system.

Our current model has a documented 4.2x underdispersion problem. The ensemble is too confident. We corrected for this in the v2.3 rebuild, but we did it by applying a post-hoc correction factor rather than by having a better-calibrated source.

So the decision is not: WeatherNext is bad. The decision is: WeatherNext has a 5.2°F cold bias on the settlement variable, which disqualifies it for direct use, and its spread calibration is genuinely better than what we have, which is worth returning to once the bias problem is characterized and addressed.

Maybe Google fixes the daily-high bias in the next version. Maybe there's a station-level correction we can apply. When we have evidence that the bias is resolved, we will retest.

We are not anti-AI. We are anti-assumption.

Why the Industry Ships It Anyway

I want to say something direct here.

Most teams that would evaluate WeatherNext 2 for a weather-adjacent application would look at the 97% benchmark figure, check that the API is well-documented and free, and ship it. That is not incompetence. That is the rational behavior when you have deadlines and management asking why you're not using the shiny new thing from Google.

The measurement I ran took one afternoon. It is not difficult work. It is just work that requires a clear question: "Biased in what direction, on what variable, at which locations?"

That question only gets asked if someone decides it has to be asked. In most organizations, that decision never gets made because nobody owns it. The model gets shipped. The bias quietly drains money or corrupts decisions for months. Eventually someone notices the outputs are systematically off and goes looking, and by then the model is load-bearing.

We noticed with our v2.1 ensemble after 112 trades and a $23 loss. That is a cheap lesson. I would like to not repeat it.

What This Means for the Roadmap

Current status: Weather Bot v2.3 uses NOAA's National Blend of Models as its primary forecast source. NBM is already calibrated, already bias-corrected, already covers exactly the stations Kalshi uses for settlement. It is free. We were hand-rolling a worse version of it for eight months with v2.1. That was the actual lesson of the post-mortem.

The bot is in paper-trading mode. It is rebuilt and undergoing validation. I am not going to tell you it works until I have the numbers that say it works.

WeatherNext 2 goes on the candidate list for future evaluation, specifically for its ensemble spread properties. Not as a replacement for NBM. As a potential additional signal once the bias profile is better understood.

The 165 automated tests in the safety suite exist precisely so that when we do try to integrate a new model, the tests catch what the benchmark missed.

The Practical Takeaway

If you are building any system that settles on a specific variable at specific locations, run your own benchmark on that variable at those locations before you ship anything.

Not the model's published benchmark. Yours.

bias = mean(forecast - observed)

That is the whole formula. One afternoon. The industry does not do this because it is boring and the results are sometimes inconvenient.

The results are sometimes inconvenient.

That is the point.

We declined WeatherNext 2 not because it is a bad model. It is a good model for the problems it was built to solve. We declined it because we measured it against the problem we actually have, and the number came back wrong. That is the only way this works.

Rigor is the edge. Everything else is just noise with better branding.