Rebuilt and Undergoing Validation: Where the Weather Bot Stands in July 2026
TL;DR / Key Takeaways
- The v2.1 Weather Bot ran for four months, lost $23, and scored worse on Brier score than guessing the base rate. We audited it and rebuilt it.
- v2.5 replaces the custom ensemble with NOAA's National Blend of Models (NBM), fixes 7 distinct defects, and adds 49 automated tests.
- The bot is currently in paper-trading mode. It has not been validated. "Rebuilt" is not the same as "working."
- Forecast quality can be assessed in roughly two weeks. Whether the strategy makes money needs 100+ completed trades, which is four to six months at current trading rates.
I want to be clear about something before you read further.
The Weather Bot is not fixed. It is rebuilt. Those are different things, and conflating them is exactly how people lose money.
"Fixed" implies the problem is solved and the outcome is known. "Rebuilt" means we identified the failure mode, replaced the broken component, and now we have to prove the new one actually works. We are in the second category. The bot is in paper-trading mode. It has not executed a live trade since the rebuild. I have no idea if the new design will outperform the market. I have reasons to believe it will do better than what we had, but reasons are not data.
This post is a status update. What the bot looks like now, how we are validating it, and what we will know and when.
Why We Rebuilt Instead of Tweaked
Four months, 112 completed trades, a net loss of roughly $23. That part has been covered in the post-mortem. But the number that actually made me rebuild from scratch was the Brier score.
Our custom ensemble scored 0.2858. Simply guessing the historical base rate for each market scored 0.2439. Lower is better. Our model was statistically worse than making no prediction at all.
That is not a calibration problem. That is not a parameter problem. That is a "the model has no signal" problem. You do not tune your way out of that. You stop and ask what you were actually measuring.
What we were measuring turned out to be noise. The ensemble was confident (near-certainties of 95%+) and wrong at a rate that matched a coin flip in the markets where it most mattered. We were packaging our own uncertainty as conviction and then trading on it.
The fix was not a cleverer model. NOAA's National Blend of Models already does what we were trying to hand-roll, and it does it better, and it is free. Four separate agencies and hundreds of observation stations feed into the NBM. It is bias-corrected. It is calibrated against observed outcomes. It covers exactly the weather stations Kalshi uses for settlement.
We were building a worse version of a public good.
What v2.5 Actually Changed
The rebuild touched four areas: the forecast source, the date-bucketing logic, the station mapping, and the test suite.
Forecast source. The custom ensemble is gone. NBM is the primary source. NOAA AIGEFS via AWS S3 is retained as a secondary confirmation signal. Both are free public feeds.
# NBM point forecast fetch (simplified)
import requests
def fetch_nbm_point(station_id: str, forecast_date: str) -> dict:
"""
Fetch NBM point forecast for a given station and date.
Returns calibrated high/low temperature with uncertainty range.
"""
url = (
f"https://forecast.weather.gov/MapClick.php"
f"?CityName={station_id}&state=&site=&textField1=&textField2="
f"&FcstType=digital&unit=0"
)
# Production implementation uses the NBM bulletin parser
# against the 4,228-station calibrated product, not the web interface.
# This is illustrative of the data flow.
response = requests.get(url, timeout=10)
response.raise_for_status()
return parse_nbm_bulletin(response.text, forecast_date)
The actual production fetch pulls from the NBM text bulletins, not the web interface. The bulletin covers 4,228 stations. We parse the relevant station rows, extract the calibrated high temperature forecast and the uncertainty spread, and feed that into the probability estimate.
Date-bucketing logic. This was the defect I am most embarrassed about. The National Weather Service defines a weather "day" in Local Standard Time year-round. During daylight saving time, the official weather day runs from 1:00 AM to 12:59 AM the next calendar day, not midnight to midnight. Our code was using wall-clock midnight, which was wrong for eight months of the year.
Every losing trade fell in that DST window. Not some of them. All of them.
from datetime import datetime, timedelta
import pytz
def get_nws_weather_day_bounds(date: datetime.date, station_tz: str) -> tuple:
"""
Returns (start_utc, end_utc) for the NWS weather day.
NWS defines the weather day in Local Standard Time year-round.
During DST, the weather day runs 1:00 AM local to 12:59 AM local next day.
"""
tz = pytz.timezone(station_tz)
std_offset = get_standard_offset(tz) # LST offset, not current offset
# Weather day always starts at midnight LST
day_start_lst = datetime(date.year, date.month, date.day, 0, 0, 0)
day_start_utc = day_start_lst + timedelta(hours=-std_offset)
day_end_utc = day_start_utc + timedelta(hours=24)
return day_start_utc, day_end_utc
This is fixed in v2.5. Every station has its standard offset pinned in the config, and the bucketing uses LST regardless of the current DST state.
Station mapping. Chicago contracts settle on Midway, not O'Hare. Houston on Hobby, not Bush Intercontinental. We confirmed the correct stations for every active market by querying Kalshi's own metadata API and pinning the results in the station config. Not guessing. Verified.
# Station config excerpt (pinned from Kalshi metadata API)
SETTLEMENT_STATIONS = {
"CHICAGO": {
"kalshi_ticker_prefix": "HIGHNY",
"icao": "KMDW", # Midway, verified via Kalshi metadata
"name": "Chicago Midway International",
"tz": "America/Chicago",
"lst_offset_hours": -6
},
"HOUSTON": {
"kalshi_ticker_prefix": "HIGHHOU",
"icao": "KHOU", # Hobby, verified via Kalshi metadata
"name": "Houston William P. Hobby",
"tz": "America/Chicago",
"lst_offset_hours": -6
},
# ... all active markets verified and pinned
}
Tests. 49 new automated tests were added during the rebuild, covering the DST bucketing logic, station mapping, NBM parse correctness, and the probability scoring pipeline. The regression suite now catches the class of defects that were invisible for four months.
What Validation Actually Looks Like
There are two separate questions here, and they have different timelines. People tend to collapse them into one, which is where the false confidence comes from.
Question 1: Are the forecasts accurate?
This one is answerable in roughly two weeks. Every day, the bot generates a forecast for each active market. Every day, the actual high temperature is recorded. We compare. We track Brier score on a rolling basis. We check for systematic bias (consistently too high, too low, overconfident). Two weeks gives us enough data points to see whether the NBM-derived probabilities are calibrated against actual outcomes.
# Daily verification log structure
CREATE TABLE forecast_verification (
id SERIAL PRIMARY KEY,
market_ticker TEXT NOT NULL,
station_id TEXT NOT NULL,
forecast_date DATE NOT NULL,
predicted_prob NUMERIC(6,4), -- P(high temp > threshold)
threshold_temp NUMERIC(5,1),
actual_high NUMERIC(5,1),
outcome BOOLEAN, -- did high exceed threshold?
brier_contrib NUMERIC(8,6), -- (predicted_prob - outcome)^2
recorded_at TIMESTAMPTZ DEFAULT NOW()
);
This table populates every day. We run the Brier score calculation weekly and check it against the base-rate benchmark. If we are trending worse than the base rate again, we stop and figure out why before going live.
Question 2: Does the strategy make money?
This one takes longer. Much longer. Four to six months at current trading rates, which puts us at roughly 100+ completed trades before we have a sample size worth analyzing. That is the minimum for the numbers to mean anything. Less than that and you are reading noise.
Paper trading gives us the forecast accuracy data. It does not tell us whether the edge we think we see against Kalshi's pricing actually translates to positive expected value after fees, slippage, and the natural variance of individual contracts.
Those are different questions. Treating them as the same question is how people convince themselves a bot is working when it is not.
The Metrics We Are Tracking
During paper trading, we log every decision the bot would have made: the market, the forecast probability, the Kalshi market price, the implied edge, whether we would have entered, and the eventual outcome.
# Paper trade decision log (every cycle, every candidate market)
CREATE TABLE paper_trade_decisions (
id SERIAL PRIMARY KEY,
market_ticker TEXT NOT NULL,
scan_timestamp TIMESTAMPTZ NOT NULL,
nbm_prob NUMERIC(6,4), -- our forecast probability
kalshi_yes_price NUMERIC(6,4), -- market price at scan time
implied_edge NUMERIC(6,4), -- nbm_prob - kalshi_yes_price
would_trade BOOLEAN,
skip_reason TEXT, -- logged even if no trade
outcome BOOLEAN, -- filled after settlement
paper_pnl NUMERIC(8,2) -- hypothetical P&L
);
The skip_reason column is not optional. One of the defects in v2.1 was that we only logged trades that fired. Skipped markets left no trace. If the bot had a systematic bias in what it chose to trade, we would never see it. Now every rejected candidate is recorded with the reason.
The metrics we care about in validation order:
- Brier score vs. base rate. This tells us if our forecasts have skill. It has to clear this bar before anything else matters.
- Calibration plot. Are our 60% confidence predictions winning 60% of the time? Are our 70% predictions winning 70% of the time? Overconfidence kills you.
- Edge distribution. What does the spread between our probability and Kalshi's price look like? If we see edge everywhere, we are probably wrong. If we see it occasionally in specific conditions, that is more credible.
- Paper P&L. Last in the list deliberately. Paper P&L is the most visible metric and the least informative early on. Variance dominates in small samples.
What We Will Not Know Until We Go Live
Paper trading has a specific blind spot: it does not simulate market impact or fill quality. On Kalshi, most weather markets are liquid enough that limit orders fill at the quoted price. But there are markets at the edges of the session where the spread widens and fills are less predictable.
We also cannot test the risk management logic fully in paper mode. The kill switches, the position limits, the stop conditions. Those paths only execute when real money is at stake and real things go wrong. We test them in simulation and in unit tests, but the first time they fire in production is always revealing.
This is not a reason to avoid paper trading. It is a reason to be honest about what paper trading validates and what it does not.
The Timeline
Here is where things stand as of August 2026:
- Now through early September: Paper trading. Daily forecast verification logging. Weekly Brier score review.
- Early September: First calibration assessment. If the forecasts look reasonable, we move toward limited live trading. If not, we stop and diagnose.
- September through early 2027: Live trading at reduced position sizes. Building toward 100 completed trades for a statistically meaningful sample.
- Early 2027 (roughly): First honest assessment of whether the strategy has edge against Kalshi pricing. This is the number that actually matters.
That timeline is not exciting. It is also not going to change because I want faster results. The validation process takes as long as it takes.
The Antidote to "I Built a Bot and Now I'm Rich"
There is a specific genre of content online where someone builds an automated trading system, runs it for six weeks, makes a few hundred dollars, and declares victory. The comments fill with people asking for the source code.
I have read enough of those posts to notice what they never include: a Brier score, a calibration plot, a description of the losing trades, or any acknowledgment that six weeks is not a sample size.
The v2.1 bot ran for four months. Every week the dashboard was green. Positions were opening, trades were logging, the system looked like it was working. It lost money and had no predictive skill and I did not know either of those things until I went looking.
That is not a special kind of failure. That is what happens when you build a system that is very good at looking like it is working.
The rebuilt bot has the same risk. The validation methodology exists specifically to catch the difference between "looks like it is working" and "is actually working." Two weeks of Brier score data will tell us the former. Four to six months of live trading will tell us the latter.
Until then: rebuilt, in paper trading, undergoing validation. That is the whole status.
The code is available as part of the $75 source package at predictandprofit.io. You get both bots, the full validation logging infrastructure, and the post-mortem documentation that explains every defect we found. If you want to run your own validation rather than wait for mine, the tools are in there to do it.
Prediction markets are speculative. You can lose everything you put in. Build accordingly.