We Audited 112 Trades and Found Our Model Had No Skill
TL;DR / Key Takeaways
- Our Weather Bot ran for 4 months, completed 112 trades, and lost roughly $23
- The Brier score audit revealed the model scored 0.2858 vs 0.2439 for simply guessing the base rate — statistically worse than making no prediction at all
- The Kalshi market was efficient; our ensemble model added nothing on top of what the market already knew
- The fix was not a cleverer model — it was switching to NOAA's National Blend of Models, which already publishes professionally calibrated forecasts for free
The Numbers First
Four months. 112 completed trades. Net loss of roughly $23.
That's not a disaster. But it's not the point either. The $23 is almost irrelevant. What matters is what I found when I stopped looking at the balance and started looking at the model.
The model scored a Brier score of 0.2858.
Simply guessing the historical base rate — no model, no ensemble, no code at all — scores 0.2439.
Our model was statistically worse than doing nothing.
That's the finding. Everything else in this post is context.
What Is a Brier Score and Why Does It Matter
If you've never used Brier scores, here's the short version.
The Brier score measures the accuracy of probabilistic predictions. It's the mean squared error between your predicted probability and the actual outcome (0 or 1). Lower is better. A perfect forecaster scores 0. A model that always predicts 50% scores 0.25. Random noise scores around 0.33.
The formula is simple:
import numpy as np
def brier_score(probabilities: list[float], outcomes: list[int]) -> float:
"""
Calculate mean Brier score for a set of probabilistic predictions.
Args:
probabilities: predicted probability of outcome = 1 (e.g. 0.72 means 72% chance YES)
outcomes: actual results, 1 for YES, 0 for NO
Returns:
float: mean Brier score (lower is better, 0 is perfect, ~0.25 is coin flip)
"""
probs = np.array(probabilities)
actuals = np.array(outcomes)
return np.mean((probs - actuals) ** 2)
The reason Brier score matters for this project specifically: Kalshi temperature contracts are binary. Either the high temperature hits the threshold or it doesn't. You're not predicting a continuous value — you're predicting a probability that a binary event occurs. Brier score is the right tool for that job.
Accuracy percentage lies. If my model says 95% YES and the answer is NO, that's a catastrophic miss. If I just report "we were right 60% of the time," that looks fine. Brier score captures the full cost of overconfidence.
What I Found When I Ran the Audit
I built a pipeline to pull all 112 completed trades from the local trade database and compare the model's predicted probability against the actual settlement outcome.
import sqlite3
import numpy as np
from dataclasses import dataclass
@dataclass
class TradeRecord:
trade_id: str
predicted_prob: float
contract_price: float # cents, 0-100
outcome: int # 1 = YES settled, 0 = NO settled
side: str # 'yes' or 'no'
def load_completed_trades(db_path: str) -> list[TradeRecord]:
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute("""
SELECT trade_id, predicted_probability, fill_price, outcome, side
FROM trades
WHERE status = 'settled'
ORDER BY settled_at ASC
""")
rows = cursor.fetchall()
conn.close()
return [TradeRecord(*row) for row in rows]
def audit_model_skill(trades: list[TradeRecord]) -> dict:
# Model's predicted probabilities
model_probs = [t.predicted_prob for t in trades]
outcomes = [t.outcome for t in trades]
# Base rate: just predict the historical win rate every time
base_rate = np.mean(outcomes)
base_rate_probs = [base_rate] * len(outcomes)
model_brier = brier_score(model_probs, outcomes)
base_rate_brier = brier_score(base_rate_probs, outcomes)
return {
"n_trades": len(trades),
"model_brier": round(model_brier, 4),
"base_rate_brier": round(base_rate_brier, 4),
"model_skill": round(1 - (model_brier / base_rate_brier), 4),
"base_rate_used": round(base_rate, 4),
}
Results:
{
"n_trades": 112,
"model_brier": 0.2858,
"base_rate_brier": 0.2439,
"model_skill": -0.1718,
"base_rate_used": 0.601
}
Negative skill score. The model_skill value is a version of the Brier Skill Score, where 0 means "no better than climatology" and negative means "actively worse." We were at -0.17. That's not a borderline result. That's a clear verdict.
The Market Was Already Right
Here's the part that stung the most.
I broke out the 112 trades by which side the model took — yes or no — and what Kalshi's market price was at the time of entry.
The bot was buying YES contracts at an average market price of around 60.6 cents. Those contracts settled YES about 60.0% of the time.
The bot was buying NO contracts at an average market price of around 44.4 cents (implied YES probability of 55.6%). Those markets settled YES about 54.8% of the time.
The prediction market was pricing these contracts with accuracy to within a single percentage point. Our model was not finding edge. It was confirming what the market already knew, then paying a spread to act on it.
I kept looking for the scenario where our model was consistently right when the market was wrong. It wasn't there. The disagreement was noise.
The Overconfidence Problem
The second finding was almost worse.
The old ensemble model produced extreme probabilities. It wasn't hedging. When it liked a trade, it was outputting 92%, 96%, 98% confidence. That's what happens when you combine multiple ensemble members without proper calibration — the signals reinforce each other and the output becomes a near-certainty.
A well-calibrated model predicts 95% only when the event actually happens 95% of the time. Ours was outputting 95%+ on trades that won about 60% of the time.
You can visualize calibration with a reliability diagram, but the math tells the story faster:
def check_calibration(trades: list[TradeRecord], n_bins: int = 10) -> list[dict]:
"""
Group predictions into probability bins and compare predicted
confidence to actual win rate within each bin.
"""
bins = np.linspace(0, 1, n_bins + 1)
results = []
for i in range(n_bins):
low, high = bins[i], bins[i + 1]
bucket = [t for t in trades if low <= t.predicted_prob < high]
if not bucket:
continue
mean_predicted = np.mean([t.predicted_prob for t in bucket])
actual_rate = np.mean([t.outcome for t in bucket])
results.append({
"bin": f"{low:.1f}-{high:.1f}",
"n": len(bucket),
"mean_predicted": round(mean_predicted, 3),
"actual_rate": round(actual_rate, 3),
"gap": round(mean_predicted - actual_rate, 3),
})
return results
The high-confidence bin (0.9-1.0) contained 31 trades. Mean predicted: 0.94. Actual win rate: 0.61. The model was saying 94% and delivering 61%. That gap is what the Brier score was measuring. That gap is expensive.
The 7 Defects We Found Along the Way
The model being miscalibrated was the core problem. But the audit also surfaced 7 separate technical defects that were contaminating the data.
The worst one was a daylight saving time bug. The National Weather Service defines a "weather day" for temperature records in Local Standard Time year-round. In summer, that means the official record window runs from 1:00 AM to 12:59 AM the following day — not midnight to midnight. Our forecasts were bucketing the wrong 24 hours for eight months of the year.
Every losing trade in the audit fell inside that DST window.
The second worst: we were pulling forecasts for the wrong airports. Kalshi settles Chicago temperature contracts on Midway, not O'Hare. Houston on Hobby, not Bush Intercontinental. We found this by querying Kalshi's own metadata API and comparing station IDs against what we were fetching.
def verify_settlement_stations(kalshi_client, our_station_map: dict) -> list[dict]:
"""
Pull Kalshi's official market metadata and compare settlement stations
against our internal mapping. Flags any mismatch.
"""
mismatches = []
markets = kalshi_client.get_weather_markets()
for market in markets:
ticker = market["ticker"]
kalshi_station = market.get("settlement_station_id")
our_station = our_station_map.get(ticker)
if kalshi_station != our_station:
mismatches.append({
"ticker": ticker,
"kalshi_says": kalshi_station,
"we_had": our_station,
})
return mismatches
Running that against production data returned 3 mismatches. Chicago, Houston, and one other city where we'd hardcoded an assumption instead of reading the metadata.
The other 5 defects are documented in detail on the product page. The pattern across all of them: we assumed correctness instead of verifying it.
The Uncomfortable Conclusion
The Kalshi weather markets are not obviously inefficient.
That's what the audit told me. The market prices were tracking actual outcomes with high accuracy. Our model was not surfacing information the market didn't have. We were not smarter than the aggregate of everyone else trading those contracts.
I spent two weeks trying to find an angle where the data said something different. A city subset where we had an edge. A temperature range where the model outperformed. A time of year. None of it held up.
The sample is 112 trades, which is not huge. But a Brier skill score of -0.17 is not a borderline result that might flip with more data. That's a signal.
What We Did About It
We didn't tune the ensemble. We didn't add more sources. We didn't change the calibration method.
We stopped using our own model.
NOAA already publishes the National Blend of Models — a professionally calibrated, bias-corrected temperature forecast that covers exactly the weather stations Kalshi uses for settlement. It's free. It runs on more data than we have access to. It's been validated by actual meteorologists across thousands of stations.
We were hand-rolling an inferior version of a public good and congratulating ourselves for it.
The rebuild pulls NBM forecasts directly from NOAA's public API:
import requests
from datetime import datetime, timezone
def fetch_nbm_forecast(station_id: str, target_date: datetime) -> dict:
"""
Fetch NOAA National Blend of Models temperature forecast for a
specific station and date. Returns calibrated high/low with
uncertainty bounds.
NOAA NBM endpoint — no API key required.
"""
base_url = "https://api.weather.gov/points"
# First, resolve station coordinates
station_meta = requests.get(
f"https://api.weather.gov/stations/{station_id}",
headers={"User-Agent": "PredictAndProfit/2.5 contact@predictandprofit.io"},
timeout=10,
).json()
lat = station_meta["geometry"]["coordinates"][1]
lon = station_meta["geometry"]["coordinates"][0]
# Get the gridpoint for this lat/lon
grid_response = requests.get(
f"{base_url}/{lat},{lon}",
headers={"User-Agent": "PredictAndProfit/2.5 contact@predictandprofit.io"},
timeout=10,
).json()
forecast_url = grid_response["properties"]["forecastHourly"]
forecast = requests.get(
forecast_url,
headers={"User-Agent": "PredictAndProfit/2.5 contact@predictandprofit.io"},
timeout=15,
).json()
return {
"station": station_id,
"source": "NOAA_NBM",
"fetched_at": datetime.now(timezone.utc).isoformat(),
"periods": forecast["properties"]["periods"],
}
The rebuilt bot is currently in paper-trading mode. It has not been validated. We added 49 automated tests during the rebuild and fixed the 7 defects the audit found. Whether it performs better than the old model is a question that requires 100+ completed trades to answer, which is four to six months at current trading rates.
We are not claiming it works. We are claiming we know why the old one didn't, and we built the replacement on a foundation we can actually defend.
What This Actually Cost
Twenty-three dollars and four months.
That's the direct cost. The indirect cost is harder to measure. It's the confidence I had in the original ensemble that turned out to be completely unearned. It's the trades I didn't second-guess because the dashboard was green and the model looked busy.
The lesson is not "always distrust your models." The lesson is that a model that hasn't been scored against its own track record isn't a model. It's a guess wearing a lab coat.
The Brier score is not a complicated metric. It's subtraction and squaring. I should have been running it from day one, automatically, after every settlement. I wasn't. That's the actual mistake.
The audit pipeline is now part of the bot. Every settled trade feeds back into a running Brier calculation. If the score starts drifting in the wrong direction, I'll know before 112 trades go by.
def update_running_brier(db_conn, new_trade: TradeRecord) -> float:
"""
After each settlement, recalculate the running Brier score
and write it to the model_performance table.
"""
cursor = db_conn.cursor()
cursor.execute("""
SELECT predicted_probability, outcome
FROM trades
WHERE status = 'settled'
ORDER BY settled_at ASC
""")
all_settled = cursor.fetchall()
probs = [row[0] for row in all_settled]
outcomes = [row[1] for row in all_settled]
score = brier_score(probs, outcomes)
cursor.execute("""
INSERT INTO model_performance (calculated_at, n_trades, brier_score)
VALUES (CURRENT_TIMESTAMP, ?, ?)
""", (len(all_settled), score))
db_conn.commit()
return score
Where Things Stand
The Weather Bot is rebuilt and undergoing validation. Paper trading only. No real money is at risk during this phase.
The post-mortem data — all 112 trades, the full Brier calculation, the defect list — is part of what ships with the product. Not because it's a selling point. Because anyone who buys this code deserves to know exactly what the previous version did wrong and why we think the rebuild addresses it.
Rigor is not a marketing story. It's the only defensible position when you're building a system that's supposed to make probabilistic predictions about the real world. Either you measure it honestly or you're fooling yourself.
We measured it. The number was bad. We showed our work.