< Back to Blog

Seven Defects in Four Months: What We Found When We Audited the Weather Bot

TL;DR / Key Takeaways

  • 112 completed trades. Net loss of roughly $23. Model Brier score of 0.2858 against a base-rate baseline of 0.2439. The model was statistically worse than making no prediction at all.
  • Seven distinct defects found during the audit. Four were silent failures: the system ran, logged green, and produced garbage.
  • The pattern: data access bugs and date-handling bugs are the hardest to catch because they fail quietly.
  • The lesson is not "write more tests," though we did add 49. The lesson is: audit your system against its own output before you blame the market.

Four months. 112 completed trades. A net loss of roughly $23.

That is not a catastrophic result. It is a mediocre one. And mediocre is worse than catastrophic in one specific way: it is ambiguous enough that you can talk yourself out of doing the hard work.

I did not talk myself out of it. I ran the audit.

What I found was seven defects. Some were embarrassing. One was genuinely alarming. All of them were fixable. Here is each one, in the order I found it.

The Audit Setup

Before I get into the defects, let me explain what "audit" means here. I wrote a pipeline that pulled every completed trade from the local database, matched it against the actual weather outcomes, and scored the model's predictions using the Brier score.

Brier score measures probabilistic accuracy. Lower is better. A score of 0.25 is what you get if you always predict 50% certainty on a binary outcome. Simply guessing the historical base rate for each market scored 0.2439. Our model scored 0.2858.

The model was not just bad. It was systematically worse than the simplest possible alternative. That is the result that made me stop tweaking and start auditing.

Defect 1: Daylight Saving Time Was Bucketing the Wrong 24 Hours

I wrote a full post on this one. The short version: the National Weather Service defines a weather "day" in Local Standard Time, year-round. In summer, the official weather day runs from 1:00 AM to 12:59 AM the following calendar day, not midnight to midnight.

Our forecast pipeline was using midnight-to-midnight UTC offsets. For eight months of the year, we were aggregating the wrong 24-hour window and comparing it against contract settlement values that used the correct window.

Every losing trade from the summer months fell in this DST window. I do not mean most of them. I mean every one.

This kind of bug is invisible in the logs. The pipeline runs, produces a temperature estimate, and moves on. Nothing raises an exception. Nothing looks wrong. You have to know that the input definition is wrong to catch it.

Defect 2: Wrong Settlement Airports

Also covered in its own post. The short version: Kalshi settles Chicago temperature contracts on Midway, not O'Hare. Houston settles on Hobby, not Bush Intercontinental.

We were pulling forecasts for the wrong stations. The markets settled on different data than we were predicting against.

The fix was to query Kalshi's own metadata API rather than guess. Every station is now confirmed against the exchange's published settlement data and pinned in tests.

# Before: hardcoded assumption
CITY_STATIONS = {
    "Chicago": "KORD",   # O'Hare. Wrong.
    "Houston": "KIAH",   # Bush. Wrong.
}

# After: pulled from Kalshi metadata and verified
CITY_STATIONS = {
    "Chicago": "KMDW",   # Midway. Correct.
    "Houston": "KHOU",   # Hobby. Correct.
}

Simple fix. Painful discovery.

Defect 3: xarray Truthiness Error

This one is a Python gotcha that bites anyone who works with NumPy or xarray without reading the documentation carefully enough.

The original forecast code had a fallback check that looked roughly like this:

forecast_data = get_nbm_forecast(station, date)
fallback_data = get_backup_forecast(station, date)

result = forecast_data or fallback_data

That looks reasonable. In plain Python, or returns the first truthy value. The problem is that forecast_data is an xarray DataArray, not a scalar. And xarray explicitly raises a ValueError when you use or on a DataArray, because the truthiness of a multi-element array is ambiguous.

ValueError: The truth value of an array with more than one element is ambiguous.
Use a.any() or a.all()

In certain code paths, this exception was being caught by a broad except Exception handler that logged a warning and returned None. So the fallback never ran. The system silently used no forecast data and continued.

The fix is explicit None checks:

forecast_data = get_nbm_forecast(station, date)
fallback_data = get_backup_forecast(station, date)

result = forecast_data if forecast_data is not None else fallback_data

That is it. One line. But the original version was quietly failing in specific edge cases and logging nothing useful.

Defect 4: kalshi_client.py Was Reading the Wrong Dict Key

This is the one that alarmed me.

The Kalshi API returns position data in this structure:

{
  "event_positions": [...],
  "market_positions": [...]
}

Our get_positions() wrapper in kalshi_client.py was reading the wrong key. The exact key name had drifted from the actual API response at some point during development, probably during an API version update. The wrapper returned an empty list every time it was called.

# What the code was doing
def get_positions(self):
    response = self._get("/portfolio/positions")
    # Wrong key. Returns empty list silently.
    return response.get("positions", [])

# What it should have been doing
def get_positions(self):
    response = self._get("/portfolio/positions")
    return response.get("market_positions", [])

The bot literally could not see its own open positions. Every call to get_positions() returned an empty list. The system thought it had no open trades.

The downstream consequences were compounding. The open-trade counter thought the book was empty, so it never refused new entries based on position limits. The risk management layer that checks existing exposure before placing new orders was effectively disabled.

Every green status light in the dashboard was real. The bot was trading. It just had no idea what it already owned.

This defect was in production for the entire four-month run. I found it by writing a test that compared what the database said we owned against a direct API call for current positions. They disagreed. That disagreement is what led me to the key name.

Defect 5: Stale Settlement Status

When a Kalshi contract settles, the bot is supposed to mark the trade as settled in the local database. Here is the relevant query from db/connection.py:

# The broken version
def settle_trade(self, trade_id: int, outcome: str, pnl: float):
    self.cursor.execute("""
        UPDATE trades
        SET outcome = %s,
            pnl = %s,
            settled_at = NOW()
        WHERE id = %s
    """, (outcome, pnl, trade_id))

Notice what is missing: status = 'settled'.

The outcome and pnl columns got updated correctly. But the status column stayed as 'open'. Every settled trade looked open in the database.

# The fixed version
def settle_trade(self, trade_id: int, outcome: str, pnl: float):
    self.cursor.execute("""
        UPDATE trades
        SET outcome = %s,
            pnl = %s,
            settled_at = NOW(),
            status = 'settled'
        WHERE id = %s
    """, (outcome, pnl, trade_id))

This interacted badly with Defect 4. The position-checking code that was already broken was querying the local database as a fallback. With settled trades marked as open, even the fallback count was inflated. The bot was overcounting its own exposure from two directions simultaneously.

Defect 6: Trade Decision Logging Gaps

The trade_decisions table was supposed to record every candidate market the bot evaluated, including the ones it skipped and why. That is how you audit a trading system. You need the full population of decisions, not just the ones that fired.

The original implementation only logged trades that executed. Skipped candidates disappeared without a trace.

# The original loop, simplified
for market in candidates:
    signal = evaluate_market(market)
    if signal.should_trade:
        place_order(market, signal)
        log_decision(market, signal, action="TRADE")
    # Skipped markets: nothing logged here

This made the audit pipeline blind to most of the bot's actual behavior. I could see what it traded. I could not see what it passed on, or why.

The fix was adding logging at every exit point:

for market in candidates:
    signal = evaluate_market(market)

    if signal is None:
        log_decision(market, action="SKIP", reason="signal_error")
        continue

    if signal.edge < MIN_EDGE_THRESHOLD:
        log_decision(market, signal, action="SKIP", reason="insufficient_edge")
        continue

    if open_positions >= MAX_OPEN_TRADES:
        log_decision(market, signal, action="SKIP", reason="position_limit")
        continue

    if signal.should_trade:
        place_order(market, signal)
        log_decision(market, signal, action="TRADE")

That added 15 new call sites across the codebase. After the fix, the database started showing the full picture: how many candidates were evaluated per cycle, how many were skipped for each reason, and whether the skip reasons were reasonable or masking a deeper problem.

Without this, you are flying blind. A trading system that only logs its wins and executions is not a trading system you can audit.

Defect 7: Overconfident Probability Calibration

This last one is not a bug in the traditional sense. The code ran exactly as written. The problem was what it was computing.

The custom ensemble model produced extreme probabilities. It would output 95% or 98% confidence on markets where the actual outcome was roughly a coin flip. The calibration was wildly off.

When I matched predictions against outcomes across the 112 trades, the pattern was stark. Trades where the model said 95%+ confidence resolved correctly about 60% of the time. The model had learned to be loud, not accurate.

The Brier score confirms this. A model that says 95% and is right 60% of the time is doing enormous damage to its score. Each of those trades is effectively a large miss, because the score penalizes confident wrong predictions much more than uncertain wrong predictions.

Brier score = mean((predicted_probability - actual_outcome)^2)

Model at 0.95, outcome = 0:  (0.95 - 0)^2 = 0.9025
Model at 0.60, outcome = 0:  (0.60 - 0)^2 = 0.3600

The model was not slightly overconfident. It was producing near-certainties on genuinely uncertain markets. Every one of those misfires hit the score hard.

The fix for this one was not a calibration adjustment. The root cause was that I was hand-rolling a probability model on top of raw ensemble forecast data, and doing it badly. NOAA's National Blend of Models already publishes calibrated, bias-corrected probability forecasts for exactly the weather stations Kalshi contracts settle on. For free. The entire custom calibration layer was a worse version of a public good that already existed.

The rebuilt system uses NBM directly. The probability calibration is NOAA's problem, which they have been solving professionally for decades.

The Pattern

Seven defects. Four of them were silent failures.

The xarray truthiness error: silent. The wrong dict key: silent. The stale settlement status: silent. The logging gaps: silent by definition.

Silent failures are the dangerous ones. The system runs. The logs look normal. The dashboard shows green. Nothing tells you that the bot is trading blind, counting phantom positions, and bucketing the wrong 24 hours.

The only way I found them was by writing code that compared what the system said it was doing against what it actually did. The database said we owned 10 open positions. The API said 4. That gap is what cracked it open.

If I had stopped at "the model is probably just not good enough," I would have tweaked hyperparameters for another four months and found nothing. The market was not the problem. The pipeline was the problem.

The Rebuild

The v2.5 customer release contains all seven fixes, 49 new automated tests, and a rebuilt forecast core using NOAA NBM as the primary source. The Weather Bot is currently in paper-trading mode and has not been validated. That is the honest status. Rebuilt and undergoing validation is the accurate phrase. Nothing more.

The audit pipeline that found these defects is now part of the standard tooling. It runs against every batch of completed trades. The goal is to catch the next silent failure before it runs for four months.

If you are running any kind of automated system against real markets, audit your own pipeline before you audit your model. The market is not usually the first place to look.

The code ships as part of the $75 package at predictandprofit.io. Both bots, full source, no subscription. The post-mortem is built into the documentation because I am not interested in selling something I cannot explain honestly.