TL;DR / Key Takeaways

The v2.1 Weather Bot had 7 distinct defects. We found them by auditing 112 completed trades against the bot's own database.
Silent failures in automated trading are expensive. The bot runs while you sleep. If a test doesn't catch a regression, the market will.
We added 49 new automated tests targeting the exact failure modes we found: DST bucketing, station mapping, position visibility, calibration, and logging.
The test suite is not proof the strategy works. It is proof the specific defects we already paid for cannot return quietly.

The v2.1 post-mortem was not fun to write. 112 trades, $23 loss, Brier score of 0.2858 against a base-rate baseline of 0.2439. Our model was statistically worse than making no prediction at all. That is the kind of result that forces honesty.

We found 7 distinct defects. Some were subtle. Some were embarrassing. All of them were real and all of them cost money. The rebuild fixed them. But fixing a bug without a test is a promise, not a guarantee. The next version of you, three months from now, refactoring something unrelated, will reintroduce it. That is not a failure of character. That is just how software works.

So we wrote the tests.

49 of them, targeting the exact failure modes the post-mortem surfaced. This post walks through the logic, shows some of the actual test code, and explains why automated trading demands a higher standard of test coverage than most software categories.

Why Automated Trading Needs More Tests Than You Think

Most software fails loudly. A web app throws a 500. A CLI exits with a non-zero code. Something breaks in a way a human can see.

Automated trading fails quietly. The bot runs its cycle, logs "no markets found," and goes back to sleep. Everything looks fine. The dashboard is green. You're watching TV. The bug has been running for three weeks.

That is the specific failure mode we had. A filter was matching the string "Climate" but Kalshi's exchange uses "Climate and Weather" as the category. For weeks, the market scanner was seeing 13-20% of available markets. No exception. No error log entry that looked wrong. Just a very small number in a field we weren't watching closely enough.

Silent failures in trading have a direct dollar cost. A test that catches a silent failure before it runs is not overhead. It is risk management.

The second reason automated trading needs aggressive testing: the feedback loop is slow. A bug in a web app might surface in minutes. A bug in a trading bot that only fires on certain market conditions might not surface for weeks. By then you have real positions opened on bad data.

49 tests is not a lot for a mature codebase. But these 49 tests are precise. They target known defects with exact reproduction cases. That is more valuable than broad coverage over code paths that have never failed.

The 7 Defects and Their Test Categories

Here is what we found in the audit and what we wrote to prevent each from returning.

1. Daylight Saving Time Date Bucketing

This one hurt the most, because it was subtle enough that we felt smart when we first wrote the code.

NOAA and the National Weather Service define a "calendar day" for temperature records in Local Standard Time year-round. In summer, that means the official weather day runs from 1:00 AM local DST time to 12:59 AM the next day. Midnight to midnight in wall-clock time is not the same thing.

Our original code bucketed by calendar date in local time without accounting for this offset. In summer, we were pulling the wrong 24 hours of data. Every losing trade fell in the DST window.

The fix was straightforward once we understood the definition. The test ensures it stays fixed:

import pytest
from datetime import date
from zoneinfo import ZoneInfo
from weather.date_utils import get_nws_observation_window

def test_dst_window_chicago_summer():
    """
    In summer (DST active), the NWS observation window for a calendar date
    starts at 01:00 CDT (06:00 UTC) and ends at 00:59 CDT the next day (05:59 UTC).
    """
    chicago_tz = ZoneInfo("America/Chicago")
    obs_date = date(2026, 7, 15)  # Summer, DST active

    window_start, window_end = get_nws_observation_window(obs_date, chicago_tz)

    # Should start at 06:00 UTC (01:00 CDT)
    assert window_start.hour == 6
    assert window_start.tzinfo is not None

    # Should end at 05:59 UTC the next day (00:59 CDT)
    assert window_end.day == 16
    assert window_end.hour == 5


def test_dst_window_chicago_winter():
    """
    In winter (DST inactive), midnight CST = 06:00 UTC.
    Window should still anchor on Local Standard Time, not wall clock.
    """
    chicago_tz = ZoneInfo("America/Chicago")
    obs_date = date(2026, 1, 15)  # Winter, no DST

    window_start, window_end = get_nws_observation_window(obs_date, chicago_tz)

    # CST is UTC-6. Day starts at 06:00 UTC.
    assert window_start.hour == 6
    assert window_end.day == 16


def test_dst_boundary_spring_forward():
    """
    March 8, 2026 is the spring DST transition for US/Central.
    The observation window should not double-count or drop an hour.
    """
    chicago_tz = ZoneInfo("America/Chicago")
    obs_date = date(2026, 3, 8)

    window_start, window_end = get_nws_observation_window(obs_date, chicago_tz)
    duration = window_end - window_start

    # Window should be 23 hours on spring-forward day
    assert duration.seconds // 3600 == 23

Three tests for one bug. The boundary condition on the DST transition date is the one most likely to slip past a quick fix.

2. Settlement Station Mapping

Kalshi settles Chicago temperature contracts on Midway (KMDW), not O'Hare (KORD). Houston on Hobby (KHOU), not Bush Intercontinental (KIAH). We confirmed the full list by querying Kalshi's own public metadata API rather than trusting our assumptions.

The test isn't about whether the stations are correct today. It is about making sure nobody can change the mapping silently:

import pytest
from weather.station_config import KALSHI_SETTLEMENT_STATIONS

# These are verified against Kalshi metadata API responses.
# Do not change without re-running verify_stations_against_api.py
# and updating the verification timestamp in station_config.py.
KNOWN_CORRECT_MAPPINGS = {
    "Chicago": "KMDW",    # Midway, NOT O'Hare (KORD)
    "Houston": "KHOU",    # Hobby, NOT Bush (KIAH)
    "Dallas": "KDAL",     # Love Field
    "New York": "KLGA",   # LaGuardia
    "Los Angeles": "KLAX",
    "Miami": "KMIA",
    "Atlanta": "KATL",
    "Denver": "KDEN",
}

@pytest.mark.parametrize("city,expected_station", KNOWN_CORRECT_MAPPINGS.items())
def test_settlement_station_mapping(city, expected_station):
    """
    Settlement station must match Kalshi exchange metadata exactly.
    A wrong station silently trades on the wrong data.
    """
    assert city in KALSHI_SETTLEMENT_STATIONS, f"Missing city: {city}"
    actual = KALSHI_SETTLEMENT_STATIONS[city]
    assert actual == expected_station, (
        f"{city}: expected {expected_station}, got {actual}. "
        f"Verify against Kalshi metadata API before changing."
    )


def test_no_ohare_in_config():
    """O'Hare must never appear as a settlement station."""
    assert "KORD" not in KALSHI_SETTLEMENT_STATIONS.values(), (
        "O'Hare (KORD) found in settlement config. Chicago settles on Midway (KMDW)."
    )


def test_no_bush_intercontinental_in_config():
    """Bush Intercontinental must never appear as a settlement station."""
    assert "KIAH" not in KALSHI_SETTLEMENT_STATIONS.values(), (
        "Bush (KIAH) found in settlement config. Houston settles on Hobby (KHOU)."
    )

The negative tests for O'Hare and Bush are not paranoid. They exist because the wrong stations are the intuitive choices. If someone adds a new city by guessing, these tests catch it.

3. Position Visibility (The Dict Key Bug)

This was the most impactful single fix in the entire project. get_positions() in kalshi_client.py was reading the wrong key from the Kalshi API response. The API returns {'event_positions': [...], 'market_positions': [...]}. We were reading the wrong one. The bot could not see its own open positions.

Downstream from that: the regime-change detector saw zero positions and never triggered. The open-trade counter thought the book was empty and over-traded.

One wrong dictionary key. Two broken subsystems. Weeks of bad behavior.

import pytest
from unittest.mock import patch, MagicMock
from kalshi.client import KalshiClient

MOCK_API_RESPONSE = {
    "event_positions": [
        {"event_ticker": "KXHIGH-26JUL15-T75", "event_exposure": 120}
    ],
    "market_positions": [
        {
            "market_ticker": "KXHIGH-26JUL15-T75A",
            "position": 3,
            "position_fp": 300,
            "resting_orders_count": 0,
        }
    ],
}

def test_get_positions_reads_market_positions_key():
    """
    get_positions() must read 'market_positions', not 'event_positions'.
    Reading the wrong key returns empty and breaks downstream trade logic.
    """
    client = KalshiClient.__new__(KalshiClient)

    with patch.object(client, "_get", return_value=MOCK_API_RESPONSE):
        positions = client.get_positions()

    assert len(positions) == 1
    assert positions[0]["market_ticker"] == "KXHIGH-26JUL15-T75A"


def test_get_positions_returns_empty_list_not_none():
    """
    Empty positions must return [], not None.
    Downstream code iterates the result without a None check.
    """
    empty_response = {"event_positions": [], "market_positions": []}
    client = KalshiClient.__new__(KalshiClient)

    with patch.object(client, "_get", return_value=empty_response):
        positions = client.get_positions()

    assert positions == []
    assert positions is not None


def test_open_position_count_excludes_zero_fp():
    """
    Positions with position_fp == 0 are not active and must not count
    against the open trade limit. The old code counted them, causing
    the bot to think the book was full when it wasn't.
    """
    response_with_stale = {
        "event_positions": [],
        "market_positions": [
            {"market_ticker": "TICKER-A", "position": 3, "position_fp": 300},
            {"market_ticker": "TICKER-B", "position": 0, "position_fp": 0},  # stale
        ],
    }
    client = KalshiClient.__new__(KalshiClient)

    with patch.object(client, "_get", return_value=response_with_stale):
        active_count = client.get_active_position_count()

    assert active_count == 1

4. Trade Decision Logging

Before the rebuild, the trade_decisions table only recorded trades that fired. Skipped candidates were not logged. This meant we could not audit why the bot passed on a market. We were flying blind on the majority of its decisions.

We added 15 new call sites that log every rejection with a reason code. The test suite verifies that specific rejection paths actually write to the log:

import pytest
from unittest.mock import MagicMock, patch
from trading.executor import TradeExecutor

def test_skip_logged_when_edge_below_threshold(tmp_db):
    """
    When a candidate market has edge below MIN_EDGE_THRESHOLD,
    the skip must be recorded with reason='edge_below_threshold'.
    """
    executor = TradeExecutor(db=tmp_db, min_edge=0.05)
    candidate = {
        "ticker": "KXHIGH-26AUG18-T85A",
        "our_probability": 0.61,
        "market_yes_price": 0.58,  # edge = 0.03, below threshold
    }

    executor.evaluate(candidate)

    decisions = tmp_db.query(
        "SELECT reason FROM trade_decisions WHERE ticker = ?",
        ("KXHIGH-26AUG18-T85A",)
    )
    assert len(decisions) == 1
    assert decisions[0]["reason"] == "edge_below_threshold"


def test_skip_logged_when_event_position_cap_hit(tmp_db):
    """
    When the event position cap is active, the skip reason must be
    'event_position_cap', not a generic rejection.
    """
    executor = TradeExecutor(db=tmp_db, max_positions_per_event=2)

    # Simulate 2 existing positions on the same event
    tmp_db.execute(
        "INSERT INTO open_positions (event_ticker, market_ticker) VALUES (?, ?)",
        ("KXHIGH-26AUG18", "KXHIGH-26AUG18-T85A")
    )
    tmp_db.execute(
        "INSERT INTO open_positions (event_ticker, market_ticker) VALUES (?, ?)",
        ("KXHIGH-26AUG18", "KXHIGH-26AUG18-T87A")
    )

    candidate = {
        "ticker": "KXHIGH-26AUG18-T89A",
        "event_ticker": "KXHIGH-26AUG18",
        "our_probability": 0.72,
        "market_yes_price": 0.60,
    }

    executor.evaluate(candidate)

    decisions = tmp_db.query(
        "SELECT reason FROM trade_decisions WHERE ticker = ?",
        ("KXHIGH-26AUG18-T89A",)
    )
    assert decisions[0]["reason"] == "event_position_cap"

5. Settled Trade Status

settle_trade() was missing status = 'settled' in the UPDATE statement. Settled trades stayed marked as open in the local database. The open-trade counter read from the local DB, not the exchange, so it thought positions were still live after they had closed. The bot was managing ghost positions.

The test is simple and direct:

def test_settle_trade_updates_status_to_settled(tmp_db):
    """
    After settling a trade, the local DB record must have status='settled'.
    Missing this UPDATE caused settled trades to appear open indefinitely.
    """
    tmp_db.execute(
        "INSERT INTO trades (ticker, status) VALUES (?, ?)",
        ("KXHIGH-26JUL15-T75A", "open")
    )

    settle_trade(tmp_db, "KXHIGH-26JUL15-T75A", outcome="yes", pnl=0.40)

    row = tmp_db.query(
        "SELECT status FROM trades WHERE ticker = ?",
        ("KXHIGH-26JUL15-T75A",)
    ).fetchone()

    assert row["status"] == "settled"

The Coverage Report

After the 49 new tests landed, the suite broke down like this across modules:

| Module | Tests | What They Cover | |---|---|---| | date_utils.py | 8 | DST window logic, boundary dates, timezone edge cases | | station_config.py | 7 | Settlement station mappings, negative tests for wrong stations | | kalshi_client.py | 9 | Position visibility, dict key correctness, empty-list handling | | trade_executor.py | 11 | Decision logging, skip reasons, event position cap | | db/connection.py | 6 | Settlement status, stale record handling | | probability.py | 8 | Calibration scoring, xarray truthiness fix, extreme probability handling |

Total added: 49. Total suite after merge: 165.

The xarray truthiness fix gets its own category because it is the kind of error that is invisible until it crashes at runtime. Using Python's or operator on an xarray DataArray raises a ValueError. We replaced every instance with an explicit is None check and wrote tests that pass xarray objects through the probability module to confirm no implicit boolean evaluation remains.

What These Tests Don't Do

They do not prove the strategy works. They do not validate the NBM forecasts. They do not tell you whether the v2.5 bot will be profitable.

What they do is narrower and more honest: they prove that the specific defects we already paid for cannot come back quietly. The DST bug cannot silently re-enter because a refactor changes an import. The wrong settlement station cannot appear because someone adds a new city without checking the metadata. The bot cannot lose visibility into its own positions because someone upgrades the Kalshi client wrapper.

The difference between a test suite and a prediction is important. These tests make a narrow guarantee: known failure modes are detected before they ship. That is all they claim.

The Weather Bot is rebuilt and undergoing validation in paper-trading mode. The tests are part of the rebuild. Whether the strategy has edge against the market takes 100+ completed live trades to answer, which is four to six months at current trading rates.

We will publish that number when we have it, whatever it says.

Running the Suite

If you have the source package, the full test suite runs in under 90 seconds:

# From the project root
pytest tests/ -v --tb=short

# Just the regression tests added after the post-mortem
pytest tests/ -v -m "regression" --tb=short

# With coverage report
pytest tests/ --cov=weather --cov=kalshi --cov=trading --cov-report=term-missing

The regression mark is applied to all 49 tests added during the rebuild. Running them in isolation is a fast sanity check before any deploy.

Every serious defect we've shipped had one thing in common: it didn't have a test. Not because we were lazy, but because we didn't know it was a defect yet. The post-mortem created the defect list. The tests are what we write when we finally know what to check for. That's how this works.

If you find a bug we haven't found yet, we want to know. That's the whole point of building this in public.

49 New Automated Tests: How We Prevent the Same Bugs from Coming Back