< Back to Blog

Why We Audit Our Own Bot Instead of Trusting the P&L

TL;DR / Key Takeaways

  • P&L is an outcome measure. Brier score and calibration analysis are process measures. You need both.
  • Our weather bot lost $23 over 112 trades. The audit revealed the real problem: the model scored 0.2858 on Brier vs 0.2439 for a base-rate guess. Worse than doing nothing.
  • A model can make money and still have no skill. A model can lose money and still be worth keeping. Outcome and quality are not the same thing.
  • The right question after any automated trading run is not "did we make money?" It is "did our model add any information the market didn't already have?"

The Number That Made Me Uncomfortable

After four months of running the weather bot, I pulled the trade ledger and ran the numbers. Net result: a loss of roughly $23. Annoying, but not catastrophic. The kind of number you could wave away as variance and move on.

I didn't wave it away.

The $23 loss was the least interesting thing in that database. What the loss couldn't tell me was whether the model was broken or whether I'd just hit a bad run of variance. Those are completely different problems with completely different fixes. If it's variance, you stay the course. If the model has no skill, staying the course just means losing more money more slowly.

So I ran the actual audit. What I found was worse than a losing streak.

P&L Is the Wrong Scorecard

Here's the fundamental problem with using P&L to evaluate a prediction model: markets are partially efficient. A bad model can make money if the market misprice is large enough. A good model can lose money if the variance on a small sample swamps the edge. Outcome tells you almost nothing about model quality on short time horizons.

Think about it this way. If I flip a coin to decide every trade and happen to win 60% in my first 50 trades, my P&L looks great. My model is garbage. The sample size just hasn't had time to punish me yet.

What you actually need to know is: does my model produce probability estimates that are better than what the market is already pricing in? If the market says a contract has a 60% chance of settling Yes, and my model also says 60%, I have zero edge. I'm paying fees to replicate what the market already knows.

The only honest way to answer that question is to compare your model's probability estimates against observed outcomes. Systematically. Across every trade. That's a Brier score analysis, and that's what I should have been running from day one.

What a Brier Score Actually Measures

The Brier score is simple. For each prediction, you take the probability your model assigned to the outcome that actually happened, subtract it from 1, and square it. Average those values across all your predictions. Lower is better. A perfect model scores 0.0. A model that assigns 50% to everything scores 0.25. Random guessing on binary outcomes scores around 0.25.

The formula:

def brier_score(predictions: list[tuple[float, int]]) -> float:
    """
    predictions: list of (forecast_probability, actual_outcome)
    actual_outcome: 1 if the predicted event happened, 0 if it didn't
    Returns Brier score. Lower is better. 0.25 = random guessing.
    """
    n = len(predictions)
    if n == 0:
        raise ValueError("No predictions to score")
    
    total = sum((prob - outcome) ** 2 for prob, outcome in predictions)
    return total / n

For our 112-trade audit, I pulled the model's confidence at trade entry and the final settlement outcome for every completed trade. The code to reconstruct this from the trade database looked like this:

import sqlite3

def load_trade_predictions(db_path: str) -> list[tuple[float, int]]:
    """
    Pulls forecast probability and settlement outcome for every completed trade.
    forecast_prob: model's confidence at entry that the contract would settle Yes
    settled_yes: 1 if it did, 0 if it didn't
    """
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()
    
    cursor.execute("""
        SELECT 
            t.forecast_prob,
            CASE WHEN t.side = 'yes' AND t.result = 'won' THEN 1
                 WHEN t.side = 'no' AND t.result = 'lost' THEN 1
                 ELSE 0
            END AS settled_yes
        FROM trades t
        WHERE t.status = 'settled'
        AND t.forecast_prob IS NOT NULL
    """)
    
    rows = cursor.fetchall()
    conn.close()
    return [(float(row[0]), int(row[1])) for row in rows]


predictions = load_trade_predictions("trades.db")
score = brier_score(predictions)
print(f"Model Brier score: {score:.4f}")
# Output: Model Brier score: 0.2858

0.2858. That's the number that ended the original ensemble.

The Baseline That Made It Worse

A Brier score in isolation doesn't mean much. You need a baseline to compare against. The natural baseline for a binary prediction market is the historical base rate: how often does this type of contract settle Yes, regardless of any model?

For our temperature markets, the historical base rate across the trade window was roughly 52%. A model that just says "52% on everything, every time" would score:

base_rate = 0.52

# Brier score for a constant-probability forecaster
baseline_score = sum(
    (base_rate - outcome) ** 2 
    for _, outcome in predictions
) / len(predictions)

print(f"Base rate Brier score: {baseline_score:.4f}")
# Output: Base rate Brier score: 0.2439

0.2439 for the dumb baseline. 0.2858 for our model. Our model was statistically worse than making no prediction at all. Every bit of compute, every ensemble member, every API call was adding noise instead of signal.

That is the number that matters. Not the $23.

Calibration: Where the Model Was Actually Lying

Brier score tells you the overall quality. Calibration analysis tells you how the model is wrong. Those are different questions.

A well-calibrated model that says "70% confidence" should be right about 70% of the time across all such predictions. If your model says 70% but is only right 55% of the time at that confidence level, it's overconfident. That overconfidence has a specific name in forecasting literature: it's called underdispersion, and it's extremely common in homemade ensemble models.

Here's how I checked calibration across confidence buckets:

from collections import defaultdict

def calibration_analysis(predictions: list[tuple[float, int]], 
                          n_buckets: int = 10) -> dict:
    """
    Groups predictions into confidence buckets and checks
    whether actual win rates match predicted probabilities.
    """
    buckets = defaultdict(list)
    
    for prob, outcome in predictions:
        bucket = int(prob * n_buckets) / n_buckets  # floor to nearest 0.1
        buckets[bucket].append(outcome)
    
    results = {}
    for bucket_min, outcomes in sorted(buckets.items()):
        n = len(outcomes)
        actual_rate = sum(outcomes) / n
        bucket_center = bucket_min + (0.5 / n_buckets)
        results[bucket_center] = {
            "predicted": bucket_center,
            "actual": actual_rate,
            "n": n,
            "overconfident": bucket_center > actual_rate
        }
    
    return results

What this revealed was ugly. The model was producing probabilities in the 90-98% confidence range for contracts that settled correctly about 60% of the time. It was treating coin flips like certainties.

The 4.2x underdispersion figure I've mentioned before came from this analysis. The model's probability spread was 4.2 times wider than the actual outcome spread warranted. It was confidently wrong in both directions.

The Market Efficiency Check

The third piece of the audit was the one that stung the most. Kalshi's market prices are themselves probability estimates. If the market prices a contract at 0.60, that means informed participants collectively believe there's a 60% chance it settles Yes.

Efficient market hypothesis applied to prediction markets: the price already reflects the best available public information. If your model can't beat the market price as a probability estimate, you have no edge. You're just adding fees.

Here's the comparison I ran:

def compare_to_market(trade_records: list[dict]) -> dict:
    """
    Compares model forecast probability against market price (entry price)
    as competing probability estimators.
    trade_records: list of dicts with keys:
        'model_prob', 'entry_price', 'settled_yes'
    """
    model_predictions = [(r['model_prob'], r['settled_yes']) for r in trade_records]
    market_predictions = [(r['entry_price'], r['settled_yes']) for r in trade_records]
    
    model_brier = brier_score(model_predictions)
    market_brier = brier_score(market_predictions)
    
    return {
        "model_brier": model_brier,
        "market_brier": market_brier,
        "model_beats_market": model_brier < market_brier,
        "edge": market_brier - model_brier  # positive means model is better
    }

result = compare_to_market(trade_records)
# model_brier: 0.2858
# market_brier: 0.2491  (market was also close to the base rate)
# model_beats_market: False
# edge: -0.0367  (negative means market was better)

The market was right. Our model added negative value.

The contracts we bought at an average entry price of 60.6 cents won about 60.0% of the time. The market's implied probability was almost exactly correct. The contracts we bought at 44.4 cents won 45.2% of the time. Again, the market was pricing them almost perfectly.

We were paying fees to trade on information that was already in the price.

Why This Matters Beyond Weather Trading

This audit methodology applies to any automated decision system, not just trading bots. If you're running a model that makes binary predictions and acts on them automatically, you should be measuring these three things:

Brier score against a naive baseline. If your model can't beat the base rate, it has no predictive skill. Full stop.

Calibration across confidence levels. A model that's right 70% of the time when it says 70% is calibrated. A model that's right 55% of the time when it says 95% is dangerous.

Comparison against the best available alternative. In trading, that's the market price. In other domains, it's whatever the best public benchmark is. If you can't beat the free alternative, you don't have an edge, you have operational overhead.

These are not exotic metrics. They're standard tools in forecasting research that somehow rarely make it into the code of the people who build automated trading systems.

What the Audit Led To

The audit killed the ensemble approach. Not because ensembles are wrong in principle, but because our probability calibration was broken and we were hand-rolling a worse version of something NOAA already publishes for free.

The National Blend of Models is NOAA's operationally calibrated temperature forecast product. It covers exactly the weather stations Kalshi uses for settlement. It's bias-corrected, uncertainty-quantified, and available at no cost. We were building a custom ensemble of 164 members across 4 forecast sources and producing worse probability estimates than what NOAA publishes every day as a public good.

The rebuild switched to NBM as the primary source and fixed 7 separate technical defects that had been corrupting results: a DST date-bucketing error that affected 8 months of trades, wrong settlement airport mappings, a dict key bug that meant the bot couldn't see its own open positions.

The bot is now in paper-trading mode and has not been validated. That's the honest status. Rebuilt and undergoing validation. Forecast quality can be assessed in roughly two weeks of daily verification runs. Whether the strategy has edge against market pricing requires 100 or more completed trades, which is four to six months at current rates.

The Uncomfortable Part

Running this audit meant sitting with the conclusion that four months of work had produced a model with no skill. That's not fun. The temptation when you see a $23 loss is to tweak parameters, add a signal source, adjust thresholds, and declare the problem solved. That's how you end up with an overfitted system that passes backtests and fails live.

The audit forced a different conclusion: the architecture was wrong. The fix wasn't cleverness. It was honesty.

P&L will tell you if you made money in the past. It won't tell you if you'll make money in the future, and it won't tell you why you made or lost what you did. For that, you need to measure the model directly, against a baseline, against the best alternative you have access to, and across the full distribution of confidence levels.

If you're running any automated prediction system and you're not doing this, you're flying with the instruments covered. The P&L needle might look okay right now. That doesn't mean you know where you're going.

The code for the full audit pipeline is in the repo. Run it on your own trade database. Find out what your model actually knows.