TL;DR / Key Takeaways

The Kalshi market priced our weather contracts to within one percentage point of actual outcomes, with no help from us
Our custom forecasting model scored 0.2858 on the Brier scale; guessing the base rate scores 0.2439 — we were statistically worse than doing nothing
In prediction markets, all relevant data is public by definition, which makes beating market prices with public data extremely difficult
If your model uses only public data and still can't beat the market price, you don't have edge — you have noise

The Numbers That Ended an Argument

After four months and 112 completed trades, I sat down and ran the audit I had been quietly dreading.

The bot had been buying temperature contracts on Kalshi. Some contracts predicted that the daily high would exceed a threshold. Others predicted it wouldn't. The bot had a model. The model had confidence. The confidence was wrong.

Here are the two numbers that broke the argument:

Contracts bought at an average price of 60.6 cents won 60.0% of the time
Contracts bought at an average price of 44.4 cents won 45.2% of the time

One percentage point. The market was off by one percentage point. Across 112 trades, spanning multiple cities and months, the crowd-sourced price was essentially the correct probability.

My model added nothing. The market already knew.

What a Prediction Market Actually Is

Before getting into why this is so damaging to the "beat the market with public data" thesis, it helps to understand what Kalshi is doing when it sets a price.

A Kalshi temperature contract pays $1 if the daily high at a specific airport exceeds a specific temperature threshold on a specific date. If it's trading at 62 cents, the market is saying: there's roughly a 62% chance that happens.

That price is set by traders buying and selling against each other. Buyers who think the temperature will exceed the threshold buy contracts. Sellers who think it won't take the other side. The price moves until there's equilibrium.

This is straightforward. The important part is what information is embedded in that price.

In an equity market, the price reflects a mix of public information and private information. Institutional players have access to better data, faster data, proprietary models, and in some cases information that retail traders simply don't have. The playing field is not level. There are real edges for people with real information advantages.

In a prediction market about tomorrow's weather, all of the relevant information is public. NOAA publishes its forecasts. Weather Underground publishes station readings. The National Weather Service publishes model output. Every piece of data that a trader might use to estimate tomorrow's high temperature in Chicago is available to every trader.

That changes the math considerably.

Why Public Data Strategies Are Harder Than They Look

Here's the uncomfortable implication: if every trader in the market has access to the same public forecast data, and the market price reflects their collective judgment, then a model that also only uses public data is fighting against its own inputs.

The market price is already downstream of NOAA, Open-Meteo, Weather Underground, and every other public source. When I built a custom ensemble using those same sources and tried to find edge against market prices, I was essentially asking: "Can I process public data slightly better than the collective judgment of everyone else who is also processing public data?"

The answer, apparently, was no.

This is not a shocking finding if you think about it. It is, in a narrow sense, a version of the efficient market hypothesis applied to a prediction market. The strong form of that hypothesis — that prices reflect all information, public and private — is a description of what happened to us in practice.

The Brier score confirmed it quantitatively. The Brier score measures probabilistic forecast accuracy. Lower is better. A score of 0.25 represents random guessing on a 50/50 event. Our model scored 0.2858. The base rate naive model — simply predicting historical average frequencies — scored 0.2439.

We were not just failing to beat the market. We were worse than not having a model at all.

import numpy as np

def brier_score(probabilities, outcomes):
    """
    Brier score: mean squared error between predicted probability
    and actual binary outcome.
    Lower is better. 0.25 = random on 50/50. 0.0 = perfect.
    """
    probs = np.array(probabilities)
    actual = np.array(outcomes)
    return np.mean((probs - actual) ** 2)

# Our model's predictions vs outcomes across 112 trades
model_score = brier_score(model_probabilities, outcomes)
# Result: 0.2858

# Naive baseline: just use historical win rate for every prediction
base_rate = np.mean(outcomes)
naive_predictions = [base_rate] * len(outcomes)
naive_score = brier_score(naive_predictions, outcomes)
# Result: 0.2439

print(f"Model: {model_score:.4f}")
print(f"Naive baseline: {naive_score:.4f}")
print(f"Model is {'better' if model_score < naive_score else 'worse'} than guessing")
# Output: Model is worse than guessing

That output is not a fun thing to see. But it is an honest thing to see.

The Confidence Problem

The Brier score tells you accuracy. The calibration plot tells you something nastier.

Our model was producing probability estimates in the 90-98% range on contracts that were genuine coin flips. The model didn't know it was uncertain. It was confident the way a first-year engineer is confident: without enough experience to know what it didn't know.

When you're wrong at 95% confidence on a contract that wins 55% of the time, you've made a much worse decision than just buying at 55 cents. You've sized up based on false certainty. The financial damage from overconfident errors is disproportionate to the number of errors.

Here's what calibration looks like when it's working, versus what ours looked like:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.calibration import calibration_curve

# Perfect calibration: predicted 70% = wins 70% of the time
# Our model: predicted 95% = wins ~60% of the time

def plot_calibration(probabilities, outcomes, label):
    fraction_of_positives, mean_predicted_value = calibration_curve(
        outcomes,
        probabilities,
        n_bins=10,
        strategy='uniform'
    )
    plt.plot(
        mean_predicted_value,
        fraction_of_positives,
        marker='o',
        label=label
    )

# Perfect calibration reference line
plt.plot([0, 1], [0, 1], 'k--', label='Perfect calibration')

plot_calibration(model_probabilities, outcomes, 'v2.1 ensemble')
plot_calibration(noaa_probabilities, outcomes, 'NOAA NBM (current)')

plt.xlabel('Mean predicted probability')
plt.ylabel('Fraction of positives')
plt.title('Calibration curve')
plt.legend()
plt.tight_layout()
plt.savefig('calibration_comparison.png', dpi=150)

The v2.1 ensemble line was bent hard to the right. High predicted probabilities corresponded to mediocre actual win rates. The NOAA NBM line sits much closer to the diagonal, because NOAA spends considerable effort on exactly this problem. Calibration is not a nice-to-have for a forecast model. It is the job.

What Makes Prediction Markets Different From Equity Markets

I want to be precise here, because the efficiency argument is often overstated in one direction and understated in another.

Equity markets are not fully efficient. There is substantial evidence that markets with information asymmetry allow for real edges. If you have better data, faster data, or a smarter model applied to data others don't have, you can make money. Quantitative hedge funds exist and are profitable. This is not controversial.

Prediction markets are different in a specific way: the underlying question is usually resolvable with public data. "Will the daily high at Midway Airport exceed 72°F on August 20?" is a question where all the inputs are public. NOAA publishes the relevant forecasts. The settlement station is specified in the contract. The methodology is documented.

This doesn't mean prediction markets are perfectly efficient. But it does mean the margin for information advantage is narrower. The person on the other side of your trade has access to the same NOAA bulletin you do.

There are a few legitimate sources of edge that don't require private data:

Systematic processing at scale. If you can scan 291 markets per cycle and identify the ones where market prices are least consistent with calibrated forecasts, you might find opportunities human traders miss from exhaustion or bandwidth limits.
Faster reaction to new data. When NOAA updates the NBM at 1 AM, a bot that processes that update in 30 seconds has a brief window before other traders adjust prices. This edge is narrow and requires honest measurement to confirm it exists at all.
Specific miscalibration in thin markets. Markets with low volume may have wider spreads and less efficient prices, because there aren't enough traders to quickly correct mispricings. This requires identifying which markets are thin and whether the inefficiency is real or illusory.

None of these are guaranteed edges. They are hypotheses that require testing. Our v2.1 model didn't test any of them honestly. It assumed edge existed and then tried to measure something else.

The Real Fix: Use What NOAA Already Built

After the audit, the question was: what should the replacement model actually do?

The answer was uncomfortable in a different way. NOAA's National Blend of Models already publishes calibrated, bias-corrected temperature forecasts with full uncertainty ranges. For free. For exactly the weather stations Kalshi uses for settlement. We were building a worse version of a thing that already existed and ignoring the thing that existed.

The v2.3 rebuild replaced the custom ensemble with NBM as the primary source. The NBM downloads are free, structured, and keyed to the station identifiers that Kalshi uses for settlement. A basic pull looks like this:

import requests

def fetch_nbm_bulletin(station_id: str, date_str: str) -> dict:
    """
    Fetch NBM point forecast bulletin for a given station.
    Station IDs come from Kalshi contract metadata, not from guessing.
    """
    # NBM text bulletins are served via NOAA's public API
    url = (
        f"https://api.weather.gov/points/"
        f"{station_lat},{station_lon}"
    )

    headers = {
        "User-Agent": "PredictAndProfit/2.5 contact@predictandprofit.io"
    }

    response = requests.get(url, headers=headers, timeout=10)
    response.raise_for_status()

    forecast_url = response.json()["properties"]["forecastHourly"]
    forecast_response = requests.get(
        forecast_url,
        headers=headers,
        timeout=10
    )
    forecast_response.raise_for_status()

    return forecast_response.json()

The point here is not that this code is clever. It isn't. The point is that we stopped trying to be clever and started using what already worked.

What This Means for Anyone Building on Public Data

If you're building a model that trades against a prediction market price, and your model uses only publicly available data, here is the minimum intellectual honesty requirement:

You have to demonstrate that your model's probability estimates are better than the market price, not just that they are different.

Being different from the market is not edge. Markets are wrong sometimes. You have to be wrong less often than the market, or right in the specific direction the market is wrong, and you have to be able to prove that on held-out data before you deploy real money.

The way to measure this is the same way we measured our failure: Brier score, calibration curves, and comparison against a naive baseline. If your model can't beat the base rate, stop. If it can beat the base rate but can't beat the market price, you don't have trading edge even if you have forecasting skill.

One percentage point is not a rounding error. It is a signal. When the market prices contracts to within one percentage point of actual outcomes across 112 trades, it is telling you that a lot of smart people have already incorporated the information you think is your advantage.

Where We Are Now

The Weather Bot is rebuilt and in paper-trading mode. It is not validated. We will not claim it has edge until we have enough completed paper trades to run the same audit we ran on v2.1. That takes roughly four to six months at current trading frequency.

The Brier score comparison, the calibration plot, the audit pipeline — those all run on paper trades exactly the same way they ran on real trades. The goal is to know, with the same rigor we applied to the failure, whether the rebuild actually performs differently.

If the numbers come back the same way, we'll say so. That's the job.

Prediction markets are hard to beat. The market that priced our trades to within one percentage point was not being kind to us. It was being honest. We could stand to do the same.

The Market Was Right: How Kalshi Priced Our Trades to Within One Percentage Point