When to Stop Building and Start Using What Already Exists
TL;DR / Key Takeaways
- We built a custom 164-member weather ensemble for 4 months. NOAA's National Blend of Models does the same job better, with professional calibration, for free.
- The sunk cost trap is real. The longer you build, the harder it is to admit you should have used what already exists.
- There is a decision tree for this. Build custom only when no maintained public solution exists, or when you have measured evidence your version is better.
- The best code is sometimes no code. Deleting 400 lines of ensemble logic and replacing it with one NOAA API call was the right engineering move.
The Thing I Built That Shouldn't Have Needed Building
Four months into running my automated Kalshi weather trading bot, I audited it against its own trade database. 112 completed trades. Roughly $23 in losses. Brier score of 0.2858 on a scale where lower is better, compared to 0.2439 for simply guessing the historical base rate every time.
My custom forecasting model was statistically worse than making no prediction at all.
That is a specific kind of bad. Not "underperforming." Not "needs tuning." Worse than a coin flip dressed up in Python.
I had built a 164-member ensemble model pulling from 4 independent weather sources: GFS via Open-Meteo, NOAA AIGEFS via AWS S3, ECMWF IFS, ECMWF AIFS-ENS. I had probability calibration logic. I had weighted averaging. I had code I was quietly proud of.
And NOAA had already done all of it better, published it for free, and called it the National Blend of Models.
What the NBM Actually Is
The National Blend of Models is NOAA's operational multi-model blend. It takes output from global and regional numerical weather prediction models, applies statistical post-processing and bias correction, and produces calibrated probabilistic forecasts for specific observation stations across the US.
The key word is calibrated. When NBM says there's a 70% chance the high temperature at Chicago Midway exceeds 85°F, that probability is grounded in historical verification against actual observed outcomes. It is not a raw model output. It has been corrected for known biases using years of performance data across thousands of stations.
My ensemble was not calibrated. It averaged 164 member forecasts and produced a probability. The fact that I had 164 members sounded rigorous. It was not rigorous. It was confident.
The NBM covers the exact ASOS weather stations that Kalshi uses for settlement. It publishes forecasts in GRIB2 format on NOAA's public servers. No API key. No rate limit. No cost.
import requests
def fetch_nbm_bulletin(station: str, date: str) -> dict:
"""
Fetch NBM short-range text bulletin for a specific station.
Station codes are 4-character ICAO identifiers (e.g. KMDW for Midway).
"""
base_url = "https://forecast.weather.gov/product.php"
params = {
"site": station[:3], # NWS office code
"issuedby": station,
"product": "NBM",
"format": "txt",
"version": "1",
"glossary": "0"
}
response = requests.get(base_url, params=params, timeout=10)
response.raise_for_status()
return {
"station": station,
"date": date,
"raw": response.text,
"source": "NOAA_NBM"
}
That is the core of what replaced 400+ lines of ensemble averaging code.
The Decision Tree I Should Have Used Before I Started
I did not consciously decide to build a custom ensemble. I built it because it felt like the right engineering approach. More data sources means more signal. More members means better coverage of model disagreement. This logic is not wrong in principle. It is wrong when a professionally maintained public solution already solves the problem.
Here is the decision tree I use now before writing a single line of code:
1. Does a professionally maintained public solution exist for this specific problem?
If yes, go to step 2. If no, build.
For weather forecast calibration: yes, NBM exists.
2. Does it actually cover your use case?
Not "close enough." Actually. Kalshi settles on specific ASOS stations. Does NBM publish calibrated forecasts for those stations? I verified this before switching.
# Stations we verified against Kalshi settlement metadata
KALSHI_SETTLEMENT_STATIONS = {
"Chicago": "KMDW", # Midway, not O'Hare
"Houston": "KHOU", # Hobby, not Bush
"New York": "KNYC", # Central Park
"Los Angeles": "KLAX",
"Phoenix": "KPHX",
"Miami": "KMIA",
# ... verified against Kalshi metadata API, not assumed
}
def verify_nbm_coverage(stations: dict) -> dict:
"""
Confirm NBM has recent data for each settlement station.
Returns coverage report with last available forecast timestamp.
"""
coverage = {}
for city, icao in stations.items():
try:
result = fetch_nbm_bulletin(icao, date="latest")
coverage[city] = {
"icao": icao,
"covered": True,
"chars": len(result["raw"])
}
except Exception as e:
coverage[city] = {
"icao": icao,
"covered": False,
"error": str(e)
}
return coverage
NBM covers every station on that list. Use case confirmed.
3. Have you measured whether your custom version is actually better?
Not assumed. Measured. Against real outcomes.
This is the step I skipped. I assumed my ensemble would outperform a simpler baseline because it was more complex. Complexity is not a proxy for accuracy. I found out the hard way, via 112 trades and a Brier score that was embarrassing to write down.
4. Is the maintenance burden of your custom solution worth the marginal gain?
Even if your custom version is marginally better, you now own it. You maintain it. When NOAA updates their model, your ensemble may silently drift. When a new source adds a schema change, your parser breaks. The NBM team handles all of that. Their job is to make this data accurate. Your job is to trade markets, not maintain a meteorological ensemble.
If the answer to step 3 is "yes, I measured and my version is better," then weigh that gain against the maintenance load. If it is "I think it's probably better," go back to step 3.
The Sunk Cost Trap Is Not Metaphorical
I knew about the NBM before I built the ensemble. I had looked at it. I decided to build my own because I thought I could tailor it more precisely to the Kalshi use case.
Four months in, with the post-mortem finished, I had to confront that I had spent significant time building something that was strictly worse than the free alternative I had dismissed.
The rational move at that point is obvious: switch to NBM. But there is a real psychological pull toward "let me tune the ensemble a bit more first." You have sunk time and attention into something. Abandoning it feels like failure. Tuning it one more time feels like progress.
It is not progress. It is avoidance.
The switch to NBM took less than a week of work. The ensemble had taken months. The ROI on that comparison is not close.
Here is what the probability extraction looks like after the switch. The old ensemble code was a class with multiple weighted averaging methods. This is what replaced it:
import re
from typing import Optional
def extract_nbm_high_temp_probability(
bulletin_text: str,
threshold_f: float
) -> Optional[float]:
"""
Extract probability of exceeding a temperature threshold from
NBM text bulletin. Returns float [0.0, 1.0] or None if not found.
NBM bulletins include lines like:
TX MAX/MIN 88 86 84 ...
P12 67 58 52 ... (prob of precip)
For temperature exceedance we use the Q-Q distribution lines.
"""
lines = bulletin_text.split("\n")
# Find the maximum temperature forecast line
tx_pattern = re.compile(r"^TX\s+MAX/MIN\s+([\d\s]+)")
q_pattern = re.compile(r"^Q(\d+)\s+([\d\s]+)")
tx_values = []
quantiles = {}
for line in lines:
tx_match = tx_pattern.match(line.strip())
if tx_match:
tx_values = [int(v) for v in tx_match.group(1).split()]
q_match = q_pattern.match(line.strip())
if q_match:
pct = int(q_match.group(1))
values = [int(v) for v in q_match.group(2).split()]
quantiles[pct] = values
if not quantiles or not tx_values:
return None
# Use quantile distribution to estimate exceedance probability
# for the first forecast period (index 0)
sorted_pcts = sorted(quantiles.keys())
for i, pct in enumerate(sorted_pcts):
if i == 0:
continue
lower_pct = sorted_pcts[i - 1]
upper_val = quantiles[pct][0]
lower_val = quantiles[lower_pct][0]
if lower_val <= threshold_f <= upper_val:
# Linear interpolation between quantiles
range_val = upper_val - lower_val
if range_val == 0:
return (100 - pct) / 100.0
position = (threshold_f - lower_val) / range_val
interpolated_pct = lower_pct + position * (pct - lower_pct)
return (100 - interpolated_pct) / 100.0
# Threshold is outside the quantile range
if threshold_f > quantiles[sorted_pcts[-1]][0]:
return 0.02 # Very low probability
if threshold_f < quantiles[sorted_pcts[0]][0]:
return 0.98 # Very high probability
return None
That is the whole probability extraction. No ensemble weighting. No calibration layer on top of calibration. NOAA already did the calibration. I just parse the output.
When Custom Is Still the Right Answer
I am not arguing you should never build anything. That would be a stupid conclusion to draw from one post-mortem.
Custom is the right answer when:
-
No maintained public solution exists. The Econ Bot's homemade weighted nowcast pulls from Cleveland Fed, BLS, and BEA and combines them into a single probability estimate. There is no pre-packaged solution for this. We built it.
-
You have measured evidence your version outperforms the alternative. Measured. Not intuited. If your Brier score is 0.18 and the public baseline is 0.24, and you have enough samples to be confident in that gap, build and maintain the custom version.
-
The public solution does not cover your specific use case. If Kalshi settled on stations NBM did not cover, that would be a real gap requiring a custom solution.
-
The public solution has licensing restrictions that affect your use. NBM data is US government-produced and in the public domain. Not everything is.
None of those conditions applied to weather temperature forecasting for Kalshi markets. NBM covered every station, had better calibration than anything I was going to build, and was free.
What the Rebuild Actually Produced
The v2.5 Weather Bot ships with NBM as the primary forecast source. The 164-member custom ensemble is gone. NOAA AIGEFS is retained as a secondary confirmation source, not because I think it adds a lot, but because having a second independent check on forecast direction costs almost nothing.
49 new automated tests were added during the rebuild. Some of those test the new NBM parsing. Some test the DST bug that was corrupting the 24-hour forecast window for 8 months of the year. Some test the station mapping verification.
The bot is in paper-trading mode. It has not been validated. That is the honest status and I am not pretending otherwise.
What I can say is that the architecture is no longer built on top of an unvalidated custom forecasting layer. The probability inputs come from a source that NOAA's meteorologists maintain, verify daily, and bias-correct against observed outcomes across 4,228 weather stations. That is a better foundation than whatever I was going to tune my ensemble to.
The Lesson, Which Is Obvious in Retrospect
The best code is sometimes no code.
Before you write a line of implementation, ask whether someone with more domain expertise, more historical data, and a full-time team has already solved this problem and published the solution.
If they have, use it. Verify it covers your use case. Measure whether it actually performs. Then ship.
The engineering pride wrapped up in "I built this myself" is real. I feel it. But it is not a good reason to maintain an inferior solution. Four months and 112 trades taught me that in a way that reading about it never would have.
The NBM switch took a week. The post-mortem that forced the switch took four months and $23 to generate. That is the actual cost of skipping the decision tree at the start.