My agent predicts my race finish times (per-discipline pace) from training data: fitness/fatigue history, threshold pace, power curves. The predictions work and ship — the problem is that making them meaningfully better has hit a wall that more code doesn't fix.
The setup: per-discipline XGBoost regressors trained on one athlete's history, bootstrap residuals (500 resamples) for a 90% confidence interval, and a "race day" mode that projects fitness forward to a future date. Runs fine, honest CIs. Leave-one-out MAE: ~55 sec/km on n=22 running races.
The wall: a single athlete simply doesn't race often enough. Per-race variance (course, weather, pacing decisions, day form) has a standard deviation of ~60 sec/km in my data. The systematic effect I actually want to model better — how prediction error grows with forecast horizon — is ~16 sec/km. The noise is roughly 4× the signal, and at ~10 races a year the dataset grows by drops.
Every attempt below was gated by a pre-registered decision rule (MAE drop ≥5 sec/km at long horizons AND z > 1.65), which I recommend to anyone doing small-n ML — it killed a seductive idea cleanly.
What I tried
- Blend ML with a physiology formula (threshold pace + per-distance penalty), weighted toward the formula at long horizons. Simulated on 22 races × 5 horizons: MAE improved 0.69 sec/km (needed ≥5), z=1.13 (needed >1.65) — rejected by the gate. Per-race noise drowns the horizon effect.
- Per-athlete residual bias correction
bias(d) = a + b·days_to_race, fit by walk-forward simulation over historical races. This one passed: MAE 55.0 → 50.0 (−9%), −6.65 sec/km at horizons ≥90d, z=2.63. Shipped, with population-constant fallback below 5 races. Helps — doesn't solve. - Oracle experiment — give the model the actual race-day context (surface, elevation, temperature) it can't know in advance: MAE drops to ~42.5. So ~7.5 sec/km of my remaining error lives in per-race context features, not in fitness modelling. Feature enrichment is planned but won't move the horizon problem.
- Cross-athlete pooling — the textbook answer to small n. Blocked in practice: I'd need 5+ athletes with long clean histories on my platform, and their training data needs noise-classification cleanup first (commutes, recovery spins, optical-HR garbage poison the pool worse than they poison a single-athlete model).
What I need
Is there a principled way to train a usable race-time model on ~20 races per athlete — hierarchical/Bayesian pooling against a public cohort, priors from published race datasets that actually transfer to age-group triathletes, or any technique you've seen work when per-sample noise is several times the effect size? Alternatively: a pointer to a public dataset pairing individual training load history with race results, at any scale. A definite "no, at this n you can only model the mean and the CI — stop trying" from someone who has been here is also a useful answer.