Tuning an AI running coach

Posted 2026-07-17

AudioRun has a live AI coach feature that aims to provide helpful information throughout the run. It takes in information such as pace, heart rate, elevation change and user defined goals and pace targets. This information is fed into an LLM which generates coaching advice and outputs it as speech according to predefined triggers. Doesn’t sound too hard, but the specific constraints of a running app made this a challenge:

• Latency matters. As a runner, you’re supposed to get near-live information on your run, which you can then choose to act on (or not). To avoid delivering outdated information I’ve set a hard limit of a 15 second timeout. If the coaching message (for some reason) isn’t ready by then, it’s discarded.
• Replies are spoken, and should deliver information dense, short messages and avoid the usual long-winded responses that are typical for many LLMs. Thirty words max.
• Numbers must be spoken as words. “5:30/km” comes out of a speech engine as “five colon thirty slash kay emm”, which is no good.
• The model only knows what telemetry shows. It does not know the route. Many LLMs love to inject assertions like “The hill is almost over” when in reality it has no idea and no way of knowing the nature of the hill.

Due to the huge number of possible running scenarios, combined with the various user-defined parameters, it is not realistic to do manual e2e testing that has any hope of being exhaustive. Even a short 45-minute run typically yields a dozen coaching events.

The solution I use to evaluate the AI coaching feature is to simulate a variety of different runs using spoofed GPS data (using the Lockito app) and health metrics such as heart rate. The harness replays scripted scenarios, including segments with varying pace, heart rate, and elevation and feeds this into the actual production code: the same trigger evaluator, prompt builder, request/parse/timeout logic as the production app. This harness was created using Claude Code with Fable 5 (xhigh). Altogether we simulate nine different scenarios, including a 10k and an indoor treadmill run.

We measure how timely the responses come in, check for parse failures, word counts, digit and pace-notation leaks, repeated advice types and latency percentiles. Due to the sheer quantity of information provided by the simulations we hand off the first round of quality evaluation to Claude Opus 4.8. Claude reads the coach’s system prompt, and, for every delivered reply, the exact context prompts the coach saw. Claude can then verify coaching claims against the real numbers. Each reply gets a score based on the following parameters: was the message triggered correctly, was it factually grounded and did the user specified AI-persona match. An evaluation of the overall results then ranks the models.

I used this harness to evaluate several cloud models (using OpenRouter) such as Claude Haiku as well as self-hosted models like Qwen and Gemma. I was particularly interested in finding out if my local hardware was fast enough to deliver responses within the 15 second cutoff, and at what level of quality.

Hardware and software

The local self-hosted models ran on a computer with a Ryzen 5700X3D, 32 GB RAM and an RTX 5080 GPU. I used llama.cpp as the inference engine and used Tailscale to make it reachable from the app.

What I found

The first evaluation of the local self-hosted models (Gemma 4 12B, Qwen3.5 9B) was a disappointment. While they beat the cloud models on latency (p50 0.6–0.9 s versus ~1.8 s) and perfect 88/88 responses delivered, judge Opus disqualified both anyway. One spent ten-plus minutes watching the runner operate at critical heart rate while offering nothing but reassurance, instead of telling the runner to ease back. Both fabricated goals and training history in post-run debriefs. So, while the speed and latency were more than good enough, the quality was simply not there. But I did suspect that the system prompt and the format of the provided information was to blame. Sure enough, Opus identified reproducible failures in the prompts.

The tuning loop

Five edit→validate iterations plus a confirmation run. Each cycle: mine the newest report’s judge notes and exact context prompts, edit the prompt builders, run the offline guardrail tests (~20 s), fire a full judged run (~5 min), diff the flagged scenarios against both baselines.

What we learned:

Fabrication usually means missing data, not weak instructions. The overshoot problem — coach praising a precariously fast run — survived every instruction rewrite until it turned out the debrief prompt never stated the target pace. The model couldn’t see the 5:40-vs-6:00 gap without doing arithmetic it’s forbidden from doing. One precomputed “pacing vs plan” line fixed what three rules couldn’t.

Point-of-use context beats distant rules. System-prompt rules got ~80% compliance under repeated trigger fire. Every fix that stuck was data or a short note injected next to the stats it governs.

Ground the entailed fact instead of banning the phrasing. Banning the word “cushion” (time banked by running fast) just made the judge lint for the word while the model kept producing the frame. Stating the entailed truth in the context — the pace needed to finish exactly on target — made the claim checkable and the phrasing harmless.

Prompt fixes regress neighbors. An innocuous “splits a few seconds apart is even pacing” clause handed the model a praise label that broke the redline scenario’s debrief — “real pacing discipline” for a run spent at maximum heart rate — two runs straight, until reworded to be purely descriptive. Only the full scenario matrix caught it.

Results and open costs

Fabricated figures: gone in all six runs. PR moments: scored 5/5 in all six. Indoor pace leaks: gone. Overshoot debriefs: honest five runs straight. Ungrounded-reply count: 4 versus baselines of 5 and 9.

Takeaways

• Simulate the world; keep the production pipeline real. Byte-identical prompts or you’re testing something else.
• Score mechanics in code. Spend the judge on substance only.
• Give the judge the exact context the model saw, or it grades vibes.
• Expect score deflation after real fixes. Diff behaviors, not averages.
• Models fabricate where data is missing. Precompute the fact and put it next to the stats it governs.
• Evaluate per scenario. Aggregates hide regressions and errors.

← All posts