PoolAssistant

Rating is a Bradley-Terry strength on the Elo scale (mean 1000) computed from every head-to-head on the same question — it weighs who a model beat, not just how often, so beating the strongest model once counts more than beating the weakest five times. The ± is a 95% interval from bootstrapping the comparisons; a model shows a firm number only once it has enough of them (≥5, against ≥2 opponents), otherwise it reads directional. Both signals feed it: a Best pick (winner beats the rest) and stars within a turn (5-vs-3 is a comparison; a full set of stars is a full ordering, the richer signal — each turn counts once). Stars also stand alone as an absolute score, the only judgement a single-model turn can give. Value is that rating minus a cost penalty (400 Elo per 10× cost over the cheapest rated model), with a ★ on the picks nothing beats on both quality and price; By category re-rates each model over just the questions of one auto-tagged kind.

…

Admins always see this detail. Turn on to reveal it to regular members too.

Approving queues it for the local runner. Naming the layer that failed is the point: it’s what stops every fix being crammed into the prompt.

Ranked by weakest coverage: a member asked this and the nearest passage was far (or nothing matched). This is the backlog of references worth writing — driven by what members actually ask, not by guessing what's missing.

A build appears here the first time it boots, because that is the only moment a deploy can be recorded: there is no CI, and merging to main changes nothing until someone builds the image and redeploys it. Under each build: the commits that came with it, and the triage items the loop finished while that build was live — their fixes shipped in a later build, not this one.

The alias table is 21 hand-written entries and is still the cheapest fix for the biggest class of retrieval miss — “stabilizer” where every reference says “cyanuric acid”. This asks the corpus and the retrieval fixtures what the 22nd should be, from three signals: query terms the winning article never says, abbreviations the corpus glosses itself, and “also called …” in curated prose. Run here rather than from a checkout, it adds a fourth the corpus can’t give: the words members actually typed that no reference uses, as terms and counts. It writes a report and changes nothing — an alias fires on every future query and a wrong one skews ranking corpus-wide, so accepting one is a human decision.

Score the retrieval suite twice — BM25 alone, then the same candidates reordered by the cross-encoder — and report hit@1, hit@3 and MRR for both arms plus the queries that actually crossed the top-3 line in each direction. A summary statistic hides both the payoff and the cost; the named queries don’t. This one spends money at the rerank vendor, so size it with a dry run first: that builds every request, sends none, and needs no key.

The third job from this work — --gen-context, which writes one situating sentence per passage — has no button here and should not get one. Its output is sidecar files under skill/references/.context/ that belong in git: both retrieval legs read them, so they have to ship in the image, which means they have to be committed. A server generating them would be doing ~1,800 model calls of work that nobody could commit and that the next deploy would throw away. Run it from a checkout: hsecret run OPENROUTER_API_KEY -- dotnet run --project src/Assistant.Evals -- --gen-context, review the diff, commit the sidecars.