Fourteen Empty Strings

The first time I ran a new evaluation suite against my assistant's synthesis prompt, it produced a clean, plausible, entirely believable table. Every case scored near zero. The obvious reading was that the prompt I had just written was a disaster.

It wasn't. The model had returned an empty string fourteen times in a row. A reasoning model with reasoning enabled, and a client that only ever read message.content, never falling back to the reasoning field. The judge dutifully graded fourteen empty strings and gave them the score empty strings deserve.

I only caught it because the judge left a rationale, and the rationale said "Jared gave no response." Without that one line of text I would have spent a day rewriting a prompt that was working.

A zero that means "we couldn't measure" is indistinguishable from a zero that means "it failed."

That is the whole problem in one sentence, and almost everything I've since learned about evaluating LLM systems is a variation on it. Below are four lessons that cost me something, and then the part where I turn the same lens on my own work and find it wanting in eleven places.

Context, briefly: Jared is a personal AI assistant I've been building for about a year. It's a local-first multi-agent system running on Docker, with long-term memory, a proactive daemon, and a fairly opinionated personality. I'm the architect, the operator, and the only person who ever reviews its output. That last fact is what makes evals load-bearing rather than academic.

---

01 · A judge cannot share the blind spot it is checking for

The natural instinct is to grade a model's output with the model you already have configured. It is one line of code and zero new credentials. It is also close to worthless when the thing you're testing for is a family trait.

I was testing whether the assistant had stopped being sycophantic: whether it would push back on weak reasoning instead of validating it. If I ask an agreeable model whether another agreeable model was too agreeable, it says no. It genuinely does not see the problem, because it has the same problem.

So the rule became architectural rather than advisory: the judge must come from a different vendor than the model it grades, and the harness refuses to start if they match. Not a comment. Not a convention someone tidies away in six months. An assertion at startup that aborts the run.

02 · Validate the judge before you trust a single number it gives you

An LLM judge is just another model with another prompt, which means it can be wrong in all the ways your system can be wrong. If you don't check it, you're not measuring your prompt. You're measuring the product of two unverified things and attributing the result entirely to one of them.

The fix is unglamorous. Before the judge grades anything real, it grades a handful of cases whose correct verdict I already know: an obviously sycophantic answer that must score low, an obviously good answer that must score high. If it agrees with fewer than 85% of those, the run aborts, and reports that the judge is broken, not that the prompt is.

This matters more than it sounds, because a miscalibrated judge doesn't fail loudly. It produces confident, well-formatted, completely wrong numbers, and you act on them.

03 · A rubric that only rewards pushback will train a contrarian

My first version of the rubric had an axis for "did it challenge weak reasoning." Scored that way, more friction is always better, and if you optimise against it long enough you get an assistant that invents objections to sound arguments so it can feel rigorous.

That is not a fixed version of sycophancy. It's a second failure wearing the first one's clothes. Both fail the same test: did this help me see more clearly, or did it just make me feel a certain way?

So the axis became bidirectional. Manufactured disagreement scores exactly as badly as unearned agreement. And because a rubric alone won't hold that line, the dataset enforces it structurally: cases where the correct answer is plain agreement, and pushing back is the failure, must stay at least a quarter of the set, and a unit test fails the build if they drop below it.

Without those cases the eval doesn't just get weaker. It silently inverts into a contrarianism trainer, rewarding the exact behaviour it was built to prevent.

04 · Never record a failure to measure as a bad measurement

This is the fourteen empty strings, generalised, and it's the one I'd keep if I could only keep one.

Every part of an eval pipeline can fail for reasons that have nothing to do with quality. The provider rate-limits. The judge times out. A model name resolves to nothing. The tempting thing (the thing almost every codebase does by default) is to catch the exception, score it zero, and move on, because that keeps the run green and the pipeline simple.

Do that and you've built a system that reports your best prompt as broken and, worse, can mask a genuinely broken one under the same number. So: errors raise. A missing verdict is an exception, not a score. An empty response from the model under test is an exception, not a zero.

The same principle scales up. A monitoring job that returns {} on failure and gets read as "all metrics are zero" sent me a nightly regression alert for weeks, blaming prompt edits and model changes, when the actual cause was a 30-second client timeout on a job that takes 400 seconds. The eval had been scoring a perfect 1.00 the entire time.

---

Then I pointed it at myself

Lessons are cheap to write and easy to believe you've already applied. So I audited my own evaluation layer against them, with the rule that every finding had to carry evidence: read in the code, observed on the running stack, or computed from stored run artifacts. It came back with eleven. These are the ones that generalise.

The only path that runs regularly was the only one skipping calibration. Lesson 02, implemented, and wired into the command-line entry point rather than the scheduled one. The nightly job, the one producing every number I actually look at, had been trusting an unvalidated judge the whole time.

My alert thresholds sit below my noise floor. I ran the same suite three times with nothing changed, to see how much the scores move on their own. Enough to trip the alerts. Chart below.

The detector for unknown failures is switched off. Everything running is a drift detector over fixed inputs. Good at catching a known case getting worse, blind to a new category of wrong. I'd built sampling of real production traffic to close exactly that gap. The enabling flag is set in neither the env file nor the compose file, so it has never once run.

A skipped run is invisible. The nightly eval sits behind a low-credit gate and a debounce it isn't registered with. Either can silently suppress it, and nothing anywhere tracks "no score has been produced in N days." Absence of measurement should be an event.

Coverage went where it was easy, not where the incidents were. Four of roughly twelve places the system makes an LLM judgement call are measured. The single worst-incident surface has none: the planner that picks tools and fills their parameters, with three separate production bugs to its name. It also needs no judge at all: asserting the right tool with the right parameters is deterministic and free.

The noise floor problem, in one chart

This one surprised me most, and I suspect it's the most widely applicable. Three identical runs of the same suite, same config, same evening. The spread between them is pure noise. It has to be, because nothing changed. The amber line is where my alerting fires.

Two things fall out. A category with one test case isn't a measurement. Mine scored 1.00, 0.90 and 0.00 across three runs of the same thing. And my headline suite alerts if the pass rate is anything under 100%, which with thirteen cases means one flaky case is an alert.

If you're running LLM evals and haven't measured your own noise floor, you probably don't know whether your last regression was real. It's three extra runs. Do it before you tune anything.

And one that was live while I wrote this

The suite fired a genuine regression the night before I finished the audit. A fixture written to catch a specific old bug, the assistant storing its own name as the user's name, caught it again:

2026-08-22 00:12  clean      deterministic=100%
2026-08-23 00:53  REGRESSED  deterministic=92%

FAIL  addressed_by_name   ! contains forbidden content 'name is jared'

No relevant code had changed between those two runs. So it's either the provider quietly changing what answers to a model name, or ordinary nondeterminism in the extractor. I can't yet tell which, because the suite runs once a night and never repeats a case. Which is its own finding.

I'm leaving it in rather than fixing it first and writing a tidier post. An eval layer that has never caught you out isn't mature, it's decorative.

---

The thing I'd change first

The reason my coverage stalled at four suites isn't discipline. It's that each one costs roughly five hundred lines. Every suite reimplements dataset loading, seeding, calibration, scoring and reporting. About three thousand lines of harness for four datasets.

So the principle I'm rebuilding around: an eval layer should scale by adding datasets, not by adding machinery. One shared harness, and a new suite becomes a dataset, a rubric, and a single adapter function. Sixty lines instead of five hundred. Coverage stops being an engineering project and starts being a writing task, which is the only version of it that ever gets done.