One of the first cards I caught NextLang getting wrong was escalate. The deck was English to Turkish, level C1, topic "verbs of motion you need at an airport", and the back of the card said yürüyen merdivenle çıkmak: to go up by escalator. It is a perfectly reasonable mistake for a model that saw "airport" and "escalate" in the same prompt. It is also a mistake the person studying that deck cannot catch, because they are studying it precisely since they do not know the word yet. A wrong flashcard does not look wrong. It looks like a flashcard, and the learner memorizes it.
That is the problem every AI product that generates content for a non-expert has, and the honest answer to "how do you know the output is good?" is usually "I looked at a few". This post is about what I built instead, and about the part that surprised me most: the tools that grade the model need grading too.
Two failures that look nothing alike
A generated deck can fail in two ways, and they have almost nothing in common.
The first is mechanical. The file does not import: a translation with a comma in it splits into two columns, a tag lands in the wrong field, the deck has 28 cards when you asked for 30, an Anki cloze card has no {{c1::}} marker and turns into a blank. These failures are binary, cheap to detect with plain code, and they cost the user a credit, which NextLang refunds automatically.
The second is semantic. The file imports fine and the card is wrong: the wrong sense of a word, an unnatural phrase no native speaker would say, an example sentence that does not match the level. No parser catches that, and no refund fires, because nothing looks broken.
For a long time I treated these as one problem, and it cost me. Every prompt tweak that improved meaning was judged by whether the deck imported, and every formatting fix was judged by reading cards. Separating them was the first real decision: mechanics get a test matrix, meaning gets a judge.
The matrix measures mechanics, not quality
The matrix is a set of generation requests that covers the combinations that matter: four platforms, card types, separators, quoting, headers, levels, card counts. Each cell is a real call to the generator, graded by plain code on separators, tags, empty fields, card count and which language ended up in which field.
The run that justified it was the switch from asking the model to write CSV to asking it for plain JSON fields and building the file in code. Same 267 cells before and after:
- clean cells went from 234 to 260
- separator correctness from 0.946 to 1.000
- cells where whole rows were discarded for having the wrong number of columns, from 5 to 0
That is a result I could not have seen by reading cards. It is also a result that says nothing about whether a single translation was right.
Languages got their own run on purpose. The matrix tells me if the deck breaks; it does not tell me if Bengali works. So there is a separate sweep where everything except the language is pinned (Anki, basic cards, tab separator, 10 cards, B1) and the language pair is the only variable: 19 pairs, each run twice, including every pair with no English on either side. First run, 38 out of 38 clean.
Clean, again, meaning "it imports". Not "it is right".
A judge has to pass an exam before it grades anything
For meaning, the only scalable reader is another model. I sample cards from a finished run and ask a stronger model to grade each one on five things: meaning, naturalness, relevance to the deck's topic, fit for the CEFR level, and whether the card is answerable from its front.
Two design choices did most of the work.
The judge translates in the reverse direction. It reads the back first and only then sees how it compares to the front. If it read the front first, it would make the same association the generator made: escalate plus "airport" pulls both of them toward the escalator.
The sample is picked, not random. About 105 cards, weighted toward where things go wrong: every non-English pair, distant languages, advanced levels, and a deliberately nasty C1 Turkish cell that I include every time. Easy pairs at A1 almost never fail, so they mostly tell you nothing when you change a model.
But the decision I would repeat on any project is the calibration gate. Before the judge sees a single real card, it grades six cards whose correct verdict I already know: the escalate card and a wrong-sense alight it must catch, a Spanish-Portuguese cognate (comer / comer) it must not flag, and clean cards it must let through. If it misses even one, the run stops.
const calibration = await inBatches(CALIBRATION, async (item) => {
const verdict = await judgeCard(item);
return { item, verdict, passed: verdict !== null && item.expect(verdict) };
});
if (!calibration.every(({ passed }) => passed)) {
await saveText(dir, REPORT_FILE, buildSemanticReport(stamp, calibration, []));
return;
}
The reason is simple. A judge that cannot spot an error I already know about produces a number, and the number looks exactly as authoritative as a real one. The calibration costs about a cent and turns "the judge said 64 clean" from a vibe into a measurement.
The first real run passed calibration 6 out of 6 and flagged 28 cards as questionable on answerability, all of them bare words with several meanings: board, return, approach. That looked like 28 defects. It was one finding: a single English word on the front of a card is ambiguous, and the deck needs a hint of which sense is meant. Reading the cluster as one issue instead of 28 was the difference between rewriting the generator and changing one rule.
The second judge exists to check the first
The generator is an OpenAI model. So was the judge. Two models from the same family can share blind spots in exactly the same translations, and the judge would then confirm the generator's mistake with full confidence.
So the same calibration, the same 105 cards and the same rubric also run through Claude. It costs about half a dollar for the whole sample. The first side-by-side:
- OpenAI's judge called 57 cards fully clean, Claude 45
- 40 cards were flagged by both
- on hard meaning errors, they agreed on exactly one:
shuffletranslated into Japanese as "walk slowly", instead of dragging your feet
The interesting part was where they disagreed. OpenAI's judge marked three more cards as wrong in meaning, and all three were correct translations: despertarse as acordar and llamar as ligar in Brazilian Portuguese, and dash into Turkish. Claude graded them fine. Claude, in turn, caught a circular Bengali definition of "value for money" that the other judge let through.
The rule that came out of it: a meaning error that both judges report is real, and a single judge calling something wrong gets read by a human before anything changes. A single judge, it turned out, was wrong about a correct card more often than I expected.
The graders lie too
The same lesson applied one level down. One of the plain-code graders flagged cards where the front and back were identical, on the theory that the model had given up and echoed the word. Across 343 cells it fired five times and was wrong all five: oven and warm in English and Dutch, compact in English and French, descongelar in Spanish and Portuguese. Related languages share words. One identical card is normal; a model that gave up produces many.
The grader now fires only when identical cards reach at least 3 and at least a quarter of the deck. And because a grader change shifts every score, I re-graded every stored run with the fixed version, so the before and after numbers in old comparisons still compare the same thing.
What the evals are actually for: spending money on the right thing
All of this sounds like quality work, and it is, but its most practical use has been model decisions.
When a cheaper model generation came out, the cheap tier dropped to about half the price per token. The eval said: parity on card quality, the language sweep untouched, but a real regression on cloze cards, where the new model drops the {{c1::}} marker more often. That told me where to watch refunds after the switch instead of finding out from users.
It is also where the one routing rule in NextLang comes from. Pairs with no English on either side go to the more expensive model, because early evals kept finding more translation errors there. The rule is crude. Spanish to Portuguese probably does not need the expensive tier, Bengali to Japanese definitely does, and both get the same treatment. In the latest run the cheap model everywhere was not even worse on those pairs, but the sample is small and the saving is pennies, so I keep paying. A wrong translation shipped to someone who cannot spot it costs more than the difference.
What it taught me, as the engineer and as the owner
As an engineer: an eval is only as honest as its least-checked part. The generator gets tested by the matrix, the matrix gets read by graders, the graders get re-run when they are wrong, and the judge has to pass an exam before its numbers count. Every layer I skipped checking turned out to have a bug in it, and none of those bugs announced themselves. They just produced plausible numbers.
As the owner: the evals changed which arguments I take seriously, including my own. "The new model feels better" and "this prompt reads cleaner" stopped being reasons to ship. The question became what the matrix, the sweep and both judges say, and when they disagree, which one has been wrong before. It made me slower to change things and much faster to undo a bad change.
If you are building something similar and want to compare notes on judges, calibration or what to sample, LinkedIn (opens in a new tab) is the fastest route.