Why evaluation matters
Training is practice on the homework; evaluation asks whether the model can do the test: and whether the test is the real job: with CheckList as the door into how one accuracy number can hide systematic failures.
Primary source: Ribeiro, Wu, Guestrin & Singh, CheckList (ACL 2020): accuracy can hide systematic failures; behavioral tests catch what a single score misses: https://aclanthology.org/2020.acl-main.442/
Training tells you how well the model fit the homework. Evaluation tells you whether it can do the test, and whether the test is the real job.
That sounds obvious. In AI news it often isn’t. A demo score gets a headline. The harder questions, what was measured, on whose data, under which cost of being wrong?, get a shrug.
Homework vs the test
The practice loop is familiar from How a model learns: guess, measure error, nudge the weights, repeat. That loop is training. It answers: how well did the model fit the examples it was allowed to practice on?
Evaluation is the grade after practice. You put the model in front of examples it should not have memorized, score the answers against a rule you chose, and decide whether “good enough” is actually good enough for the job.
Nothing here is conscious. Nothing “knows it is being tested.” Evaluation is measurement design, same family as any other engineering check.
Held-out data (one-line gloss)
Held-out data means examples the model did not train on, the exam set kept aside so you aren’t grading the homework.
Honest caveat: “held out” is necessary but not magic. If the test pile is almost a twin of the training pile, same hospitals, same phrasings, same shortcuts, the model can look general while riding familiar cues. Leakage and near-duplicates turn an exam into a rehearsal.
One number can hide a failure mode
Accuracy, AUROC, “beats the baseline”, a single score is easy to quote. It can also average away the failures that matter.
Think of a sentiment model that looks strong overall, then flips when you add a simple negation (“not good”), or stays stuck when you change a name that shouldn’t matter (invariance). The average still looks fine. The behavior is broken.
That is the punchline of this piece’s primary source: CheckList, by Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh (ACL 2020). Inspired by behavioral testing in software, they argue that held-out accuracy often overestimates how well NLP models generalize. CheckList offers a matrix of linguistic capabilities and test types, plus tooling to generate many cases fast, so teams can probe behaviors a headline number can miss. In their studies, commercial and research models showed critical failures; practitioners with CheckList found more bugs than those without it.
Related door, different paper: Geirhos and coauthors on shortcut learning (Nature Machine Intelligence, 2020), decision rules that ace standard benchmarks but fail when the easy cue goes away. Same family of worry: high score, wrong reason.
How this ties to the Easy series
Evaluation is where the other Easy posts stop being abstract:
- How a model learns, memorizing vs generalizing. Evaluation is how you notice the difference.
- What training data means, shortcuts and watermarks. A good eval asks whether the model learned the disease or the hospital mark.
- What hallucination means, fluency ≠ truth. TruthfulQA is an eval designed to catch imitation of popular myths, not just smooth prose.
- Can AI read medical scans?, site shift, operating points, and what false positives cost per scan. A ranking score on one dataset is not a clinic.
Chatbots are one product shape on language data. Scan readers, spam filters, and ranking systems each need their own exams. “The AI scored 95%” without naming the test is marketing, not measurement.
What “good eval” looks like in plain words
No single recipe fits every product. These questions travel well:
- Right task? Are you scoring the job people will actually use, not a convenient proxy?
- Held-out sites and people? New hospitals, new accents, new time periods, not just a random shuffle of the same pile.
- Rare cases? Edge conditions and minorities in the data are where averages lie.
- Cost of mistakes? A false alarm on a worklist is not the same as a missed critical finding. Thresholds (operating points) encode that trade-off.
- Human review on high-stakes? Health, law, money, safety: treat model output as a draft with receipts, not a verdict.
Optional History door: The Deep Learning Turn, ImageNet as a public exam that made vision progress comparable. Benchmarks accelerate research; they also tempt shortcut learning when the exam stops matching the world.
Easy → technical doors
| If you’re curious about… | Easy / History next step | Technical door |
|---|---|---|
| Guess → error → update | How a model learns | Sources linked in that post |
| Curriculum, shortcuts, watermarks | What training data means | Datasheets for Datasets via that post |
| Fluency ≠ truth; myth-imitation evals | What hallucination means | TruthfulQA via that post |
| Site shift and scan operating points | Can AI read medical scans? | DeGrave et al. via that post |
| Behavioral tests beyond accuracy | This piece’s primary source | Ribeiro et al., CheckList (ACL 2020) |
| Shortcut learning as a wider lens | Geirhos et al. (related door) | Nat Mach Intell 2020 |
| Public exams that shaped vision | The Deep Learning Turn | ImageNet / AlexNet trail via History |
The honest caveat
Evaluation is not a vibe and not a vibe check on a demo. It is a designed exam: task, data, metrics, and who pays for each kind of error. One accuracy number can hide systematic failures; CheckList-style behavioral tests are one way to look underneath. Good reporting names the test. Bad marketing quotes the score and stops talking. On this site, Easy reading aims for the first kind of story, clear claims, clear limits, and a path to the paper when you want to verify.