PrecepTron: scaling physician judgment to grade medical AI

An eval methods brief: LoRA-tuned PrecepTron plus GRAND-ROUNDS (9,217 physician scores) aim for physician-level grading of open-ended clinical LLM answers, including piecemeal case presentation.

Primary source: arXiv:2609.12822: https://arxiv.org/abs/2609.12822

What’s new: Blinded physician grading is often treated as the gold standard for clinical reasoning in LLMs, but small, single-site panels do not scale and make reproduction hard. PrecepTron (arXiv:2609.12822, submitted 11 Sep 2026) is an LLM fine-tuned for physician-level evaluation of open-ended responses: LoRA on a 32-billion-parameter model, trained on a small number of physician examples.

The authors also release GRAND-ROUNDS, a physician-annotated benchmark of 9,217 scores from 11 physicians across seven studies. They report that typical frontier “LLM-as-a-judge” setups often disagree with physicians and with each other, while fine-tuned PrecepTron reaches physician-level consistent scoring across tasks. They use it to reproduce headline findings from five influential clinical-LLM studies in JAMA, Science, and Nature Medicine without new human grading, then ask questions that would be expensive with humans alone, including diagnostic accuracy when cases arrive piecemeal, even token by token. Code, data, and labels are released.

Why it matters: Medical AI eval is stuck between tiny expert panels and generic LLM judges that do not track clinicians. A reusable grader plus a large shared score corpus is the infrastructure move: reproduce published claims, stress-test information drip, and run studies that human bandwidth could not fund. That is Bitware’s eval lane: how we know what models do in medicine, not another chat demo.

Caveats: Preprint. “Physician-level” consistency is relative to this 11-physician, seven-study pool, not every specialty or health system. LoRA on few examples can inherit those physicians’ priors and miss out-of-distribution failure modes. Reproducing journal headlines without new humans is powerful and also circular if PrecepTron’s training distribution overlaps the studies being checked: read the paper’s holdout and disagreement analyses before treating it as a drop-in IRB replacement. Piecemeal/token-by-token probes are synthetic stress tests, not bedside workflows. Use PrecepTron and GRAND-ROUNDS as shared eval tooling with known rater limits.