MedBenchAgent: compiling medical VLM benchmarks from annotations, not just generating questions

A multi-agent framework derives the benchmark specification itself, reaching 90.9% Task-Space F1 and 994 of 1,000 sampled items passing human audit.

Primary source: arXiv:2610.11312: https://arxiv.org/abs/2610.11312

What’s new: Most automated benchmark construction generates evaluation items inside a specification a human has already written. MedBenchAgent: Towards Systematic Automation of Medical VLM Benchmark Construction (arXiv:2610.11312, 8 Oct 2026) tackles the earlier step: deriving what to evaluate, which annotations support each task, and how to turn that evidence into reliable items. The authors frame this as constrained compilation, where the specification is progressively derived from evaluation requirements, heterogeneous annotations, and medical knowledge.

The multi-agent system uses a Benchmark Intermediate Representation (BIR) to encode task definitions, evidence mappings, evaluation protocols, and item specifications across stages. Planning, which derives and verifies the specification, is separated from instantiation, which builds and audits items under a locked specification. MedBenchAgent reaches a Task-Space F1 of 90.9%, against 79.2 to 80.0% for direct task induction and 85.1% for prior-guided induction. Of 1,000 sampled items from correctly identified tasks, 994 pass human audit. The authors also show portability to a specialized medical domain and evaluate twelve VLMs, finding task- and setting-specific variation that aggregate scores obscure.

Why it matters: Medical VLM benchmarks are expensive to build and quickly saturate. Making the specification explicit and auditable gives a path to refreshing benchmarks as annotated imaging datasets grow, and the per-task results are a reminder that a single aggregate score can mask where a model fails.

Caveats: The pipeline depends on the quality and coverage of the underlying annotations, and on LLMs inside the planning and audit loop. The 994 of 1,000 audit figure covers items from correctly identified tasks only, so errors in task identification are counted separately. Generated benchmarks still need independent clinical validation before they are used to judge deployment readiness.