MedVTok unifies 2D images and 3D volumes into a shared visual token interface for clinical AI

Ma et al. present MedVTok, a unified medical visual tokenizer pretrained on 28 million images and 70,000 volumes across nine modalities, using slice differential regularization to retain anatomical coherence.

Primary source: arXiv:2610.35842: https://arxiv.org/abs/2610.35842

What’s new: Modern multimodal medical AI systems are hindered by representational fragmentation: different clinical modalities (such as X-rays, ultrasound, CT, MRI, and digital pathology) and spatial formats (2D slices versus 3D volumetric scans) typically require custom, incompatible vision backbones. In MedVTok: A General-Purpose Medical Visual Tokenizer (arXiv:2610.35842, 26 Oct 2026), Chenglong Ma, Yuanfeng Ji, Junzhi Ning, Jiyao Liu, Yujie Wei, and collaborators present MedVTok, a unified visual tokenizer that encodes both 2D images and 3D volumetric scans across diverse modalities into a single discrete token space.

To overcome the challenges of preserving continuous anatomical structure and capturing diverse clinical semantics across modalities, MedVTok introduces two core methodological components: slice differential regularization and multi-expert representation alignment. Slice differential regularization explicitly models inter-slice gradient continuity, correcting the anatomical incoherence that arises from standard independent per-pixel or per-voxel reconstruction losses. Multi-expert representation alignment distills representations from specialized domain-specific encoders (such as pathology, radiology, and ultrasound models) into a unified codebook while preserving complementary modality-specific details. Pretrained on an extensive dataset of 28 million images and 70,000 3D volumes spanning nine distinct imaging modalities, MedVTok serves as a drop-in token interface for downstream medical foundation models. Across 24 comprehensive benchmarks spanning classification, cross-modal retrieval, semantic segmentation, image generation, visual question answering, and report drafting, MedVTok establishes new state-of-the-art results for unified medical tokenizers.

Why it matters: In clinical practice, patient evaluations frequently require correlating heterogeneous imaging sources, such as comparing 2D mammograms or chest radiographs with 3D abdominal CT scans or brain MRIs. Fragmented architectures require distinct tokenizers and complex cross-attention bridges for each modality, drastically increasing model parameters and engineering overhead. By establishing a shared discrete visual vocabulary across both 2D and 3D medical data, MedVTok simplifies multimodal foundation model architectures and enables unified generative and discriminative modeling across clinical workflows.

Caveats: Pretraining MedVTok demands substantial computational resources across vast multi-institutional image datasets. Moreover, while unified tokenization simplifies downstream model ingestion, dense 3D volumetric token sequences can still result in substantial sequence lengths, requiring downstream LLMs or transformers to implement hierarchical compression or sparse attention for long scans.