MedHEB: Benchmarking Medical Embeddings Across Heterogeneous Clinical Evidence
Motivation
Clinicians rarely read a scan on its own: they compare it with prior studies, other modalities, reports and localized findings, so retrieving the right evidence is central to interpretation. Foundation models now encode all of these sources into shared embedding spaces, but existing benchmarks evaluate one regime at a time — biomedical text, image–text, or 3D volumes. It remains unclear whether a single embedding space still places clinically relevant evidence near the query when the modality, the relation or the granularity changes.
The MedHEB Benchmark
MedHEB fixes one ranking protocol (embed the query and the candidate pool, rank by cosine similarity) and varies what counts as relevant along three axes:
- Evidence form: 2D images, 3D volumes and clinical text.
- Pairing type: image–image, image–text, text–text, and modality-bridging visual retrieval (e.g. finding the CT series that matches a chest X-ray).
- Relevance granularity: global label, question-conditioned, and region-level (locating the image region that matches a text description).
In total: 107 tasks from 54 datasets, 75,233 queries, 8 anatomical domains and 8 retrieval families, with both in-distribution and held-out-source splits.
A: Task composition by evidence form, retrieval family and source dataset. B: Representative cases, from modality-bridging and region-level retrieval to VQA and biomedical text retrieval.
MedEmb: A Reference Baseline
MedEmb adapts Qwen3-VL-Embedding (2B / 8B) with LoRA on the MedHEB in-distribution split. Text, 2D images, sampled 3D volumes and interleaved image–text inputs all go through one MLLM, and the final-token hidden state is the embedding — no task-specific fusion modules. Training combines:
- Instruction-conditioned contrastive learning over query–target pairs.
- False-negative masking, so clinically equivalent in-batch candidates (e.g. another normal chest X-ray) are not pushed away.
- Token-budgeted dynamic batching for inputs that range from short labels to long reports and volumetric studies.
MedEmb serializes text, 2D images, 3D volumes and mixed inputs into one MLLM and uses the final-token state as the retrieval embedding.
Key Findings
We evaluate eight public embedding models — dual-tower encoders (SigLIP, BLIP-2, MedSigLIP, MedImageInsight) and MLLM-based models (VLM2Vec-V2.0, RzenEmbed, Qwen3-VL-Embed-2B/8B) — alongside MedEmb.
- Neither saturated nor unsolved. By public-baseline Hit@1, 15 tasks collapse (<3%), 24 are weak, 45 medium and 23 tractable; the spread between models grows from SD 1.9 to 17.1 across these buckets, so MedHEB separates models wherever signal exists.
- Strong on familiar tasks does not mean strong on new relations. Every model drops 7.7–26.6 Hit@1 from established families (image–image, image–text, text–text) to modality-bridging and region-level retrieval.
- 3D and subtle findings are the bottleneck. Ophthalmology, dermatology and microscopy are more tractable; thoracic, musculoskeletal, neuro and multi-organ tasks, and 3D volumes in general, remain hard.
- MedEmb is the most balanced. MedEmb-8B reaches 46.1 mean family-level Hit@1 (best public model: 25.3) with the lowest cross-family variation (CV 0.34 vs. ≥0.65 for every public baseline).
A: Model performance across difficulty regimes. B: Established retrieval families (upper bar) vs. underexplored clinical-relation families (lower bar).
Retrieval difficulty across anatomical domains (A) and input modalities (B).
BibTeX
@misc{li2026medhebbenchmarkingmedicalembeddings,
title={MedHEB: Benchmarking Medical Embeddings Across Heterogeneous Clinical Evidence},
author={Yingshu Li and Shaoyang Zhou and Zhanyu Wang and Yunyi Liu and Xinyu Liang and Xi Zhang and Lingqiao Liu and Lei Wang and Luping Zhou},
year={2026},
eprint={2611.02519},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2611.02519},
}
