Radiology MLLMs | Xi Zhang

🚨 Preprint out — Clinical Consensus Selection！

Fri, 29 May 2026 00:00:00 +0000

“CCS: Clinical Consensus Selection for Radiology Report Generation”. The preprint is now available.

For detailed model information and source code, please visit our Project Page: CCS

Overview

Radiology report generation (RRG) is usually treated as a single-path task: a multimodal large language model (MLLM) emits one decoded report and commits to it. Yet a fixed model often places clinically stronger reports elsewhere in its candidate pool than the one chosen by default decoding—so inference-time decision making remains an overlooked bottleneck. We introduce Clinical Consensus Selection (CCS), a decoder-agnostic, reference-free framework that samples multiple candidate reports and selects the one with the highest clinical consensus across the rollout pool. CCS unifies text-based utilities with a radiology-adapted utility from an image–report-trained multimodal embedder, measuring candidate agreement beyond surface-level text. Across three datasets and multiple radiology MLLMs, CCS consistently improves over single-path decoding and generic Best-of-N baselines, with particularly clear gains on clinical metrics.

Key Resources

GitHub Repository — Explore the CCS project on GitHub

CCS: Clinical Consensus Selection for Radiology Report Generation

Fri, 29 May 2026 00:00:00 +0000

For the latest updates and details, visit the CCS Project Website.

Why CCS? 🔬

Most radiology MLLMs commit to a single decoded report token by token, so one unfavourable step can omit a finding or assert one unsupported by the image, with no way to recover. Yet a fixed model often places clinically stronger reports elsewhere in its candidate pool — the bottleneck is not what the model can generate, but which candidate it commits to.

How CCS Works ⚙️

Clinical Consensus Selection (CCS) is a reference-free, decoder-agnostic inference-time framework that reframes RRG as candidate selection — no retraining or extra parameters.

Rollout sampling: draw N candidate reports from the radiology MLLM via stochastic decoding.
Textual consensus utility: repurpose report-evaluation metrics as reference-free pairwise agreement scores.
Image-grounded utility: measure candidate agreement with Qwen3-VL-Embed, a multimodal embedder adapted on image–report pairs, capturing clinical agreement beyond surface text.
Highest-consensus selection: return the candidate with the highest mean pairwise consensus over the pool.

Key Contributions ✨

Inference-time perspective: We show candidate pools routinely contain reports with higher clinical reliability and consistency than single-path outputs.
CCS framework: A decoder-agnostic Best-of-N method that aggregates pairwise clinical consensus using both textual and image–report-adapted multimodal utilities.
Consistent gains: Across three datasets and multiple radiology MLLMs, CCS improves RRG at inference time and identifies image-grounded utility as a distinct selection axis.

Main Results 📈

On MIMIC-CXR, CCS improves all radiology-specific metrics over Sampling (e.g. RadGraph-F1 0.1989 → 0.2134, CheXbert-F1⁵ 0.5041 → 0.5370), with gains statistically significant (p < 0.05). Unlike generic Best-of-N selectors (Self-Certainty, ModeX) that give inconsistent or negative clinical gains, CCS improves consistently across all backbones and datasets.

Analysis 📊

Per-label F1 shows CCS recovers abnormal findings that text-only consensus suppresses — confirming that image-grounded utility is a selection axis distinct from textual consensus, with substantial headroom still remaining.

Case Study 🔍

On a real MIMIC-CXR case, single-path decoding asserts factual errors (false “clear lungs”, mislocalised catheter), whereas CCS selects a more image-grounded report (CheXbert-F1⁵ 1.0000 vs Sampling 0.5000).

BibTeX 📚

@article{zhang2026ccsclinicalconsensusselection,
 title={CCS: Clinical Consensus Selection for Radiology Report Generation},
 author={Xi Zhang and Yingshu Li and Zaiqiao Meng and Jake Lever and Edmond S. L. Ho},
 year={2026},
 eprint={2605.30131},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2605.30131},
}

🎉 Paper Accepted — See You at ACL 2026!

Mon, 06 Apr 2026 00:00:00 +0000

“CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding” has been accepted to ACL 2026!

📍 ACL 2026 — The 64th Annual Meeting of the Association for Computational Linguistics will take place in San Diego, California, July 2026

See you there!

🪧 Poster

CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding

Mon, 06 Apr 2026 00:00:00 +0000

For the latest updates and details, visit the CCD Project Website.

Why CCD? 🔬

Radiology MLLMs remain vulnerable to prompt-induced hallucinations when clinical sections contain counterfactual details or ambiguous guidance. The figure above contrasts baseline predictions with CCD-enabled outputs across report generation and question answering tasks. Red highlights mark unsupported findings that can compromise patient care, while blue text reflects misleading prompt context the model must resist.

Clinical Contrastive Decoding (CCD) mitigates these risks at inference time. Rather than retraining models or relying on retrieval corpora, CCD injects trustworthy, image-grounded signals distilled from specialist expert models. The result is a decoding policy that maintains fluency yet stays faithful to the radiograph.

CCD at a Glance ⚙️

Clinical Contrastive Decoding (CCD) is a plug-and-play inference framework designed to reduce medical hallucinations in radiology MLLMs. It introduces structured clinical supervision from expert models (e.g., DenseNet or MedSigLIP, or others) at decoding time, without modifying model weights or requiring external retrieval.

Given a chest radiograph, the expert model predicts symptom-level probabilities across 14 CheXpert categories. CCD integrates this signal through a dual-stage logit refinement strategy:

Symptom-grounded Contrastive Decoding (SCD): constructs an anchor prompt using high-confidence findings (e.g., “Atelectasis, Cardiomegaly”) and generates a contrastive logits path conditioned on this prompt. The final logits are a weighted interpolation between the anchor-conditioned and original paths, encouraging the model to mention supported findings and suppress false negatives.
Expert-informed Contrastive Decoding (ECD): transforms expert probabilities into token-level logit biases via log-odds conversion. These biases are injected into the logits from the first stage, softly penalising unsupported findings and reducing false positives.

This two-stage mechanism enables CCD to progressively guide generation with both symbolic supervision (via anchor prompts) and probabilistic constraints (via confidence scores), achieving robust improvements in both report generation and VQA. CCD is fully model-agnostic and integrates seamlessly with state-of-the-art radiology MLLMs such as MAIRA-2, Libra, LLaVA-Rad, and LLaVA-Med.

Unlike prior contrastive decoding approaches that rely on perturbed visual or textual inputs, CCD leverages clinically grounded signals from expert models to provide task-specific and symptom-level control during generation.

Key Contributions ✨

Empirical Insight: We conduct a systematic analysis of prompt-induced hallucinations in radiology MLLMs, revealing that noisy clinical sections (such as irrelevant or contradictory clinical details) can trigger unsupported findings across multiple datasets.
Inference-time Framework: We introduce Clinical Contrastive Decoding (CCD) — a dual-stage inference-time strategy that leverages expert-derived labels as anchor prompts and performs probabilistic logit adjustments, requiring no retraining or architectural modifications.
Consistent Gains: Extensive experiments on MIMIC-CXR, IU-Xray, and CheXpert Plus demonstrate that CCD achieves up to +17% improvement in RadGraph-F1 with SOTA radiology MLLMs (MAIRA-2), along with higher VQA accuracy, all without altering model weights or structures.

Evaluation Highlights 📈

Achieves up to +17% RadGraph-F1 on MIMIC-CXR when attached to state-of-the-art report generators.
Reduces mention-level hallucinations on IU X-Ray while maintaining BLEU / ROUGE scores, confirming that CCD improves factuality without harming surface-level metrics.

Ablation Studies 🧪

Ablations show both contrastive stages matter: removing the anchor projection (SCD) or the logit sharpening (ECD) degrades factual metrics by ~5–8%.

My Musings ⏳

As I reflect on the development and impact of Clinical Contrastive Decoding (CCD),

It’s better to be roughly right than precisely wrong.
— Carveth Read, Logic: Deductive and Inductive

This quote perfectly captures the spirit of our approach. In the high-stakes field of radiology, the cost of hallucinations can be severe. By integrating expert-derived signals at inference time, CCD prioritises clinical accuracy over mere linguistic fluency. It’s a reminder that in medical AI, being approximately correct and grounded in reality is far more valuable than generating fluent but misleading text.

BibTeX 📚

@article{zhang2025ccd,
 title={CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding},
 author={Zhang, Xi and Meng, Zaiqiao and Lever, Jake and Ho, Edmond SL},
 journal={arXiv preprint arXiv:2509.23379},
 year={2025}
}