Publications | Xi Zhang

CCS: Clinical Consensus Selection for Radiology Report Generation

Fri, 29 May 2026 00:00:00 +0000

For the latest updates and details, visit the CCS Project Website.

Why CCS? 🔬

Most radiology MLLMs commit to a single decoded report token by token, so one unfavourable step can omit a finding or assert one unsupported by the image, with no way to recover. Yet a fixed model often places clinically stronger reports elsewhere in its candidate pool — the bottleneck is not what the model can generate, but which candidate it commits to.

How CCS Works ⚙️

Clinical Consensus Selection (CCS) is a reference-free, decoder-agnostic inference-time framework that reframes RRG as candidate selection — no retraining or extra parameters.

Rollout sampling: draw N candidate reports from the radiology MLLM via stochastic decoding.
Textual consensus utility: repurpose report-evaluation metrics as reference-free pairwise agreement scores.
Image-grounded utility: measure candidate agreement with Qwen3-VL-Embed, a multimodal embedder adapted on image–report pairs, capturing clinical agreement beyond surface text.
Highest-consensus selection: return the candidate with the highest mean pairwise consensus over the pool.

Key Contributions ✨

Inference-time perspective: We show candidate pools routinely contain reports with higher clinical reliability and consistency than single-path outputs.
CCS framework: A decoder-agnostic Best-of-N method that aggregates pairwise clinical consensus using both textual and image–report-adapted multimodal utilities.
Consistent gains: Across three datasets and multiple radiology MLLMs, CCS improves RRG at inference time and identifies image-grounded utility as a distinct selection axis.

Main Results 📈

On MIMIC-CXR, CCS improves all radiology-specific metrics over Sampling (e.g. RadGraph-F1 0.1989 → 0.2134, CheXbert-F1⁵ 0.5041 → 0.5370), with gains statistically significant (p < 0.05). Unlike generic Best-of-N selectors (Self-Certainty, ModeX) that give inconsistent or negative clinical gains, CCS improves consistently across all backbones and datasets.

Analysis 📊

Per-label F1 shows CCS recovers abnormal findings that text-only consensus suppresses — confirming that image-grounded utility is a selection axis distinct from textual consensus, with substantial headroom still remaining.

Case Study 🔍

On a real MIMIC-CXR case, single-path decoding asserts factual errors (false “clear lungs”, mislocalised catheter), whereas CCS selects a more image-grounded report (CheXbert-F1⁵ 1.0000 vs Sampling 0.5000).

BibTeX 📚

@article{zhang2026ccsclinicalconsensusselection,
 title={CCS: Clinical Consensus Selection for Radiology Report Generation},
 author={Xi Zhang and Yingshu Li and Zaiqiao Meng and Jake Lever and Edmond S. L. Ho},
 year={2026},
 eprint={2605.30131},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2605.30131},
}

CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding

Mon, 06 Apr 2026 00:00:00 +0000

For the latest updates and details, visit the CCD Project Website.

Why CCD? 🔬

Radiology MLLMs remain vulnerable to prompt-induced hallucinations when clinical sections contain counterfactual details or ambiguous guidance. The figure above contrasts baseline predictions with CCD-enabled outputs across report generation and question answering tasks. Red highlights mark unsupported findings that can compromise patient care, while blue text reflects misleading prompt context the model must resist.

Clinical Contrastive Decoding (CCD) mitigates these risks at inference time. Rather than retraining models or relying on retrieval corpora, CCD injects trustworthy, image-grounded signals distilled from specialist expert models. The result is a decoding policy that maintains fluency yet stays faithful to the radiograph.

CCD at a Glance ⚙️

Clinical Contrastive Decoding (CCD) is a plug-and-play inference framework designed to reduce medical hallucinations in radiology MLLMs. It introduces structured clinical supervision from expert models (e.g., DenseNet or MedSigLIP, or others) at decoding time, without modifying model weights or requiring external retrieval.

Given a chest radiograph, the expert model predicts symptom-level probabilities across 14 CheXpert categories. CCD integrates this signal through a dual-stage logit refinement strategy:

Symptom-grounded Contrastive Decoding (SCD): constructs an anchor prompt using high-confidence findings (e.g., “Atelectasis, Cardiomegaly”) and generates a contrastive logits path conditioned on this prompt. The final logits are a weighted interpolation between the anchor-conditioned and original paths, encouraging the model to mention supported findings and suppress false negatives.
Expert-informed Contrastive Decoding (ECD): transforms expert probabilities into token-level logit biases via log-odds conversion. These biases are injected into the logits from the first stage, softly penalising unsupported findings and reducing false positives.

This two-stage mechanism enables CCD to progressively guide generation with both symbolic supervision (via anchor prompts) and probabilistic constraints (via confidence scores), achieving robust improvements in both report generation and VQA. CCD is fully model-agnostic and integrates seamlessly with state-of-the-art radiology MLLMs such as MAIRA-2, Libra, LLaVA-Rad, and LLaVA-Med.

Unlike prior contrastive decoding approaches that rely on perturbed visual or textual inputs, CCD leverages clinically grounded signals from expert models to provide task-specific and symptom-level control during generation.

Key Contributions ✨

Empirical Insight: We conduct a systematic analysis of prompt-induced hallucinations in radiology MLLMs, revealing that noisy clinical sections (such as irrelevant or contradictory clinical details) can trigger unsupported findings across multiple datasets.
Inference-time Framework: We introduce Clinical Contrastive Decoding (CCD) — a dual-stage inference-time strategy that leverages expert-derived labels as anchor prompts and performs probabilistic logit adjustments, requiring no retraining or architectural modifications.
Consistent Gains: Extensive experiments on MIMIC-CXR, IU-Xray, and CheXpert Plus demonstrate that CCD achieves up to +17% improvement in RadGraph-F1 with SOTA radiology MLLMs (MAIRA-2), along with higher VQA accuracy, all without altering model weights or structures.

Evaluation Highlights 📈

Achieves up to +17% RadGraph-F1 on MIMIC-CXR when attached to state-of-the-art report generators.
Reduces mention-level hallucinations on IU X-Ray while maintaining BLEU / ROUGE scores, confirming that CCD improves factuality without harming surface-level metrics.

Ablation Studies 🧪

Ablations show both contrastive stages matter: removing the anchor projection (SCD) or the logit sharpening (ECD) degrades factual metrics by ~5–8%.

My Musings ⏳

As I reflect on the development and impact of Clinical Contrastive Decoding (CCD),

It’s better to be roughly right than precisely wrong.
— Carveth Read, Logic: Deductive and Inductive

This quote perfectly captures the spirit of our approach. In the high-stakes field of radiology, the cost of hallucinations can be severe. By integrating expert-derived signals at inference time, CCD prioritises clinical accuracy over mere linguistic fluency. It’s a reminder that in medical AI, being approximately correct and grounded in reality is far more valuable than generating fluent but misleading text.

BibTeX 📚

@article{zhang2025ccd,
 title={CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding},
 author={Zhang, Xi and Meng, Zaiqiao and Lever, Jake and Ho, Edmond SL},
 journal={arXiv preprint arXiv:2509.23379},
 year={2025}
}

EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery

Mon, 09 Mar 2026 00:00:00 +0000

Part of the EvoScientist project — harnessing vibe research with self-evolving AI scientists.

Extend it with EvoSkills — installable skill & knowledge packs that add domain-specific expertise to AI scientists.

Overview

EvoScientist is an evolving multi-agent AI scientist framework that continuously improves its research strategies through persistent memory and self-evolution. It addresses a key limitation of existing AI-scientist systems: static, hand-designed pipelines that overlook promising directions, repeat failed experiments, and pursue infeasible ideas.

Framework

EvoScientist coordinates three specialized agents:

Researcher Agent (RA): scientific idea generation.
Engineer Agent (EA): experiment implementation and execution.
Evolution Manager Agent (EMA): distills insights from prior interactions into reusable knowledge.

These are backed by two persistent memory modules — an ideation memory (feasible directions distilled from top-ranked ideas, plus previously unsuccessful ones) and an experimentation memory (effective data-processing and training strategies from code-search trajectories and best-performing implementations) — which the RA and EA retrieve to improve idea quality and code execution success over time.

Key Results

Outperforms 7 open-source and commercial state-of-the-art systems in scientific idea generation, with higher novelty, feasibility, relevance, and clarity under both automatic and human evaluation.
Substantially improves code execution success rates through multi-agent evolution, demonstrating the effectiveness of persistent memory for end-to-end scientific discovery.

BibTeX

@article{lyu2026evoscientist,
 title={EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery},
 author={Lyu, Yougang and Zhang, Xi and Yi, Xinhao and Zhao, Yuyue and Guo, Shuyu and Hu, Wenxiang and Piotrowski, Jan and Kaliski, Jakub and Urbani, Jacopo and Meng, Zaiqiao and others},
 journal={arXiv preprint arXiv:2603.08127},
 year={2026}
}

Automated Chest X-ray Report Generation Remains Unsolved

Thu, 01 Jan 2026 00:00:00 +0000

For the latest updates and details, visit the ReXrank Challenge Website.

Why this work?

Automated chest X-ray report generation has the potential to substantially reduce radiologist workload and improve clinical efficiency. However, despite rapid progress in vision-language models and large language models (LLMs), the true clinical reliability of these systems remains unclear. Existing evaluations are often inconsistent, rely on limited test sets, or fail to probe generalization across institutions and clinically challenging abnormal cases.

To address these limitations, we present a large-scale, standardized benchmark study through the ReXrank Challenge V1.0, designed to rigorously assess the current state of automated chest X-ray report generation under realistic and clinically meaningful conditions.

What is the ReXrank Challenge V1.0?

The ReXrank Challenge V1.0 is a comprehensive evaluation effort built on ReXGradient, the largest test-only dataset to date for radiology report generation, comprising 10,000 studies from 67 healthcare institutions. The challenge brought together submissions from academia and industry, evaluating 8 new models alongside 16 previously benchmarked state-of-the-art systems under a unified evaluation protocol.

All models were assessed using a diverse set of metrics, ranging from traditional text similarity measures to clinically grounded and LLM-based error detection metrics, enabling a multi-dimensional analysis of model performance.

Key Findings

Automated chest X-ray report generation remains fundamentally unsolved.
Even the best-performing models achieve less than 45% error-free reporting on abnormal studies, highlighting a substantial gap between current AI systems and clinical readiness.
Large performance gaps between normal and abnormal studies.
Most models perform well on normal cases (often exceeding 80–90% no-significant-error rates) but struggle significantly with abnormal findings, where clinically important errors are common.
Poor cross-institutional generalization.
Model rankings vary dramatically across healthcare sites, indicating that strong performance on one institution does not reliably transfer to others.
Evaluation metrics capture complementary but inconsistent signals.
Traditional lexical metrics correlate poorly with LLM-based and clinically focused error metrics, underscoring the need for more clinically aligned evaluation frameworks.

Implications

Our results suggest that current AI systems are not yet suitable for fully autonomous radiology reporting, particularly for abnormal cases. However, they show promise as assistive tools for generating preliminary drafts that can be reviewed and refined by radiologists. The findings emphasize the importance of developing:

methods specifically targeting abnormality detection,
strategies for robust cross-institutional generalization, and
evaluation frameworks that better reflect clinical correctness rather than surface-level textual similarity.

Conclusion

This work provides the most comprehensive benchmark to date for automated chest X-ray report generation and delivers a clear message to the community: while progress is evident, significant challenges remain before clinically reliable deployment is possible. By releasing the ReXrank Challenge V1.0 and ReXGradient dataset, we aim to establish a rigorous foundation for future research and to drive the development of more robust, clinically grounded radiology AI systems.

BibTeX

@inproceedings{zhang2025automated,
 title={Automated Chest X-ray Report Generation Remains Unsolved},
 author={Zhang, Xiaoman and Acosta, Julian Nicolas and Yang, Xiaoli and Adithan, Subathra and Luo, Luyang and Zhou, Hong-Yu and Miller, Joshua and Huang, Ouwen and Zhou, Zongwei and Hamamci, Ibrahim Ethem and others},
 booktitle={Biocomputing 2026: Proceedings of the Pacific Symposium},
 pages={236--250},
 year={2025},
 organization={World Scientific}
}

RadEval: A framework for radiology text evaluation

Mon, 22 Sep 2025 00:00:00 +0000

For the latest updates and details, visit the RadEval Project Website.

Why RadEval?

Radiology report generation has seen significant advancements with the advent of large language models (LLMs). However, evaluating the quality of these generated reports remains a complex challenge. Traditional metrics like BLEU and ROUGE often fall short in capturing the clinical relevance and accuracy required in medical contexts. To address this gap, we introduce RadEval, a comprehensive framework designed to evaluate radiology texts using a diverse set of metrics that encompass both linguistic quality and clinical accuracy.

Key Features of RadEval

Diverse Metric Integration: RadEval consolidates a wide range of evaluation metrics, including classic n-gram overlap measures (BLEU, ROUGE), contextual embeddings (BERTScore), clinical concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT, TemporalEntityF1), and advanced LLM-based evaluators (GREEN). This integration allows for a multifaceted assessment of radiology reports.
Standardized Implementations: We have refined and standardized the implementations of these metrics to ensure consistency and reliability in evaluations across different studies and datasets.
Domain-Specific Enhancements: RadEval extends the GREEN metric to support multiple imaging modalities using a more lightweight model. Additionally, we have pretrained a domain-specific radiology encoder that demonstrates strong zero-shot retrieval performance, enhancing the evaluation capabilities of the framework.
Richly Annotated Expert Dataset: We provide a dataset annotated by radiology experts, containing over 450 clinically significant error labels. This dataset serves as a valuable resource for validating and benchmarking evaluation metrics against expert judgment.
Statistical Testing Tools: RadEval includes tools for statistical testing, enabling researchers to assess the significance of their results and compare different models robustly.
Baseline Model Evaluations: The framework offers baseline evaluations across multiple publicly available datasets, facilitating reproducibility and benchmarking in radiology report generation research.

Correlation with Radiologist Judgment

We conducted extensive experiments to assess how different metrics correlate with radiologist judgment. Our findings indicate that certain metrics, particularly those incorporating clinical concepts and LLM-based evaluations, show a stronger alignment with expert assessments. This correlation underscores the importance of using clinically informed metrics in evaluating radiology reports.

Getting Started with RadEval

To get started with RadEval, you can access the codebase and documentation on our GitHub repository. The repository includes installation instructions, usage examples, and guidelines for integrating RadEval into your evaluation pipeline. Additionally, the RadEval Expert Dataset is available for download, providing a rich resource for testing and validating evaluation metrics.

Conclusion

RadEval represents a significant step forward in the evaluation of radiology texts, offering a comprehensive and standardized framework that addresses the unique challenges of this domain. By integrating diverse metrics, providing a richly annotated dataset, and facilitating robust benchmarking, RadEval aims to enhance the quality and reliability of radiology report generation research.

BibTeX

@inproceedings{xu2025radeval,
 title={RadEval: A framework for radiology text evaluation},
 author={Xu, Justin and Zhang, Xi and Abderezaei, Javid and Bauml, Julie and Boodoo, Roger and Haghighi, Fatemeh and Ganjizadeh, Ali and Brattain, Eric and Van Veen, Dave and Meng, Zaiqiao and others},
 booktitle={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
 pages={546--557},
 year={2025}
}

A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems

Sun, 10 Aug 2025 00:00:00 +0000

For the latest updates and details, visit the Awesome Self-Evolving Agents.

Background and Development Trends

This survey provides a comprehensive overview of the latest advances in the field of self-evolving AI agents, highlighting key technological shifts and development trends.

Paradigm Definition

We define the new paradigm of “Self-Evolving AI Agents” as bridging foundation models and lifelong agentic systems.

Concept Definition of Self-Evolving AI Agents

Self-evolving AI agents are autonomous systems that continuously and systematically optimise their internal components through interaction with environments, with the goal of adapting to changing tasks, contexts and resources while preserving safety and enhancing performance.

Guiding Principles: Three Laws

Inspired by Asimov’s Three Laws of Robotics, we propose the Three Laws of Self-Evolving AI Agents:

Endure (Safety Adaptation): Any modification must maintain safety and stability.
Excel (Performance Preservation): Subject to the first law, agents must preserve or enhance task performance.
Evolve (Autonomous Evolution): Subject to the first and second laws, agents must autonomously optimize their internal components in response to changes.

Unified Framework and Technical Review

We propose a unified conceptual framework with four core components: System Inputs, Agent System, Environment, and Optimisers. The survey systematically reviews evolution strategies across foundation models, prompts, memory, tools, workflows, and inter-agent communication.

Single-Agent Optimisation

We explore optimisation techniques for individual self-evolving agents, focusing on methods that enhance their learning and adaptation capabilities.

Multi-Agent Optimisation

We investigate strategies for optimising the performance of multiple self-evolving agents working collaboratively, addressing challenges such as communication, coordination, and resource sharing.

Domain Adaptation and Challenges

We also analyze domain-specific evolution methods in biomedicine, programming, and finance, and discuss key challenges in evaluation, safety, and ethics, laying the foundation for the next generation of adaptive, autonomous, lifelong agentic systems.

BibTeX

@article{fang2025comprehensive,
 title={A comprehensive survey of self-evolving ai agents: A new paradigm bridging foundation models and lifelong agentic systems},
 author={Fang, Jinyuan and Peng, Yanwen and Zhang, Xi and Wang, Yingxu and Yi, Xinhao and Zhang, Guibin and Xu, Yi and Wu, Bin and Liu, Siwei and Li, Zihao and others},
 journal={arXiv preprint arXiv:2508.07407},
 year={2025}
}

Libra: Leveraging Temporal Images for Biomedical Radiology Analysis

Wed, 01 Jan 2025 00:00:00 +0000

For the latest updates and details, visit the Libra Project Website.

Why Libra?

Temporal hallucination is a critical challenge in radiology report generation (RRG). Traditional multimodal large language models (MLLMs) struggle to integrate prior images correctly, often generating:

Spurious references to nonexistent prior studies (Single-image case).
Inaccurate interpretations of disease progression (Temporal-image case).

Libra addresses these limitations by integrating a Temporal Alignment Connector (TAC) to improve temporal awareness, ensuring:

Prior studies are correctly referenced only when available.

Hallucinated references are eliminated, avoiding misleading reports.

Temporal changes are accurately captured, ensuring clinically meaningful outputs.

Overview

We propose Libra (Leveraging Temporal Images for Biomedical Radiology Analysis), a novel framework tailored for radiology report generation (RRG) that incorporates temporal change information to address the challenges of interpreting medical images effectively.

Libra leverages RAD-DINO, a pre-trained visual transformer, as its image encoder to generate robust and scalable image features. These features are further refined by a Temporal Alignment Connector (TAC), a key innovation in Libra’s architecture. The TAC comprises:

Layerwise Feature Extractor (LFE): Captures high-granularity image feature embeddings from the encoder.
Temporal Fusion Module (TFM): Integrates temporal references from prior studies to enhance temporal awareness and reasoning.

These refined features are fed into Meditron, a specialised medical large language model (LLM), to generate comprehensive, temporally-aware radiology reports. Libra’s modular design seamlessly integrates state-of-the-art open-source pre-trained models for both image and text, aligning them through a temporal-aware adapter to ensure robust cross-modal reasoning and understanding.

Through a two-stage training strategy, Libra demonstrates the powerful potential of multimodal large language models (MLLMs) in specialised radiology applications. Extensive experiments on the MIMIC-CXR dataset highlight Libra’s performance, setting a new state-of-the-art benchmark among models of the same parameter scale.

Key Contributions

Temporal Awareness: Libra captures and synthesizes temporal changes in medical images, addressing the challenge of handling prior study citations in RRG tasks.
Innovative Architecture: The Temporal Alignment Connector (TAC) ensures high-granularity feature extraction and temporal integration, significantly enhancing cross-modal reasoning capabilities.
State-of-the-Art Performance: Libra achieves outstanding results on the MIMIC-CXR dataset, outperforming existing MLLMs in both accuracy and temporal reasoning.
Libra Repository: Our code space provides a public and detailed implementation of Libra, facilitating reproducibility and further research in the field, and integrates both training and evaluation processes, ensuring a streamlined and efficient workflow for developing and testing radiology report generation models.

Experimental Results

Libra was designed to excel in radiology report generation (RRG), and its performance speaks for itself. Here’s a quick breakdown of its achievements:

Lexical and Clinical Metrics: Libra delivers competitive results across traditional metrics like ROUGE-L, BLEU, METEOR, and RadGraph-based scores.
Radiologist-Aligned Metrics: It leads in the RadCliQ metric and CheXbert vector similarity, showcasing its alignment with clinical expectations.
CheXpert Classification: Libra achieves the highest Macro-F1 scores and remains competitive in Micro-F1, further solidifying its reliability in clinical classification tasks.

By leveraging its Temporal Alignment Connector (TAC), Libra effectively captures temporal contexts, generating radiology reports that are both accurate and clinically meaningful. While there are minor gaps in select clinical metrics, Libra’s robust performance demonstrates its potential to set new standards in RRG.

Performance Analysis

Cases Without Prior Images

In scenarios where no prior image is available, Libra shines by delivering detailed and clinically relevant descriptions without introducing spurious references. For instance, as shown in Figure 3 (a), Libra identified “sternal wires” and their type, going beyond the ground truth. This demonstrates its ability to provide meaningful insights while avoiding errors caused by nonexistent prior studies.

Cases With Prior Images

When prior images are available, Libra takes its analysis to the next level. In Figure 3 (b), new abnormalities such as pleural effusion and pneumonia were identified in the current image. Without considering the prior image, Libra accurately described the findings without inferring disease progression, maintaining clinical accuracy. However, when the prior image was included, Libra effectively captured the progressive changes, provided detailed descriptions, and explicitly referenced the comparison. This capability ensures a clear understanding of temporal changes and enhances the accuracy of disease progression descriptions.

Evaluating Temporal Consistency

To test Libra’s temporal reasoning, we swapped the image order, treating the prior image as the current image and vice versa. The generated report reflected an improved patient condition, aligning with the reversed input sequence but contradicting the original ground truth. Interestingly, the report closely resembled the original description of the prior image, as shown at the bottom of Figure 3 (b). This highlights Libra’s ability to adapt to different temporal contexts, generating accurate and contextually consistent reports that align with standard clinical practices.

Libra’s performance in these cases underscores its potential to revolutionize radiology report generation by seamlessly integrating temporal reasoning into its analysis.

BibTeX

@inproceedings{zhang-etal-2025-libra,
 title = "Libra: Leveraging Temporal Images for Biomedical Radiology Analysis",
 author = "Zhang, Xi and
 Meng, Zaiqiao and
 Lever, Jake and
 Ho, Edmond S. L.",
 editor = "Che, Wanxiang and
 Nabende, Joyce and
 Shutova, Ekaterina and
 Pilehvar, Mohammad Taher",
 booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
 month = jul,
 year = "2025",
 address = "Vienna, Austria",
 publisher = "Association for Computational Linguistics",
 url = "https://aclanthology.org/2025.findings-acl.888/",
 pages = "17275--17303",
 ISBN = "979-8-89176-256-5",
 abstract = "Radiology report generation (RRG) requires advanced medical image analysis, effective temporal reasoning, and accurate text generation. While multimodal large language models (MLLMs) align with pre-trained vision encoders to enhance visual-language understanding, most existing methods rely on single-image analysis or rule-based heuristics to process multiple images, failing to fully leverage temporal information in multi-modal medical datasets. In this paper, we introduce **Libra**, a temporal-aware MLLM tailored for chest X-ray report generation. Libra combines a radiology-specific image encoder with a novel Temporal Alignment Connector (**TAC**), designed to accurately capture and integrate temporal differences between paired current and prior images. Extensive experiments on the MIMIC-CXR dataset demonstrate that Libra establishes a new state-of-the-art benchmark among similarly scaled MLLMs, setting new standards in both clinical relevance and lexical accuracy. All source code and data are publicly available at: https://github.com/X-iZhang/Libra."
}

@inproceedings{zhang2025libra,
 title={Libra: Leveraging temporal images for biomedical radiology analysis},
 author={Zhang, Xi and Meng, Zaiqiao and Lever, Jake and Ho, Edmond SL},
 booktitle={Findings of the Association for Computational Linguistics: ACL 2025},
 pages={17275--17303},
 year={2025}
}

Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation

Sat, 20 Jul 2024 00:00:00 +0000

Introduction

Radiology reports are critical tools for interpreting medical imaging, such as chest X-rays. These reports typically consist of two main sections: FINDINGS, which detail observations from the image, and IMPRESSIONS, which summarize key takeaways and recommendations.

For example:

FINDINGS:
Increase in size of the left pleural effusion compared to the prior exam. Right lung remains clear. Mildly enlarged but stable heart size.

IMPRESSIONS:
Increase in left pleural effusion. Stable mild cardiomegaly.

Generating such reports automatically is a challenging task that requires aligning visual data with textual descriptions. While general-purpose visual language models like LLaVA and InstructBLIP have shown promise in multimodal tasks, radiology report generation demands a higher level of precision and domain-specific adaptation.

Our approach focuses on fine-tuning a visual language model specifically for radiology. By aligning chest X-ray features with a large language model and applying advanced techniques like Low-Rank Adaptation (LoRA), we enhance the model’s ability to generate accurate and clinically relevant reports. Additionally, we employ a method to process multiple images simultaneously, enabling the model to capture nuanced details across different X-rays.

This work was developed for the RRG24 Shared Task at BioNLP 2024, where our model achieved competitive results, ranking 4th on the leaderboard. Key contributions include:

Domain-specific fine-tuning: Optimizing the model for radiology tasks through visual instruction tuning.
Multi-image processing: Combining multiple X-rays into a single input for efficient and accurate interpretation.

Methodology

Our approach builds on insights from LLaVA-Med, emphasizing the advantages of starting with a language-only pretrained LLM rather than a multimodal-trained base. The model integrates an image encoder with a learnable adapter, following the LLaVA-1.5 architecture. Training involves an auto-regressive language modeling approach using cross-entropy loss, with hyperparameters aligned to LLaVA-1.5. We employ Low-Rank Adaptation (LoRA) for efficient fine-tuning, starting with adapter pretraining for one epoch, followed by joint tuning for three epochs.

Proposed Models

To tackle the RRG24 shared task, we developed two specialized models: Med-CXRGen-F for Findings and Med-CXRGen-I for Impressions. Both models leverage CLIP as the image encoder and Vicuna-1.5 as the LLM. A multi-layer perceptron (MLP) adapter with GELU activations and a uniform hidden size of 1024 processes image features before aligning them with the LLM.

How It Works

Image Encoding: The image encoder converts chest X-rays into patch tokens, extracting embeddings from the penultimate layer.

Adapter Alignment: The MLP adapter processes these embeddings, aligning them to the LLM’s input format.

Prompt Design: The model is prompted with:

Provide a description of the findings/impressions from the radiology <image>\n image.

Here, "<image>" acts as a placeholder token, guiding the LLM to generate text based on the input image, as shown in Figure.

Training

Our training process involves a two-stage approach to optimize the model for radiology report generation: (as shown in Figure)

Chest X-ray Feature Alignment
In the first stage, we align chest X-ray image features with textual embeddings in the language model. Using the provided dataset, the model predicts captions based on image inputs and instructions. During this phase, only the MLP adapter is updated, while the visual encoder and LLM weights remain frozen. This single-epoch training expands the vocabulary of aligned image-text tokens specific to the radiology domain.
Fine-tuning for Report Generation
In the second stage, we fine-tune the pre-trained LLM weights using LoRA technology. The visual encoder and adapter weights are kept frozen, while the model undergoes three epochs of training on the dataset. This phase focuses on enhancing the model’s ability to generate accurate and clinically relevant radiology reports.

For implementation details, see the Code Repo.

Evaluation

Dataset

Our models were fine-tuned and evaluated using the RRG24 dataset, hosted on the BioNLP ACL'24 platform. This dataset aggregates data from multiple sources, including MIMIC-CXR, CheXpert, PadChest, BIMCV-COVID19, and OpenI. The dataset statistics are summarized below:

Dataset	FINDINGS	IMPRESSIONS
Training	344,394	366,413
Validation	8,839	9,331
Test-Public	2,692	2,967
Test-Hidden	1,063	1,428

Results

Our models achieved competitive F1-RadGraph scores of 24.13 and 22.79 for the Findings and Impressions sections, respectively, on the public test set. On the hidden test set, the scores were 24.13 and 22.10, securing 4th place on the leaderboard at the time of submission. Additionally, the Bertscore results highlight the high-quality text generation capabilities of our models, with scores of 53.45 for Findings and 47.39 for Impressions on the public test set.

Discussion

The difference in performance between the Findings and Impressions sections comes down to their unique roles. Findings are all about objectively describing what’s seen in the images, while Impressions focus on drawing diagnostic conclusions. This difference, along with the varying lengths of these sections, makes generating Impressions a bit trickier, which is reflected in the evaluation scores.

Another factor is the variety of diseases in the test set. This uneven distribution can make it harder for the model to generalize well. Plus, when the model processes multiple images at once, it sometimes includes irrelevant ones, which can throw off its accuracy. Improving the model’s ability to filter out unnecessary images could make a big difference.

Looking ahead, there’s a lot of room for improvement. Fine-tuning the model specifically for medical data and adapting it to better handle domain-specific challenges could help. Adding the ability to track changes over time and creating smarter frameworks for multi-modal reports are also exciting directions to explore. These upgrades could make the model even more useful and reliable in real-world clinical settings.

Conclusion

In this study, we introduced a model designed to generate radiology reports by aligning visual and textual data. By fine-tuning it for specific tasks, we managed to achieve solid results, including a fourth-place finish in the RRG24 Shared Task at BioNLP 2024. This shows the potential of our approach for specialized medical applications.

Moving forward, we’re excited to explore ways to make the model even better. This includes developing methods to handle multi-modal data more effectively and incorporating time-based insights. These improvements could take the model’s accuracy and practicality to the next level for clinical use.

Limitations

While the results are promising, there are a few challenges we need to address:

Easier-to-Detect Conditions: Some diseases are simpler to identify, which might inflate the evaluation scores.
Dataset Imbalance: The training data isn’t perfectly balanced in terms of imaging types, body parts, or report lengths, which could impact performance.
Inconsistent Reporting Styles: Radiologists have different ways of writing reports, and this variability makes it harder for the model to generate consistent outputs.

The data used for the training processes were exclusively sourced from the provided dataset, which complies with ethical standards regarding Patient Health Information. Adherence to these standards guarantees the responsible handling of sensitive data throughout our research.

BibTeX

@inproceedings{zhang-etal-2024-gla,
 title = "Gla-{AI}4{B}io{M}ed at {RRG}24: Visual Instruction-tuned Adaptation for Radiology Report Generation",
 author = "Zhang, Xi and
 Meng, Zaiqiao and
 Lever, Jake and
 Ho, Edmond S.L.",
 editor = "Demner-Fushman, Dina and
 Ananiadou, Sophia and
 Miwa, Makoto and
 Roberts, Kirk and
 Tsujii, Junichi",
 booktitle = "Proceedings of the 23rd Workshop on Biomedical Natural Language Processing",
 month = aug,
 year = "2024",
 address = "Bangkok, Thailand",
 publisher = "Association for Computational Linguistics",
 url = "https://aclanthology.org/2024.bionlp-1.54/",
 doi = "10.18653/v1/2024.bionlp-1.54",
 pages = "624--634",
}