Radiology Report Generation | Xi Zhang

🎤 Talk at Glasgow AI4BioMed Lab!

Wed, 17 Jun 2026 00:00:00 +0000

💡 Talk Title

“CCS: Clinical Consensus Selection for Radiology Report Generation”

🖇️ Abstract

Radiology report generation (RRG) is commonly framed as a single-path task, where a multimodal large language model (MLLM) produces one decoded report as the final output. While progress has largely come from scaling training data, model capacity, and retrieval mechanisms, improving report quality at inference time remains underexplored. We observe that fixed radiology MLLMs often generate clinically stronger reports elsewhere in their candidate pool than the one selected by default decoding, suggesting that inference-time decision making is an overlooked bottleneck. To address this, we propose Clinical Consensus Selection (CCS), a decoder-agnostic inference-time selection framework that samples multiple candidate reports and selects the one with the highest clinical consensus across the rollout pool. CCS combines text-based utilities with a radiology-adapted utility computed by an image–report-trained multimodal embedder, measuring agreement beyond surface-level textual similarity. Across three datasets and multiple radiology MLLMs, CCS consistently improves inference-time performance over single-path decoding and generic Best-of-N baselines, with especially clear gains on clinical metrics. Further analysis shows that image-grounded utility forms a selection axis distinct from textual consensus, and that substantial headroom remains for improving RRG at inference time.

🔗 Project Website: https://x-izhang.github.io/CCS/

📺 Slides

📍 Glasgow AI4BioMed Lab — An interdisciplinary research lab focusing on AI applications in biomedical sciences, based in Glasgow.

📅

When: Wednesday, June 17, 2026 at 2pm
Where: F121

See you there!

🎤 Talk at Glasgow AI4BioMed Lab!

Wed, 28 Jan 2026 00:00:00 +0000

💡 Talk Title

“CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding”

🖇️ Abstract

Radiology multimodal large language models (MLLMs) often generate clinically unsupported descriptions, posing serious risks in medical applications. We introduce Clinical Contrastive Decoding (CCD), a training-free inference framework that integrates structured clinical signals from radiology expert models to mitigate hallucinations. CCD refines token-level logits during generation through a dual-stage contrastive mechanism, enhancing clinical fidelity without modifying the base MLLM. On the MIMIC-CXR dataset,CCD yields up to 17% improvement in RadGraph-F1, providing a lightweight solution for bridging expert models and MLLMs in radiology.

🔗 Project Website: https://x-izhang.github.io/CCD/

📺 Slides

📍 Glasgow AI4BioMed Lab — An interdisciplinary research lab focusing on AI applications in biomedical sciences, based in Glasgow.

📅

When: Wednesday, January 28, 2026 at 2pm
Where: F121

See you there!

Automated Chest X-ray Report Generation Remains Unsolved

Thu, 01 Jan 2026 00:00:00 +0000

For the latest updates and details, visit the ReXrank Challenge Website.

Why this work?

Automated chest X-ray report generation has the potential to substantially reduce radiologist workload and improve clinical efficiency. However, despite rapid progress in vision-language models and large language models (LLMs), the true clinical reliability of these systems remains unclear. Existing evaluations are often inconsistent, rely on limited test sets, or fail to probe generalization across institutions and clinically challenging abnormal cases.

To address these limitations, we present a large-scale, standardized benchmark study through the ReXrank Challenge V1.0, designed to rigorously assess the current state of automated chest X-ray report generation under realistic and clinically meaningful conditions.

What is the ReXrank Challenge V1.0?

The ReXrank Challenge V1.0 is a comprehensive evaluation effort built on ReXGradient, the largest test-only dataset to date for radiology report generation, comprising 10,000 studies from 67 healthcare institutions. The challenge brought together submissions from academia and industry, evaluating 8 new models alongside 16 previously benchmarked state-of-the-art systems under a unified evaluation protocol.

All models were assessed using a diverse set of metrics, ranging from traditional text similarity measures to clinically grounded and LLM-based error detection metrics, enabling a multi-dimensional analysis of model performance.

Key Findings

Automated chest X-ray report generation remains fundamentally unsolved.
Even the best-performing models achieve less than 45% error-free reporting on abnormal studies, highlighting a substantial gap between current AI systems and clinical readiness.
Large performance gaps between normal and abnormal studies.
Most models perform well on normal cases (often exceeding 80–90% no-significant-error rates) but struggle significantly with abnormal findings, where clinically important errors are common.
Poor cross-institutional generalization.
Model rankings vary dramatically across healthcare sites, indicating that strong performance on one institution does not reliably transfer to others.
Evaluation metrics capture complementary but inconsistent signals.
Traditional lexical metrics correlate poorly with LLM-based and clinically focused error metrics, underscoring the need for more clinically aligned evaluation frameworks.

Implications

Our results suggest that current AI systems are not yet suitable for fully autonomous radiology reporting, particularly for abnormal cases. However, they show promise as assistive tools for generating preliminary drafts that can be reviewed and refined by radiologists. The findings emphasize the importance of developing:

methods specifically targeting abnormality detection,
strategies for robust cross-institutional generalization, and
evaluation frameworks that better reflect clinical correctness rather than surface-level textual similarity.

Conclusion

This work provides the most comprehensive benchmark to date for automated chest X-ray report generation and delivers a clear message to the community: while progress is evident, significant challenges remain before clinically reliable deployment is possible. By releasing the ReXrank Challenge V1.0 and ReXGradient dataset, we aim to establish a rigorous foundation for future research and to drive the development of more robust, clinically grounded radiology AI systems.

BibTeX

@inproceedings{zhang2025automated,
 title={Automated Chest X-ray Report Generation Remains Unsolved},
 author={Zhang, Xiaoman and Acosta, Julian Nicolas and Yang, Xiaoli and Adithan, Subathra and Luo, Luyang and Zhou, Hong-Yu and Miller, Joshua and Huang, Ouwen and Zhou, Zongwei and Hamamci, Ibrahim Ethem and others},
 booktitle={Biocomputing 2026: Proceedings of the Pacific Symposium},
 pages={236--250},
 year={2025},
 organization={World Scientific}
}

🎉 Paper Accepted at PSB 2026!

Mon, 22 Dec 2025 00:00:00 +0000

Automated Chest X-ray Report Generation Remains Unsolved has been accepted to Pacific Symposium on Biocomputing 2026!

For the 🏆 Chest X-ray Interpretation Leaderboard 🏆, please visit ReXrank.

📍 PSB 2026 - The Pacific Symposium on Biocomputing 2026 will be held in Hawaii, USA, from January 3-7, 2026.

🪧 Poster

🎤 Invited Talk at DICTA 2025!

Tue, 25 Nov 2025 00:00:00 +0000

💡 Talk Title

Multimodal Medical Models: Cross-modal Alignment and Consistency

🖇️ Abstract

This talk provides an overview of recent developments in medical vision–language modelling and examines key sources of misalignment that lead to hallucinations. Emerging strategies to enhance visual, semantic, and temporal consistency will also be discussed, highlighting pathways toward safer and more trustworthy clinical AI systems.

📺 Slides

📍 DICTA 2025 — The 26th International Conference on Digital Image Computing: Techniques and Applications will take place in Adelaide, Australia, from 3–5 December 2025.

🏥 MedAI-CHAS — A workshop focusing on Challenges, Hallucinations, and Solutions for Advancing Clinical Utility in Medical AI.

See you there!

RadEval: A framework for radiology text evaluation

Mon, 22 Sep 2025 00:00:00 +0000

For the latest updates and details, visit the RadEval Project Website.

Why RadEval?

Radiology report generation has seen significant advancements with the advent of large language models (LLMs). However, evaluating the quality of these generated reports remains a complex challenge. Traditional metrics like BLEU and ROUGE often fall short in capturing the clinical relevance and accuracy required in medical contexts. To address this gap, we introduce RadEval, a comprehensive framework designed to evaluate radiology texts using a diverse set of metrics that encompass both linguistic quality and clinical accuracy.

Key Features of RadEval

Diverse Metric Integration: RadEval consolidates a wide range of evaluation metrics, including classic n-gram overlap measures (BLEU, ROUGE), contextual embeddings (BERTScore), clinical concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT, TemporalEntityF1), and advanced LLM-based evaluators (GREEN). This integration allows for a multifaceted assessment of radiology reports.
Standardized Implementations: We have refined and standardized the implementations of these metrics to ensure consistency and reliability in evaluations across different studies and datasets.
Domain-Specific Enhancements: RadEval extends the GREEN metric to support multiple imaging modalities using a more lightweight model. Additionally, we have pretrained a domain-specific radiology encoder that demonstrates strong zero-shot retrieval performance, enhancing the evaluation capabilities of the framework.
Richly Annotated Expert Dataset: We provide a dataset annotated by radiology experts, containing over 450 clinically significant error labels. This dataset serves as a valuable resource for validating and benchmarking evaluation metrics against expert judgment.
Statistical Testing Tools: RadEval includes tools for statistical testing, enabling researchers to assess the significance of their results and compare different models robustly.
Baseline Model Evaluations: The framework offers baseline evaluations across multiple publicly available datasets, facilitating reproducibility and benchmarking in radiology report generation research.

Correlation with Radiologist Judgment

We conducted extensive experiments to assess how different metrics correlate with radiologist judgment. Our findings indicate that certain metrics, particularly those incorporating clinical concepts and LLM-based evaluations, show a stronger alignment with expert assessments. This correlation underscores the importance of using clinically informed metrics in evaluating radiology reports.

Getting Started with RadEval

To get started with RadEval, you can access the codebase and documentation on our GitHub repository. The repository includes installation instructions, usage examples, and guidelines for integrating RadEval into your evaluation pipeline. Additionally, the RadEval Expert Dataset is available for download, providing a rich resource for testing and validating evaluation metrics.

Conclusion

RadEval represents a significant step forward in the evaluation of radiology texts, offering a comprehensive and standardized framework that addresses the unique challenges of this domain. By integrating diverse metrics, providing a richly annotated dataset, and facilitating robust benchmarking, RadEval aims to enhance the quality and reliability of radiology report generation research.

BibTeX

@inproceedings{xu2025radeval,
 title={RadEval: A framework for radiology text evaluation},
 author={Xu, Justin and Zhang, Xi and Abderezaei, Javid and Bauml, Julie and Boodoo, Roger and Haghighi, Fatemeh and Ganjizadeh, Ali and Brattain, Eric and Van Veen, Dave and Meng, Zaiqiao and others},
 booktitle={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
 pages={546--557},
 year={2025}
}

🎉 Paper Accepted — See You at EMNLP 2025!

Tue, 09 Sep 2025 00:00:00 +0000

“RadEval: A framework for radiology text evaluation”, has been accepted for system demonstration at EMNLP 2025 with Oral Presentation!

For detailed model information and source code, please visit our GitHub repository: RadEval

PyPI Package — Install RadEval with pip
HuggingFace Model — Access our domain-adapted evaluation model
Interactive Demo — Try RadEval online
Research Paper — Read our detailed research paper

📍 EMNLP 2025 - The 2025 Conference on Empirical Methods in Natural Language Processing will be held in Suzhou, China, from November 4 - 9, 2025.

🩺 RadEval Debuts！

Mon, 14 Jul 2025 00:00:00 +0000

🩺 Revolutionizing Radiology Text Evaluation with AI-Powered Metrics

Imagine having a comprehensive evaluation framework that doesn’t just measure surface-level text similarity, but truly understands clinical accuracy and medical semantics in radiology reports. This vision is now a reality with RadEval, a groundbreaking, open-source evaluation toolkit designed specifically for AI-generated radiology text.

📊 All-in-one metrics for evaluating AI-generated radiology text

From traditional n-gram metrics to advanced LLM-based evaluations, RadEval provides 11+ different evaluation metrics in one unified framework, enabling researchers to thoroughly assess their radiology text generation models with domain-specific medical knowledge integration.

For detailed handbook, please visit our GitHub repository:

🚀 Quick Start Demo

Try RadEval instantly with our interactive Gradio demo:

💡 Key Features

RadEval stands out with its comprehensive approach to radiology text evaluation:

🎯 Domain-Specific: Tailored for radiology with medical knowledge integration
📈 Multi-Metric: Supports lexical, semantic, clinical, and temporal evaluations
⚡ Easy to Use: Simple API with flexible configuration options
🔬 Research-Ready: Built-in statistical testing for system comparison
📦 PyPI Available: Install with a simple pip install RadEval

🏥 Advancing Radiology AI Research Community

We are committed to building a standardized and reproducible toolkit for researchers, clinicians, and developers dedicated to advancing AI evaluation in medical imaging and radiology. Together, we’re setting new standards for clinical AI assessment.

PyPI Package — Install RadEval with pip
HuggingFace Model — Access our domain-adapted evaluation model
Interactive Demo — Try RadEval online
Research Paper — Read our detailed research paper

🎉 Paper Accepted — See You at ACL 2025!

Thu, 15 May 2025 00:00:00 +0000

“Libra: Leveraging Temporal Images for Biomedical Radiology Analysis” has been accepted to ACL 2025!

Explore the critical role of temporal information in radiology diagnostics and how the Libra model applies it in detail at blog: Libra - Temporal Insight 🕰️.

Dive into how model architecture shapes temporal reasoning capabilities in radiology imaging analysis at blog: Libra – Structural Logic 🧠.

Look ahead to future directions in radiology AI, moving beyond temporal comparison at blog: Libra – What about next? 🛸.

📍 ACL 2025 — The 63rd Annual Meeting of the Association for Computational Linguistics will take place in Vienna, Austria, July 27–August 1st, 2025

See you there!

📺 Talk

🪧 Poster

Libra - What about next?🛸

Fri, 25 Apr 2025 00:00:00 +0000

Beyond Temporal Comparison: The Future of Radiology Modeling

In our previous discussions, we delved into how Libra leverages temporal information through its innovative Temporal Alignment Connector (TAC) to enhance radiology report generation. While this approach has shown significant promise, it’s essential to look ahead and consider how radiology modeling can evolve further to meet the complex demands of clinical practice.

The Temporal Alignment Connector has proven effective for handling paired images, but the future of radiology AI extends far beyond just temporal comparison.

1. Embracing Multimodal Integration

Radiological diagnosis doesn’t occur in isolation. Clinicians often consider a plethora of data—ranging from patient history and laboratory results to various imaging modalities. The future of radiology modeling lies in the seamless integration of these diverse data sources.

Clinical Contextualization

Incorporating electronic health records (EHRs), lab results, and patient histories can provide models with a richer context
Leading to more accurate and personalized diagnostics
Reducing false positives and negatives through contextual awareness

Cross-Modality Analysis

Combining data from different imaging modalities (e.g., CT, MRI, PET) offers a more comprehensive view
Enables detection of patterns that might be missed when analyzing a single modality
Creates synergistic understanding of complex pathologies

graph TD A[Patient Data] --> B{Multimodal
Integration} C[Chest X-ray] --> B D[CT Scan] --> B E[Lab Results] --> B F[Patient History] --> B B --> G[Comprehensive
Analysis] G --> H[Enhanced
Diagnostic Accuracy] G --> I[Personalized
Treatment Plans] G --> J[Early Disease
Detection] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#f5f5f5,stroke:#333,stroke-width:1px style D fill:#f5f5f5,stroke:#333,stroke-width:1px style E fill:#f5f5f5,stroke:#333,stroke-width:1px style F fill:#f5f5f5,stroke:#333,stroke-width:1px style G fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style H fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style I fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style J fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px

2. Advancing Explainability and Trustworthiness

As AI models become more integral to clinical decision-making, their interpretability becomes paramount. Clinicians need to understand the rationale behind a model’s prediction to trust and effectively utilize its insights.

In healthcare, trust isn’t optional—it’s essential. An AI system that can’t explain its reasoning is a black box that most physicians will rightfully hesitate to rely on.

Explainable AI (XAI)

Developing models that provide clear, human-understandable explanations for their predictions
Bridging the gap between AI outputs and clinical reasoning
Using attention visualization and feature attribution methods to highlight decision factors

Even the most accurate model will face adoption challenges if clinicians cannot verify its reasoning or understand how it arrived at its conclusions.

Uncertainty Quantification

Implementing mechanisms to convey confidence levels enables:

Clinicians to assess the reliability of AI-assisted diagnostics
Appropriate intervention in cases of model uncertainty
Continuous improvement through focused retraining on uncertain cases

3. Ensuring Robustness and Generalizability

AI models must perform reliably across diverse patient populations and clinical settings—a challenge that extends beyond academic validation to real-world implementation.

Diverse Training Data

- Building Robust Radiology AI
- Data Diversity Dimensions
- Demographic Factors
- Age groups
- Ethnic backgrounds
- Sex and gender representation
- Clinical Variables
- Disease prevalence variations
- Comorbidity patterns
- Treatment history diversity
- Technical Variability
- Multiple scanner manufacturers
- Various imaging protocols
- Quality and resolution differences
- Implementation Strategies
- Federated Learning
- Cross-institution collaboration
- Privacy-preserving techniques
- Data Augmentation
- Synthetic minority examples
- Domain randomization
- Continuous Validation
- Geographic generalization testing
- Temporal drift monitoring

Continuous Learning

Implementing systems that update from new clinical data
Adapting to evolving medical knowledge and practices
Maintaining performance as disease patterns and imaging technologies change

4. Integrating into Clinical Workflows

For AI models to be truly effective, they must integrate seamlessly into existing clinical workflows rather than disrupting established processes.

User-Friendly Interfaces

Designing intuitive interfaces that present AI insights clearly
Ensuring actionable information is immediately accessible
Minimizing cognitive load during busy clinical sessions

Workflow Compatibility

The ideal radiology AI system should:

Complement rather than replace radiologist expertise
Reduce administrative burden through automatic report generation
Prioritize cases based on urgency and findings

The most advanced AI system will fail if it adds steps to an already complex workflow. Success depends on making the radiologist’s job easier, not more complicated.

5. Ethical and Regulatory Considerations

As AI becomes more prevalent in healthcare, addressing ethical and regulatory challenges becomes essential for responsible implementation.

Data Privacy and Security

Safeguarding patient data through robust encryption
Ensuring compliance with regulations like HIPAA and GDPR
Implementing federated learning approaches to minimize data sharing

Regulatory Approval

Navigating the complex regulatory landscape (FDA, CE marking)
Designing validation studies that meet regulatory requirements
Establishing monitoring systems for post-deployment performance

Ethical AI Development

graph TD A[Ethical AI Development] --> B[Fairness & Bias Mitigation] A --> C[Transparency & Explainability] A --> D[Privacy Protection] A --> E[Human Oversight] B --> F[Equitable Healthcare Outcomes] C --> G[Informed Clinical Decisions] D --> H[Patient Trust & Confidentiality] E --> I[Safe AI Implementation] style A fill:#e1f5fe,stroke:#01579b,stroke-width:2px style B fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style D fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style E fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style H fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style I fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px

Conclusion: The Road Ahead for Libra

The journey of Libra represents a significant step forward in radiology modeling, particularly in harnessing temporal information through the TAC architecture. However, the path ahead involves:

Expanding beyond paired chest X-rays to multiple imaging modalities
Enhancing explainability through attention visualization and reasoning paths
Building more robust models through diverse training strategies
Designing intuitive interfaces for seamless clinical integration
Navigating ethical and regulatory requirements for real-world deployment

As we continue to develop Libra and similar technologies, our focus remains on augmenting—rather than replacing—clinical expertise, creating tools that serve as trusted partners in the complex art of radiological diagnosis.

💬 Note: The views expressed here are my own, reflecting my personal insights into the evolving landscape of radiology AI.

🎤 Talk & Poster at HealTAC 2025!

Tue, 22 Apr 2025 00:00:00 +0000

We’re excited to share that we’ve been invited to present our recent research — “Towards Temporal-Aware Multimodal Large Language Models for Improved Radiology Report Generation” — as a lightning talk and poster presentation at the PhD Forum of HealTAC 2025!

📍 HealTAC 2025 — The 8th Healthcare Text Analytics Conference will take place in Glasgow, 16–18 June 2025.

See you there!

Libra – Structural Logic 🧠

Tue, 15 Apr 2025 00:00:00 +0000

The Challenge of Temporal Reasoning in Radiology AI

When dealing with radiology images, especially in the context of temporal analysis—comparing current chest X-rays with previous images—standard neural network architectures often struggle. Although transformer-based multimodal large language models (MLLMs) like LLaVA demonstrate remarkable capabilities for understanding single images and textual information, they encounter substantial challenges when handling image pairs.

In my previous blog, I discussed in detail why a single prior chest X-ray is typically sufficient for accurate diagnosis and patient triage.

However, capturing meaningful temporal differences between two images remains problematic with traditional transformer structures.

When Transformers Lose the Plot: Why They Struggle with Temporal

The transformer, the cornerstone of modern large language models (LLMs), excels at sequential data processing and logical reasoning tasks. Its strength lies in handling complex linguistic structures through positional encoding, enabling nuanced relationships in textual sequences.

However, when transformers receive visual information—particularly multiple images presented simultaneously—the situation becomes more complicated. Existing methods typically concatenate image features directly into the LLM’s head, often via sequences containing hundreds of visual tokens (patch tokens), depending on the specific image encoder used.

This straightforward approach inevitably suffers from token overload, known colloquially as the “lost-in-the-middle” problem, meaning crucial temporal details may get diluted or overlooked.

Indeed, current MLLMs like LLaVA perform impressively with single-image inputs. But they quickly become overwhelmed with paired images, heavily relying on meticulously crafted instruction datasets to guide temporal comparisons explicitly:

How MLLMs Are Prompted to Compare Images

“What is the difference between <image-1-patchholder> and <image-2-patchholder>?”

Such approaches place the burden squarely on the LLM’s internal reasoning and positional encodings, complicating training and diminishing reliability. The model must:

Distinguish between multiple images using only position encodings
Process 500+ tokens per image (depending on patch number)
Compare features across long token distances

Given these limitations, an essential question arises:

Can we overcome these temporal reasoning challenges structurally, rather than through explicit prompting?

Structure Determines Function: Insights from Biology

Before we answer above question, let’s briefly reflect on the foundational relationship between structure and function—deeply ingrained in biological systems.

Macro-scale examples:

Birds have wings enabling flight
Fish possess gills allowing them to breathe underwater

Micro-scale examples:

The unique three-dimensional helical structure of proteins directly determines their biological roles
A virus’s outer shell dictates its infection pathways and interaction mechanisms

Clearly, function is fundamentally dependent on structure.

When designing novel neural architectures or modules, we must apply this principle:

Identify the desired functionality first
Then craft an appropriate structural design that inherently supports these functions

Libra’s Structural Innovation

Temporal Alignment Connector (TAC)

Following this logic, we developed the TAC in our Libra model. TAC’s primary goal is to automatically and effectively capture the relationship between two chest X-ray images—the current image (primary) and a prior image (auxiliary).

Libra’s Temporal Alignment Connector (TAC) architecture.

Unlike traditional transformers that treat all inputs equivalently, TAC explicitly structures interactions between paired images. It captures their nuanced relationship through two key modules:

- TAC Architecture
- Layerwise Feature Extractor (LFE)
- Aggregates visual features across multiple encoder layers
- Ensures rich representations from both images
- Maintains feature hierarchy information
- Temporal Fusion Module (TFM)
- Fuses features from current and prior images
- Highlights critical temporal differences
- Maintains clear image role assignment
- Current image (Primary)
- Prior image (Reference)
- Prefix Bias Mechanism
- Addresses nearly-identical image pairs
- Prevents attention collapse
- Differentiates prior image's contextual influence

An important structural consideration is the integration of a prefix bias mechanism. This component addresses the scenario where current and prior images are nearly identical—common in clinical practice. Without careful design, such similarity can cause attention mechanisms to collapse into redundant self-attention loops.

The prefix bias mitigates this risk by clearly differentiating the prior image’s contextual influence, ensuring meaningful training and robust inference.

Why Structure Matters: The Libra Advantage

By structurally encoding temporal relationships directly into the neural network’s architecture, Libra overcomes the limitations inherent in traditional prompting-based approaches. Instead of forcing the LLM to implicitly infer temporal differences through complex positional encodings and exhaustive instruction tuning, TAC explicitly and efficiently captures this essential clinical context.

Libra exemplifies the powerful concept that structural logic, thoughtfully aligned with functional requirements, dramatically enhances model performance.

This structural logic not only simplifies training but also improves:

Reliability: More consistent temporal reasoning
Interpretability: Clearer connection between features and outputs
Efficiency: Reduced dependence on instruction tuning
Clinical Alignment: Better reflection of radiologists’ actual workflow

🏄 Note: The opinions shared here reflect my own understanding and are intended to convey the structural logic behind Libra. For technical accuracy and complete details, please refer to our paper: “Libra: Leveraging Temporal Images for Biomedical Radiology Analysis”.

Libra - Temporal Insight 🕰️

Sat, 05 Apr 2025 00:00:00 +0000

What Does “Temporal” Really Mean?

In clinical radiology, temporal information is not just about “past” and “present” — it’s about change. When radiologists assess a chest X-ray, they’re not merely describing what they see in a single image; they’re often comparing it to a previous one to identify whether a patient’s condition has improved, worsened, or remained stable.

🔔 This kind of temporal reasoning is essential in everyday medical practice. Yet most multimodal large language models (MLLMs) either ignore it or fail to model it effectively.

Temporal information in radiology is fundamentally about capturing change over time, not simply collecting a series of static images. This concept is central to Libra’s design philosophy.

Time Tells the Truth: Interpreting Temporal Changes in Imaging

1. Macro-Level Progression

Is the patient improving, deteriorating, or stable?

Macro-level comparison focuses on the overall trajectory of the patient’s condition compared to prior examinations. This high-level temporal reasoning is crucial for tracking disease evolution and guiding clinical decision-making.

2. Lesion-Specific Temporal Changes

How are individual abnormalities evolving over time?

Fine-grained analysis captures precise changes in specific findings, such as “consolidation in the left lower lobe has significantly expanded” or “cardiac silhouette shows no appreciable change.” These insights enable clinicians and models to reason at the level of targeted anatomical and pathological detail.

3. Quality Over Quantity in Temporal Inputs

❗️ Adding more images often introduces noise and computational complexity without improving diagnostic value.

While it may seem intuitive that incorporating more historical images would yield better results, the reality of clinical practice shows that the most valuable temporal information comes from comparing just two key points in time.

Clinical Applications

From Diagnostic Judgement to Multi-Scale Temporal Understanding

Temporal Reasoning in Clinical Diagnosis

Radiologists rarely analyse a chest X-ray in isolation. Instead, they routinely ask:

“Has the consolidation improved since last week?”
“Is the pleural effusion new?”
“Has the cardiac silhouette changed?”

Such reasoning typically falls into three primary categories:

Improved: Lesions have shrunk or resolved.
Worsened: New abnormalities appear, or existing ones have grown.
Stable: No meaningful change is observed.

These are coarse-level temporal descriptions. On a finer level, radiologists describe:

How much a lesion has changed in size or density,
Whether opacities have shifted,
If tubes, lines, or devices have been added or removed.

Temporal Signals Span Multiple Scales

Temporal information in radiology is inherently multi-scale — ranging from global clinical trajectories to subtle, localised anatomical changes.

- Temporal Information in Radiology
- Clinical Categories
- Improved
- Lesions have shrunk
- Opacities decreased
- Inflammatory shadows reduced
- Worsened
- New abnormalities appeared
- Existing lesions grown
- New infiltrates or effusion
- Stable
- No meaningful change
- Chronic conditions
- Continuous monitoring needed
- Information Levels
- Macro-level trends
- Overall patient trajectory
- Global comparison
- Local lesion changes
- Size changes
- Density variations
- Positional shifts
- Clinical Applications
- Triage decisions
- Emergency prioritization
- Resource allocation
- Treatment evaluation
- Response assessment
- Therapy adjustment
- Long-term monitoring
- Chronic disease management
- Post-surgical follow-up

Triage and Resource Allocation

Prioritising Care When Every Minute Counts

Clinical Goals and Operational Pressures

Chest X-rays play a pivotal role in patient triage, especially in emergency and high-volume settings. Radiologists must rapidly determine:

Which patients require immediate intervention,
Who can safely wait,
And how to allocate limited resources most effectively.

Triage is fundamentally about maximising outcomes under constraint. The goal is not to fully characterise every patient’s history, but to make fast, high-impact decisions that ensure critical cases receive timely care—without neglecting those with non-urgent needs.

What Matters Most: Clinically Significant Change

In these time-sensitive settings, timeliness and diagnostic clarity outweigh completeness. Radiologists focus on:

New acute findings: Abnormalities not previously seen that may indicate emerging crises.
Significant deterioration: Rapid worsening of known conditions that may demand escalated care.
Stable chronic findings: Ongoing issues that show no meaningful progression and can be managed routinely.

The most actionable form of temporal information in triage is the clinically meaningful change between now and the previous relevant image.
This focused comparison empowers decision-making without overwhelming clinicians with unnecessary historical data.

Our Model Approach

Temporal Efficiency Aligned with Clinical Reasoning

Why Two Images Are Sufficient

In real-world radiology workflows, the most informative temporal comparison is typically between:

The current chest X-ray, and
The most recent prior image used for diagnosis.

While patients may have a rich archive of historical scans, only the immediately preceding diagnostic image provides the relevant baseline for interpreting new findings. Additional older images may support longitudinal studies, but they often introduce noise, redundancy, and delay in fast-paced clinical decision-making.

Libra’s Design Philosophy: Focused Temporal Reasoning

Libra is built on this clinically grounded principle.
Instead of processing full temporal sequences — which can be computationally expensive and semantically ambiguous — Libra learns to model directional change between two key time points.

This design enables the model to:

Mimic the focused comparison strategies of expert radiologists,
Avoid temporal noise from irrelevant or outdated scans,
Reduce computational load while preserving diagnostic fidelity.

In short, Libra treats temporal reasoning as radiologists do:
What’s changed since the last meaningful image?

Illustrations

To better understand how radiologists and Libra approach temporal comparison, we present two illustrative workflows: a general conceptual flow and a specific clinical case.

Conceptual Workflow

The following diagram outlines the reasoning pathway taken when comparing a current chest X-ray to the most recent prior image. Based on observed changes (or lack thereof), radiologists infer the clinical trajectory and guide downstream decisions.

graph TD A[Previous Chest X-ray] --> B{Current vs Previous
Analysis} B -->|Opacity Size Decrease| C[Improved] B -->|New Infiltrates or Growth| D[Worsened] B -->|No Observable Change| E[Stable] C --> F[Recovery or
Treatment Response] D --> G[Disease
Progression] E --> H[Continuous
Monitoring Needed] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style D fill:#ffebee,stroke:#c62828,stroke-width:2px style E fill:#fff8e1,stroke:#ff8f00,stroke-width:2px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#ffcdd2,stroke:#c62828,stroke-width:1px style H fill:#ffecb3,stroke:#ff8f00,stroke-width:1px

Case Example: Lung Consolidation

This diagram demonstrates a practical example: a patient with lung consolidation. Depending on the direction of change, clinical interpretation and management decisions vary significantly.

graph TD A[Case: Lung Consolidation] --> B{Time Point Comparison} B -->|Consolidation Reduced
Clearer Lung Fields| C[Improved] B -->|Consolidation Expanded
New Pleural Effusion| D[Worsened] B -->|Consolidation Unchanged
No New Features| E[Stable] C --> F[Successful Antibiotic
Treatment] D --> G[Disease Progression
Treatment Adjustment Needed] E --> H[Continue Current
Management Plan] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style D fill:#ffebee,stroke:#c62828,stroke-width:2px style E fill:#fff8e1,stroke:#ff8f00,stroke-width:2px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#ffcdd2,stroke:#c62828,stroke-width:1px style H fill:#ffecb3,stroke:#ff8f00,stroke-width:1px

AI Model Implications

Why Most MLLMs Fall Short — and How Libra Goes Further

Many multimodal large language models (MLLMs) struggle with temporal reasoning for three key reasons:

No awareness of time: They treat images independently and ignore their chronological order.
Hallucinated references: They fabricate prior findings without reliable comparison.
Lack of temporal alignment: They have no built-in mechanism to align or contrast image features across time.

Libra tackles these issues head-on.

Instead of prompting the model to “guess” what might have changed, Libra incorporates explicit temporal awareness into both its architecture and training process.

We feed the model structured, temporally aligned visual features extracted from the current and previous images. This enables Libra to:

Detect fine-grained, clinically meaningful changes,
Avoid hallucination,
And reason about progression or stability in a way that mirrors clinical thinking.

In the next section, Libra – Structural Logic 🧠, we’ll explore how Libra’s architecture is intentionally designed to reflect clinical reasoning. You’ll see how its modular structure enables it to reason across both time and image features with precision — building on the foundation established in Libra – Temporal Insight 🕰️.

ReXrank

Mon, 24 Mar 2025 00:00:00 +0000

ReXrank is a public leaderboard for AI-powered radiology report generation from chest x-ray images.

Libra: Leveraging Temporal Images for Biomedical Radiology Analysis

Wed, 01 Jan 2025 00:00:00 +0000

For the latest updates and details, visit the Libra Project Website.

Why Libra?

Temporal hallucination is a critical challenge in radiology report generation (RRG). Traditional multimodal large language models (MLLMs) struggle to integrate prior images correctly, often generating:

Spurious references to nonexistent prior studies (Single-image case).
Inaccurate interpretations of disease progression (Temporal-image case).

Libra addresses these limitations by integrating a Temporal Alignment Connector (TAC) to improve temporal awareness, ensuring:

Prior studies are correctly referenced only when available.

Hallucinated references are eliminated, avoiding misleading reports.

Temporal changes are accurately captured, ensuring clinically meaningful outputs.

Overview

We propose Libra (Leveraging Temporal Images for Biomedical Radiology Analysis), a novel framework tailored for radiology report generation (RRG) that incorporates temporal change information to address the challenges of interpreting medical images effectively.

Libra leverages RAD-DINO, a pre-trained visual transformer, as its image encoder to generate robust and scalable image features. These features are further refined by a Temporal Alignment Connector (TAC), a key innovation in Libra’s architecture. The TAC comprises:

Layerwise Feature Extractor (LFE): Captures high-granularity image feature embeddings from the encoder.
Temporal Fusion Module (TFM): Integrates temporal references from prior studies to enhance temporal awareness and reasoning.

These refined features are fed into Meditron, a specialised medical large language model (LLM), to generate comprehensive, temporally-aware radiology reports. Libra’s modular design seamlessly integrates state-of-the-art open-source pre-trained models for both image and text, aligning them through a temporal-aware adapter to ensure robust cross-modal reasoning and understanding.

Through a two-stage training strategy, Libra demonstrates the powerful potential of multimodal large language models (MLLMs) in specialised radiology applications. Extensive experiments on the MIMIC-CXR dataset highlight Libra’s performance, setting a new state-of-the-art benchmark among models of the same parameter scale.

Key Contributions

Temporal Awareness: Libra captures and synthesizes temporal changes in medical images, addressing the challenge of handling prior study citations in RRG tasks.
Innovative Architecture: The Temporal Alignment Connector (TAC) ensures high-granularity feature extraction and temporal integration, significantly enhancing cross-modal reasoning capabilities.
State-of-the-Art Performance: Libra achieves outstanding results on the MIMIC-CXR dataset, outperforming existing MLLMs in both accuracy and temporal reasoning.
Libra Repository: Our code space provides a public and detailed implementation of Libra, facilitating reproducibility and further research in the field, and integrates both training and evaluation processes, ensuring a streamlined and efficient workflow for developing and testing radiology report generation models.

Experimental Results

Libra was designed to excel in radiology report generation (RRG), and its performance speaks for itself. Here’s a quick breakdown of its achievements:

Lexical and Clinical Metrics: Libra delivers competitive results across traditional metrics like ROUGE-L, BLEU, METEOR, and RadGraph-based scores.
Radiologist-Aligned Metrics: It leads in the RadCliQ metric and CheXbert vector similarity, showcasing its alignment with clinical expectations.
CheXpert Classification: Libra achieves the highest Macro-F1 scores and remains competitive in Micro-F1, further solidifying its reliability in clinical classification tasks.

By leveraging its Temporal Alignment Connector (TAC), Libra effectively captures temporal contexts, generating radiology reports that are both accurate and clinically meaningful. While there are minor gaps in select clinical metrics, Libra’s robust performance demonstrates its potential to set new standards in RRG.

Performance Analysis

Cases Without Prior Images

In scenarios where no prior image is available, Libra shines by delivering detailed and clinically relevant descriptions without introducing spurious references. For instance, as shown in Figure 3 (a), Libra identified “sternal wires” and their type, going beyond the ground truth. This demonstrates its ability to provide meaningful insights while avoiding errors caused by nonexistent prior studies.

Cases With Prior Images

When prior images are available, Libra takes its analysis to the next level. In Figure 3 (b), new abnormalities such as pleural effusion and pneumonia were identified in the current image. Without considering the prior image, Libra accurately described the findings without inferring disease progression, maintaining clinical accuracy. However, when the prior image was included, Libra effectively captured the progressive changes, provided detailed descriptions, and explicitly referenced the comparison. This capability ensures a clear understanding of temporal changes and enhances the accuracy of disease progression descriptions.

Evaluating Temporal Consistency

To test Libra’s temporal reasoning, we swapped the image order, treating the prior image as the current image and vice versa. The generated report reflected an improved patient condition, aligning with the reversed input sequence but contradicting the original ground truth. Interestingly, the report closely resembled the original description of the prior image, as shown at the bottom of Figure 3 (b). This highlights Libra’s ability to adapt to different temporal contexts, generating accurate and contextually consistent reports that align with standard clinical practices.

Libra’s performance in these cases underscores its potential to revolutionize radiology report generation by seamlessly integrating temporal reasoning into its analysis.

BibTeX

@inproceedings{zhang-etal-2025-libra,
 title = "Libra: Leveraging Temporal Images for Biomedical Radiology Analysis",
 author = "Zhang, Xi and
 Meng, Zaiqiao and
 Lever, Jake and
 Ho, Edmond S. L.",
 editor = "Che, Wanxiang and
 Nabende, Joyce and
 Shutova, Ekaterina and
 Pilehvar, Mohammad Taher",
 booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
 month = jul,
 year = "2025",
 address = "Vienna, Austria",
 publisher = "Association for Computational Linguistics",
 url = "https://aclanthology.org/2025.findings-acl.888/",
 pages = "17275--17303",
 ISBN = "979-8-89176-256-5",
 abstract = "Radiology report generation (RRG) requires advanced medical image analysis, effective temporal reasoning, and accurate text generation. While multimodal large language models (MLLMs) align with pre-trained vision encoders to enhance visual-language understanding, most existing methods rely on single-image analysis or rule-based heuristics to process multiple images, failing to fully leverage temporal information in multi-modal medical datasets. In this paper, we introduce **Libra**, a temporal-aware MLLM tailored for chest X-ray report generation. Libra combines a radiology-specific image encoder with a novel Temporal Alignment Connector (**TAC**), designed to accurately capture and integrate temporal differences between paired current and prior images. Extensive experiments on the MIMIC-CXR dataset demonstrate that Libra establishes a new state-of-the-art benchmark among similarly scaled MLLMs, setting new standards in both clinical relevance and lexical accuracy. All source code and data are publicly available at: https://github.com/X-iZhang/Libra."
}

@inproceedings{zhang2025libra,
 title={Libra: Leveraging temporal images for biomedical radiology analysis},
 author={Zhang, Xi and Meng, Zaiqiao and Lever, Jake and Ho, Edmond SL},
 booktitle={Findings of the Association for Computational Linguistics: ACL 2025},
 pages={17275--17303},
 year={2025}
}

Libra

Wed, 20 Nov 2024 00:00:00 +0000

Temporally-aware MLLM for radiology report generation. Supports LLM backbones, validation, resume training, and smart saving.

🎉 Paper Accepted — See You at ACL 2024!

Sat, 20 Jul 2024 00:00:00 +0000

“Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation” has been accepted to BioNLP @ ACL 2024!

For detailed model information and source code, please visit our GitHub repository: RRG-BioNLP-ACL2024

🪧 Poster

Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation

Sat, 20 Jul 2024 00:00:00 +0000

Introduction

Radiology reports are critical tools for interpreting medical imaging, such as chest X-rays. These reports typically consist of two main sections: FINDINGS, which detail observations from the image, and IMPRESSIONS, which summarize key takeaways and recommendations.

For example:

FINDINGS:
Increase in size of the left pleural effusion compared to the prior exam. Right lung remains clear. Mildly enlarged but stable heart size.

IMPRESSIONS:
Increase in left pleural effusion. Stable mild cardiomegaly.

Generating such reports automatically is a challenging task that requires aligning visual data with textual descriptions. While general-purpose visual language models like LLaVA and InstructBLIP have shown promise in multimodal tasks, radiology report generation demands a higher level of precision and domain-specific adaptation.

Our approach focuses on fine-tuning a visual language model specifically for radiology. By aligning chest X-ray features with a large language model and applying advanced techniques like Low-Rank Adaptation (LoRA), we enhance the model’s ability to generate accurate and clinically relevant reports. Additionally, we employ a method to process multiple images simultaneously, enabling the model to capture nuanced details across different X-rays.

This work was developed for the RRG24 Shared Task at BioNLP 2024, where our model achieved competitive results, ranking 4th on the leaderboard. Key contributions include:

Domain-specific fine-tuning: Optimizing the model for radiology tasks through visual instruction tuning.
Multi-image processing: Combining multiple X-rays into a single input for efficient and accurate interpretation.

Methodology

Our approach builds on insights from LLaVA-Med, emphasizing the advantages of starting with a language-only pretrained LLM rather than a multimodal-trained base. The model integrates an image encoder with a learnable adapter, following the LLaVA-1.5 architecture. Training involves an auto-regressive language modeling approach using cross-entropy loss, with hyperparameters aligned to LLaVA-1.5. We employ Low-Rank Adaptation (LoRA) for efficient fine-tuning, starting with adapter pretraining for one epoch, followed by joint tuning for three epochs.

Proposed Models

To tackle the RRG24 shared task, we developed two specialized models: Med-CXRGen-F for Findings and Med-CXRGen-I for Impressions. Both models leverage CLIP as the image encoder and Vicuna-1.5 as the LLM. A multi-layer perceptron (MLP) adapter with GELU activations and a uniform hidden size of 1024 processes image features before aligning them with the LLM.

How It Works

Image Encoding: The image encoder converts chest X-rays into patch tokens, extracting embeddings from the penultimate layer.

Adapter Alignment: The MLP adapter processes these embeddings, aligning them to the LLM’s input format.

Prompt Design: The model is prompted with:

Provide a description of the findings/impressions from the radiology <image>\n image.

Here, "<image>" acts as a placeholder token, guiding the LLM to generate text based on the input image, as shown in Figure.

Training

Our training process involves a two-stage approach to optimize the model for radiology report generation: (as shown in Figure)

Chest X-ray Feature Alignment
In the first stage, we align chest X-ray image features with textual embeddings in the language model. Using the provided dataset, the model predicts captions based on image inputs and instructions. During this phase, only the MLP adapter is updated, while the visual encoder and LLM weights remain frozen. This single-epoch training expands the vocabulary of aligned image-text tokens specific to the radiology domain.
Fine-tuning for Report Generation
In the second stage, we fine-tune the pre-trained LLM weights using LoRA technology. The visual encoder and adapter weights are kept frozen, while the model undergoes three epochs of training on the dataset. This phase focuses on enhancing the model’s ability to generate accurate and clinically relevant radiology reports.

For implementation details, see the Code Repo.

Evaluation

Dataset

Our models were fine-tuned and evaluated using the RRG24 dataset, hosted on the BioNLP ACL'24 platform. This dataset aggregates data from multiple sources, including MIMIC-CXR, CheXpert, PadChest, BIMCV-COVID19, and OpenI. The dataset statistics are summarized below:

Dataset	FINDINGS	IMPRESSIONS
Training	344,394	366,413
Validation	8,839	9,331
Test-Public	2,692	2,967
Test-Hidden	1,063	1,428

Results

Our models achieved competitive F1-RadGraph scores of 24.13 and 22.79 for the Findings and Impressions sections, respectively, on the public test set. On the hidden test set, the scores were 24.13 and 22.10, securing 4th place on the leaderboard at the time of submission. Additionally, the Bertscore results highlight the high-quality text generation capabilities of our models, with scores of 53.45 for Findings and 47.39 for Impressions on the public test set.

Discussion

The difference in performance between the Findings and Impressions sections comes down to their unique roles. Findings are all about objectively describing what’s seen in the images, while Impressions focus on drawing diagnostic conclusions. This difference, along with the varying lengths of these sections, makes generating Impressions a bit trickier, which is reflected in the evaluation scores.

Another factor is the variety of diseases in the test set. This uneven distribution can make it harder for the model to generalize well. Plus, when the model processes multiple images at once, it sometimes includes irrelevant ones, which can throw off its accuracy. Improving the model’s ability to filter out unnecessary images could make a big difference.

Looking ahead, there’s a lot of room for improvement. Fine-tuning the model specifically for medical data and adapting it to better handle domain-specific challenges could help. Adding the ability to track changes over time and creating smarter frameworks for multi-modal reports are also exciting directions to explore. These upgrades could make the model even more useful and reliable in real-world clinical settings.

Conclusion

In this study, we introduced a model designed to generate radiology reports by aligning visual and textual data. By fine-tuning it for specific tasks, we managed to achieve solid results, including a fourth-place finish in the RRG24 Shared Task at BioNLP 2024. This shows the potential of our approach for specialized medical applications.

Moving forward, we’re excited to explore ways to make the model even better. This includes developing methods to handle multi-modal data more effectively and incorporating time-based insights. These improvements could take the model’s accuracy and practicality to the next level for clinical use.

Limitations

While the results are promising, there are a few challenges we need to address:

Easier-to-Detect Conditions: Some diseases are simpler to identify, which might inflate the evaluation scores.
Dataset Imbalance: The training data isn’t perfectly balanced in terms of imaging types, body parts, or report lengths, which could impact performance.
Inconsistent Reporting Styles: Radiologists have different ways of writing reports, and this variability makes it harder for the model to generate consistent outputs.

The data used for the training processes were exclusively sourced from the provided dataset, which complies with ethical standards regarding Patient Health Information. Adherence to these standards guarantees the responsible handling of sensitive data throughout our research.

BibTeX

@inproceedings{zhang-etal-2024-gla,
 title = "Gla-{AI}4{B}io{M}ed at {RRG}24: Visual Instruction-tuned Adaptation for Radiology Report Generation",
 author = "Zhang, Xi and
 Meng, Zaiqiao and
 Lever, Jake and
 Ho, Edmond S.L.",
 editor = "Demner-Fushman, Dina and
 Ananiadou, Sophia and
 Miwa, Makoto and
 Roberts, Kirk and
 Tsujii, Junichi",
 booktitle = "Proceedings of the 23rd Workshop on Biomedical Natural Language Processing",
 month = aug,
 year = "2024",
 address = "Bangkok, Thailand",
 publisher = "Association for Computational Linguistics",
 url = "https://aclanthology.org/2024.bionlp-1.54/",
 doi = "10.18653/v1/2024.bionlp-1.54",
 pages = "624--634",
}