Biomedical Multimodal LLM | Xi Zhang

Libra - What about next?🛸

Fri, 25 Apr 2025 00:00:00 +0000

Beyond Temporal Comparison: The Future of Radiology Modeling

In our previous discussions, we delved into how Libra leverages temporal information through its innovative Temporal Alignment Connector (TAC) to enhance radiology report generation. While this approach has shown significant promise, it’s essential to look ahead and consider how radiology modeling can evolve further to meet the complex demands of clinical practice.

The Temporal Alignment Connector has proven effective for handling paired images, but the future of radiology AI extends far beyond just temporal comparison.

1. Embracing Multimodal Integration

Radiological diagnosis doesn’t occur in isolation. Clinicians often consider a plethora of data—ranging from patient history and laboratory results to various imaging modalities. The future of radiology modeling lies in the seamless integration of these diverse data sources.

Clinical Contextualization

Incorporating electronic health records (EHRs), lab results, and patient histories can provide models with a richer context
Leading to more accurate and personalized diagnostics
Reducing false positives and negatives through contextual awareness

Cross-Modality Analysis

Combining data from different imaging modalities (e.g., CT, MRI, PET) offers a more comprehensive view
Enables detection of patterns that might be missed when analyzing a single modality
Creates synergistic understanding of complex pathologies

graph TD A[Patient Data] --> B{Multimodal
Integration} C[Chest X-ray] --> B D[CT Scan] --> B E[Lab Results] --> B F[Patient History] --> B B --> G[Comprehensive
Analysis] G --> H[Enhanced
Diagnostic Accuracy] G --> I[Personalized
Treatment Plans] G --> J[Early Disease
Detection] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#f5f5f5,stroke:#333,stroke-width:1px style D fill:#f5f5f5,stroke:#333,stroke-width:1px style E fill:#f5f5f5,stroke:#333,stroke-width:1px style F fill:#f5f5f5,stroke:#333,stroke-width:1px style G fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style H fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style I fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style J fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px

2. Advancing Explainability and Trustworthiness

As AI models become more integral to clinical decision-making, their interpretability becomes paramount. Clinicians need to understand the rationale behind a model’s prediction to trust and effectively utilize its insights.

In healthcare, trust isn’t optional—it’s essential. An AI system that can’t explain its reasoning is a black box that most physicians will rightfully hesitate to rely on.

Explainable AI (XAI)

Developing models that provide clear, human-understandable explanations for their predictions
Bridging the gap between AI outputs and clinical reasoning
Using attention visualization and feature attribution methods to highlight decision factors

Even the most accurate model will face adoption challenges if clinicians cannot verify its reasoning or understand how it arrived at its conclusions.

Uncertainty Quantification

Implementing mechanisms to convey confidence levels enables:

Clinicians to assess the reliability of AI-assisted diagnostics
Appropriate intervention in cases of model uncertainty
Continuous improvement through focused retraining on uncertain cases

3. Ensuring Robustness and Generalizability

AI models must perform reliably across diverse patient populations and clinical settings—a challenge that extends beyond academic validation to real-world implementation.

Diverse Training Data

- Building Robust Radiology AI
- Data Diversity Dimensions
- Demographic Factors
- Age groups
- Ethnic backgrounds
- Sex and gender representation
- Clinical Variables
- Disease prevalence variations
- Comorbidity patterns
- Treatment history diversity
- Technical Variability
- Multiple scanner manufacturers
- Various imaging protocols
- Quality and resolution differences
- Implementation Strategies
- Federated Learning
- Cross-institution collaboration
- Privacy-preserving techniques
- Data Augmentation
- Synthetic minority examples
- Domain randomization
- Continuous Validation
- Geographic generalization testing
- Temporal drift monitoring

Continuous Learning

Implementing systems that update from new clinical data
Adapting to evolving medical knowledge and practices
Maintaining performance as disease patterns and imaging technologies change

4. Integrating into Clinical Workflows

For AI models to be truly effective, they must integrate seamlessly into existing clinical workflows rather than disrupting established processes.

User-Friendly Interfaces

Designing intuitive interfaces that present AI insights clearly
Ensuring actionable information is immediately accessible
Minimizing cognitive load during busy clinical sessions

Workflow Compatibility

The ideal radiology AI system should:

Complement rather than replace radiologist expertise
Reduce administrative burden through automatic report generation
Prioritize cases based on urgency and findings

The most advanced AI system will fail if it adds steps to an already complex workflow. Success depends on making the radiologist’s job easier, not more complicated.

5. Ethical and Regulatory Considerations

As AI becomes more prevalent in healthcare, addressing ethical and regulatory challenges becomes essential for responsible implementation.

Data Privacy and Security

Safeguarding patient data through robust encryption
Ensuring compliance with regulations like HIPAA and GDPR
Implementing federated learning approaches to minimize data sharing

Regulatory Approval

Navigating the complex regulatory landscape (FDA, CE marking)
Designing validation studies that meet regulatory requirements
Establishing monitoring systems for post-deployment performance

Ethical AI Development

graph TD A[Ethical AI Development] --> B[Fairness & Bias Mitigation] A --> C[Transparency & Explainability] A --> D[Privacy Protection] A --> E[Human Oversight] B --> F[Equitable Healthcare Outcomes] C --> G[Informed Clinical Decisions] D --> H[Patient Trust & Confidentiality] E --> I[Safe AI Implementation] style A fill:#e1f5fe,stroke:#01579b,stroke-width:2px style B fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style D fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style E fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style H fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style I fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px

Conclusion: The Road Ahead for Libra

The journey of Libra represents a significant step forward in radiology modeling, particularly in harnessing temporal information through the TAC architecture. However, the path ahead involves:

Expanding beyond paired chest X-rays to multiple imaging modalities
Enhancing explainability through attention visualization and reasoning paths
Building more robust models through diverse training strategies
Designing intuitive interfaces for seamless clinical integration
Navigating ethical and regulatory requirements for real-world deployment

As we continue to develop Libra and similar technologies, our focus remains on augmenting—rather than replacing—clinical expertise, creating tools that serve as trusted partners in the complex art of radiological diagnosis.

💬 Note: The views expressed here are my own, reflecting my personal insights into the evolving landscape of radiology AI.

Libra – Structural Logic 🧠

Tue, 15 Apr 2025 00:00:00 +0000

The Challenge of Temporal Reasoning in Radiology AI

When dealing with radiology images, especially in the context of temporal analysis—comparing current chest X-rays with previous images—standard neural network architectures often struggle. Although transformer-based multimodal large language models (MLLMs) like LLaVA demonstrate remarkable capabilities for understanding single images and textual information, they encounter substantial challenges when handling image pairs.

In my previous blog, I discussed in detail why a single prior chest X-ray is typically sufficient for accurate diagnosis and patient triage.

However, capturing meaningful temporal differences between two images remains problematic with traditional transformer structures.

When Transformers Lose the Plot: Why They Struggle with Temporal

The transformer, the cornerstone of modern large language models (LLMs), excels at sequential data processing and logical reasoning tasks. Its strength lies in handling complex linguistic structures through positional encoding, enabling nuanced relationships in textual sequences.

However, when transformers receive visual information—particularly multiple images presented simultaneously—the situation becomes more complicated. Existing methods typically concatenate image features directly into the LLM’s head, often via sequences containing hundreds of visual tokens (patch tokens), depending on the specific image encoder used.

This straightforward approach inevitably suffers from token overload, known colloquially as the “lost-in-the-middle” problem, meaning crucial temporal details may get diluted or overlooked.

Indeed, current MLLMs like LLaVA perform impressively with single-image inputs. But they quickly become overwhelmed with paired images, heavily relying on meticulously crafted instruction datasets to guide temporal comparisons explicitly:

How MLLMs Are Prompted to Compare Images

“What is the difference between <image-1-patchholder> and <image-2-patchholder>?”

Such approaches place the burden squarely on the LLM’s internal reasoning and positional encodings, complicating training and diminishing reliability. The model must:

Distinguish between multiple images using only position encodings
Process 500+ tokens per image (depending on patch number)
Compare features across long token distances

Given these limitations, an essential question arises:

Can we overcome these temporal reasoning challenges structurally, rather than through explicit prompting?

Structure Determines Function: Insights from Biology

Before we answer above question, let’s briefly reflect on the foundational relationship between structure and function—deeply ingrained in biological systems.

Macro-scale examples:

Birds have wings enabling flight
Fish possess gills allowing them to breathe underwater

Micro-scale examples:

The unique three-dimensional helical structure of proteins directly determines their biological roles
A virus’s outer shell dictates its infection pathways and interaction mechanisms

Clearly, function is fundamentally dependent on structure.

When designing novel neural architectures or modules, we must apply this principle:

Identify the desired functionality first
Then craft an appropriate structural design that inherently supports these functions

Libra’s Structural Innovation

Temporal Alignment Connector (TAC)

Following this logic, we developed the TAC in our Libra model. TAC’s primary goal is to automatically and effectively capture the relationship between two chest X-ray images—the current image (primary) and a prior image (auxiliary).

Libra’s Temporal Alignment Connector (TAC) architecture.

Unlike traditional transformers that treat all inputs equivalently, TAC explicitly structures interactions between paired images. It captures their nuanced relationship through two key modules:

- TAC Architecture
- Layerwise Feature Extractor (LFE)
- Aggregates visual features across multiple encoder layers
- Ensures rich representations from both images
- Maintains feature hierarchy information
- Temporal Fusion Module (TFM)
- Fuses features from current and prior images
- Highlights critical temporal differences
- Maintains clear image role assignment
- Current image (Primary)
- Prior image (Reference)
- Prefix Bias Mechanism
- Addresses nearly-identical image pairs
- Prevents attention collapse
- Differentiates prior image's contextual influence

An important structural consideration is the integration of a prefix bias mechanism. This component addresses the scenario where current and prior images are nearly identical—common in clinical practice. Without careful design, such similarity can cause attention mechanisms to collapse into redundant self-attention loops.

The prefix bias mitigates this risk by clearly differentiating the prior image’s contextual influence, ensuring meaningful training and robust inference.

Why Structure Matters: The Libra Advantage

By structurally encoding temporal relationships directly into the neural network’s architecture, Libra overcomes the limitations inherent in traditional prompting-based approaches. Instead of forcing the LLM to implicitly infer temporal differences through complex positional encodings and exhaustive instruction tuning, TAC explicitly and efficiently captures this essential clinical context.

Libra exemplifies the powerful concept that structural logic, thoughtfully aligned with functional requirements, dramatically enhances model performance.

This structural logic not only simplifies training but also improves:

Reliability: More consistent temporal reasoning
Interpretability: Clearer connection between features and outputs
Efficiency: Reduced dependence on instruction tuning
Clinical Alignment: Better reflection of radiologists’ actual workflow

🏄 Note: The opinions shared here reflect my own understanding and are intended to convey the structural logic behind Libra. For technical accuracy and complete details, please refer to our paper: “Libra: Leveraging Temporal Images for Biomedical Radiology Analysis”.

Libra - Temporal Insight 🕰️

Sat, 05 Apr 2025 00:00:00 +0000

What Does “Temporal” Really Mean?

In clinical radiology, temporal information is not just about “past” and “present” — it’s about change. When radiologists assess a chest X-ray, they’re not merely describing what they see in a single image; they’re often comparing it to a previous one to identify whether a patient’s condition has improved, worsened, or remained stable.

🔔 This kind of temporal reasoning is essential in everyday medical practice. Yet most multimodal large language models (MLLMs) either ignore it or fail to model it effectively.

Temporal information in radiology is fundamentally about capturing change over time, not simply collecting a series of static images. This concept is central to Libra’s design philosophy.

Time Tells the Truth: Interpreting Temporal Changes in Imaging

1. Macro-Level Progression

Is the patient improving, deteriorating, or stable?

Macro-level comparison focuses on the overall trajectory of the patient’s condition compared to prior examinations. This high-level temporal reasoning is crucial for tracking disease evolution and guiding clinical decision-making.

2. Lesion-Specific Temporal Changes

How are individual abnormalities evolving over time?

Fine-grained analysis captures precise changes in specific findings, such as “consolidation in the left lower lobe has significantly expanded” or “cardiac silhouette shows no appreciable change.” These insights enable clinicians and models to reason at the level of targeted anatomical and pathological detail.

3. Quality Over Quantity in Temporal Inputs

❗️ Adding more images often introduces noise and computational complexity without improving diagnostic value.

While it may seem intuitive that incorporating more historical images would yield better results, the reality of clinical practice shows that the most valuable temporal information comes from comparing just two key points in time.

Clinical Applications

From Diagnostic Judgement to Multi-Scale Temporal Understanding

Temporal Reasoning in Clinical Diagnosis

Radiologists rarely analyse a chest X-ray in isolation. Instead, they routinely ask:

“Has the consolidation improved since last week?”
“Is the pleural effusion new?”
“Has the cardiac silhouette changed?”

Such reasoning typically falls into three primary categories:

Improved: Lesions have shrunk or resolved.
Worsened: New abnormalities appear, or existing ones have grown.
Stable: No meaningful change is observed.

These are coarse-level temporal descriptions. On a finer level, radiologists describe:

How much a lesion has changed in size or density,
Whether opacities have shifted,
If tubes, lines, or devices have been added or removed.

Temporal Signals Span Multiple Scales

Temporal information in radiology is inherently multi-scale — ranging from global clinical trajectories to subtle, localised anatomical changes.

- Temporal Information in Radiology
- Clinical Categories
- Improved
- Lesions have shrunk
- Opacities decreased
- Inflammatory shadows reduced
- Worsened
- New abnormalities appeared
- Existing lesions grown
- New infiltrates or effusion
- Stable
- No meaningful change
- Chronic conditions
- Continuous monitoring needed
- Information Levels
- Macro-level trends
- Overall patient trajectory
- Global comparison
- Local lesion changes
- Size changes
- Density variations
- Positional shifts
- Clinical Applications
- Triage decisions
- Emergency prioritization
- Resource allocation
- Treatment evaluation
- Response assessment
- Therapy adjustment
- Long-term monitoring
- Chronic disease management
- Post-surgical follow-up

Triage and Resource Allocation

Prioritising Care When Every Minute Counts

Clinical Goals and Operational Pressures

Chest X-rays play a pivotal role in patient triage, especially in emergency and high-volume settings. Radiologists must rapidly determine:

Which patients require immediate intervention,
Who can safely wait,
And how to allocate limited resources most effectively.

Triage is fundamentally about maximising outcomes under constraint. The goal is not to fully characterise every patient’s history, but to make fast, high-impact decisions that ensure critical cases receive timely care—without neglecting those with non-urgent needs.

What Matters Most: Clinically Significant Change

In these time-sensitive settings, timeliness and diagnostic clarity outweigh completeness. Radiologists focus on:

New acute findings: Abnormalities not previously seen that may indicate emerging crises.
Significant deterioration: Rapid worsening of known conditions that may demand escalated care.
Stable chronic findings: Ongoing issues that show no meaningful progression and can be managed routinely.

The most actionable form of temporal information in triage is the clinically meaningful change between now and the previous relevant image.
This focused comparison empowers decision-making without overwhelming clinicians with unnecessary historical data.

Our Model Approach

Temporal Efficiency Aligned with Clinical Reasoning

Why Two Images Are Sufficient

In real-world radiology workflows, the most informative temporal comparison is typically between:

The current chest X-ray, and
The most recent prior image used for diagnosis.

While patients may have a rich archive of historical scans, only the immediately preceding diagnostic image provides the relevant baseline for interpreting new findings. Additional older images may support longitudinal studies, but they often introduce noise, redundancy, and delay in fast-paced clinical decision-making.

Libra’s Design Philosophy: Focused Temporal Reasoning

Libra is built on this clinically grounded principle.
Instead of processing full temporal sequences — which can be computationally expensive and semantically ambiguous — Libra learns to model directional change between two key time points.

This design enables the model to:

Mimic the focused comparison strategies of expert radiologists,
Avoid temporal noise from irrelevant or outdated scans,
Reduce computational load while preserving diagnostic fidelity.

In short, Libra treats temporal reasoning as radiologists do:
What’s changed since the last meaningful image?

Illustrations

To better understand how radiologists and Libra approach temporal comparison, we present two illustrative workflows: a general conceptual flow and a specific clinical case.

Conceptual Workflow

The following diagram outlines the reasoning pathway taken when comparing a current chest X-ray to the most recent prior image. Based on observed changes (or lack thereof), radiologists infer the clinical trajectory and guide downstream decisions.

graph TD A[Previous Chest X-ray] --> B{Current vs Previous
Analysis} B -->|Opacity Size Decrease| C[Improved] B -->|New Infiltrates or Growth| D[Worsened] B -->|No Observable Change| E[Stable] C --> F[Recovery or
Treatment Response] D --> G[Disease
Progression] E --> H[Continuous
Monitoring Needed] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style D fill:#ffebee,stroke:#c62828,stroke-width:2px style E fill:#fff8e1,stroke:#ff8f00,stroke-width:2px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#ffcdd2,stroke:#c62828,stroke-width:1px style H fill:#ffecb3,stroke:#ff8f00,stroke-width:1px

Case Example: Lung Consolidation

This diagram demonstrates a practical example: a patient with lung consolidation. Depending on the direction of change, clinical interpretation and management decisions vary significantly.

graph TD A[Case: Lung Consolidation] --> B{Time Point Comparison} B -->|Consolidation Reduced
Clearer Lung Fields| C[Improved] B -->|Consolidation Expanded
New Pleural Effusion| D[Worsened] B -->|Consolidation Unchanged
No New Features| E[Stable] C --> F[Successful Antibiotic
Treatment] D --> G[Disease Progression
Treatment Adjustment Needed] E --> H[Continue Current
Management Plan] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style D fill:#ffebee,stroke:#c62828,stroke-width:2px style E fill:#fff8e1,stroke:#ff8f00,stroke-width:2px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#ffcdd2,stroke:#c62828,stroke-width:1px style H fill:#ffecb3,stroke:#ff8f00,stroke-width:1px

AI Model Implications

Why Most MLLMs Fall Short — and How Libra Goes Further

Many multimodal large language models (MLLMs) struggle with temporal reasoning for three key reasons:

No awareness of time: They treat images independently and ignore their chronological order.
Hallucinated references: They fabricate prior findings without reliable comparison.
Lack of temporal alignment: They have no built-in mechanism to align or contrast image features across time.

Libra tackles these issues head-on.

Instead of prompting the model to “guess” what might have changed, Libra incorporates explicit temporal awareness into both its architecture and training process.

We feed the model structured, temporally aligned visual features extracted from the current and previous images. This enables Libra to:

Detect fine-grained, clinically meaningful changes,
Avoid hallucination,
And reason about progression or stability in a way that mirrors clinical thinking.

In the next section, Libra – Structural Logic 🧠, we’ll explore how Libra’s architecture is intentionally designed to reflect clinical reasoning. You’ll see how its modular structure enables it to reason across both time and image features with precision — building on the foundation established in Libra – Temporal Insight 🕰️.