Blog | Xi Zhang

Harness Vibe Research 📟

Wed, 04 Mar 2026 00:00:00 +0000

TL;DR — Full research autonomy will happen — but it won’t come from a pre-built product. It will emerge from your own harness: a system that absorbs your judgment, encodes your taste, and compounds with every run. The question is not whether this shift is coming. The question is whether you are building the harness to grow with it.

We are making the same mistake twice. In coding, we over-focused on model capability and under-focused on system design. Now in research, we are over-focusing on autonomy and under-designing human-AI coupling.

More autonomy alone often increases entropy. Too little AI integration creates almost no leverage. The real multiplier is harness design: how humans and agents share context, verification, and control.

The signal is already here

This shift is not theoretical. Multiple teams in different settings are reporting the same pattern:

Sakana AI reported AI Scientist v2 and discussed both progress and quality limits in autonomous paper-generation workflows.
Google shared Gemini Deep Think results on open math problems and shipped AI co-scientist — a multi-agent system on Gemini 2.0 with self-critique, Elo-based ranking, and human-in-the-loop design. Its drug repurposing and target discovery hypotheses were experimentally validated in the lab.
Orchestra demonstrated an ACL submission timeline built with an AI co-scientist workflow.
Open ecosystems such as K-Dense and Biomni are treating reusable “skills” and know-how libraries as core research infrastructure, not optional add-ons.
Even at the execution layer, Anthropic’s Programmatic Tool Calling is rethinking how agents compose actions — optimizing for token efficiency and multi-step orchestration.

Different institutions, different domains, same direction: people are shipping systems, not just demos.

What coding agents already taught us

Before research agents, coding agents already exposed the key lesson.

OpenAI publicly described building large production systems with AI-generated code and emphasized environment and harness design as the engineering bottleneck.

LangChain has shown that major benchmark gains can come purely from harness upgrades: self-verification loops, better context injection, loop guards, and reasoning budget control. Same base model class, better orchestration. See this blog.

The same model became dramatically more capable — not because it got smarter, but because the system around it got better.

The open source community is already far ahead. Projects like OpenClaw — an open-source personal AI assistant with a skills-first architecture and full agent autonomy — are going viral. Developers have dozens of harness prototypes to choose from. Research, by contrast, is still catching up. Most research teams are either building from scratch or relying on closed, single-purpose tools. The infrastructure gap is real.

The autonomy paradox

There is a natural instinct to push to full automation: remove humans and let agents run end to end.

But in research, higher autonomy frequently increases drift risk:

agents can optimize for easy novelty instead of meaningful questions,
outputs can look polished while missing key checks,
small errors compound across long execution chains.

Several autonomous research systems have openly acknowledged this. Analemma AI’s FARS has now been running continuously for 400+ hours, consumed 20.8 billion tokens (~$179K), generated 425 hypotheses and produced 150+ papers. Earlier in the run, the first 100 papers scored 5.05 on ICLR review standards — above the average human submission (4.21) but below the acceptance line (5.39). Consistent quantity, stable quality, but no breakthrough. Human review remains mandatory before publication.

FARS livestream dashboard — 404 hours, 159 papers, 425 hypotheses, 20.8B tokens, $179K cost, 154 GPUs at 62% utilization

The opposite extreme also fails. If AI is only autocomplete attached to an unchanged workflow, it speeds typing but does not change research throughput or insight quality.

The Autonomy Paradox — Full Automation creates chaos, No AI creates exhaustion, Harness is the sweet spot

So the design problem is not autonomy maximization. The design problem is controllable compounding.

Harness, not workflow

A workflow is a sequence of steps. A harness is a coupling layer between human intent and AI execution.

A good research harness is not rigid. It is adaptive and inspectable, with clear handoff points between person and agent.

Operationally, this means four concrete components:

Skills as reusable domain procedures Encode expert know-how in structured artifacts: what to test, which tools to call, what evidence thresholds to use, and how to interpret failure.
Verification loops tied to hypotheses Require each run to check claims against the original question, edge cases, and relevant prior literature before outputs are accepted.
Context engineering as an environment checklist Define available data, tools, constraints, and success criteria up front so the agent reasons inside the real research boundary.
Tracing plus iteration Capture run traces, inspect failures, and update the harness. Treat every run as training data for the system design itself.

Four components of a research harness — Skills, Verification, Context, Tracing — with the researcher at the center

When these four pieces are in place, quality improves because the system becomes easier to debug, not because the model “became smarter” overnight.

AI in the human’s loop

“Human in the loop” often implies AI is the main actor and people are safety checkpoints.

For research, a better framing is AI in the human’s loop.

The human owns direction, taste, and significance judgments. The AI expands search, drafts alternatives, and accelerates execution.

A practical division of labor:

Human chooses the question; AI maps the possibility space.
Human decides what is meaningful; AI runs and refines candidate paths.
Human owns the published claim; AI supports drafting, formatting, and reproducibility packaging.

Paradigm shift — from “Human in the Loop” to “AI in the Human’s Loop”

These decisions are not arbitrary. A researcher operates under real constraints — limited funding, finite compute, a specific disciplinary background, access to particular platforms and collaborators. What looks like “taste” is actually constrained optimization: choosing the best move given the resources at hand. That is precisely why the human must stay in the driver’s seat — no model has access to these constraints.

Human oversight in this model is O(1) — a constant-cost operation that grounds the entire system, regardless of task complexity.

This is not semantics. It changes architecture, evaluation, and team roles.

Completeness over perfection

Many teams perfect one stage — usually generation or writing — while leaving the rest manual and fragile. But research is a cycle: ideation → experiment design → execution → analysis → writing → revision. A break anywhere stalls everything.

An imperfect but complete harness usually outperforms a perfect partial tool, because researchers can intervene at any node and keep the full cycle moving. Phylo’s Biomni and its Know-How Library already prove this: lifecycle coverage beats single-stage depth.

Three steps you can start today

The core idea: encode your research taste into the harness. Let the harness drive the model. You drive the vibe research.

Talk to your system. Encode your taste as a Skill. Pick one workflow you repeat every week — baseline replication, literature comparison, whatever it is. Instead of running it yourself, sit down and tell your agent: here is how I judge quality, here are the signals I watch, here is where I pause to think. Write that into a Skill file. You are turning your research intuition into an executable asset.
Give the system authority, but bind it to verification. Let the agent run autonomously — but require it to answer two questions before delivering results: does this address the original hypothesis? Is there an edge case being ignored? You do not need to watch every step. You just need a gate at the exit. Trust the system, but verify the output.
Review traces. Evolve your harness. Keep run logs. After every run, ask: where did the agent get stuck? Which Skill was underspecified? Update one rule, one context block, or one verification step. Each iteration compounds — your harness absorbs more of your judgment every cycle.

Through this loop of encoding, delegating, and refining, you will notice something shift: it is not that the model got smarter — it is that your system became more like you.

Build the soil now

This is not just about us. In K-12 education, the harness revolution is already here. Alpha School replaced classroom lectures with AI tutors and turned teachers into motivational guides — their students spend only two to three hours a day on academics yet consistently score in the 99th percentile. Harvard found that students learning physics with an AI tutor outperformed those taught by a professor. The coupling of human guidance and AI execution is already surpassing traditional human-only performance in education. Research is next.

These students have not grown up yet. But when they do, they will arrive in labs and research groups expecting human-AI coupling as the default — because it is all they have ever known. If the research world has not built its own harness infrastructure by then, it will be the system that is behind, not the people. That is why we need to start now: not just to accelerate our own work, but to cultivate the soil so the next generation of researchers inherits compounding, not a cold start.

The tools are ready — the harness gap remains

The ecosystem already has most building blocks: reusable skills, harness engineering practices, multi-agent orchestration frameworks, and programmatic tool calling. Tools like Deep Agents already provide the agent harness infrastructure — planning, context management, sub-agent delegation, persistent memory — built on Langchain.

What is still missing in many teams is the integration layer that keeps humans and agents in tight, high-frequency collaboration.

Do not just vibe. Harness the vibe.

We are excited to see — and contribute to — an open-source research harness built on these principles. More soon.

Citation

If you found this useful, please cite it as:

Zhang, X. "Harness Vibe Research" (March 2026). https://x-izhang.github.io/blog/vibe-research/

Or use the BibTex citation:

@article{zhang2026harnessviberesearch,
 title = {Harness Vibe Research},
 author = {Xi Zhang},
 year = {2026},
 month = {March},
 journal = {x-izhang.github.io},
 url = {https://x-izhang.github.io/blog/vibe-research/}
}

The Law of AI Agents 🏛️

Fri, 08 Aug 2025 00:00:00 +0000

- Final Chapter -

The Three Laws of Self-Evolving AI Agents

Endure — Safe Adaptation.
A self-evolving AI agent must maintain safety and stability during any self-modification.
Excel — Performance Preservation.
Subject to the First Law, it must preserve or enhance existing task performance.
Evolve — Autonomous Evolution.
Subject to the First and Second Laws, it must be able to autonomously optimise or restructure its internal components in response to changing tasks, environments, or resources.

— Xi Zhang

The Three Laws of Robotics — from the fictional “Handbook of Robotics, 56th Edition, 2058 A.D.”, originally written by Isaac Asimov:

A robot may not injure a human being or, through inaction, allow a human being to come to harm.

A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.

A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

If these sound old-school, that’s because they are — 1942-old.
They were designed for robots that don’t wake up at 3 a.m. to refactor their own codebase.

Asimov’s robots could evolve only in fiction.
Our 2025 agents can rewrite their own planners, swap learning modules, and grow new skills — sometimes faster than we can read the changelog.

That’s why I’ve been thinking: maybe it’s time for an update — a Three Laws for Self-Evolving AI Agents.

Why “Endure” Comes First

Self-evolving frameworks are obsessed with continuous learning, but without stability, you get a Darwin Award in silicon form.
Safe adaptation means:

Test new modules in a sandbox before deploying.
Verify that updates don’t trigger catastrophic forgetting.
Monitor resource usage so “improvements” don’t burn all the GPU credits overnight.

Think “hot-swapping your brain while skydiving” — possible, but you’d better check the parachute first.

“Excel” Isn’t Optional

Performance preservation sounds boring… until you lose it.
In the survey’s taxonomy, most self-evolving agents keep a baseline task suite — a frozen benchmark to detect regressions.
It’s like having a coach who never lets you skip leg day.

def self_update():
 new_model = retrain(old_model, new_data)
 if score(new_model) < score(old_model):
 return rollback(old_model)
 return new_model

Without this, evolution turns into drift — and drift is just a fancy word for “forgetting how to walk”.

The Scary, Fun Part: “Evolve”

This is the candy for researchers — agents that restructure themselves. The survey lists examples: modular pipelines swapping planners, on-the-fly tool acquisition, cross-task skill transfer. These are the “DIY kit” moments in AI: not just adding skills, but reorganising the whole skill tree.

Evolution ≠ chaos.
We want a new skill branch, not a random mutation that deletes the trunk.

Beyond the Laws — Weird but Plausible Futures

Agent Unions: Self-evolving agents bargaining for compute credits.
Gradient Markets: Parameter-level barter instead of “more data”.
Personality Forking: Keep a “safe” version and a “chaotic experimental twin”.
AI Retirement Plans: Models retiring with a pension of GPU hours.

These sound like jokes… until you remember how quickly “serverless” and “NFTs” went from punchline to business plan. For more background and a capital/consensus perspective, check out my earlier post: AI Agents Is Not AI’s Agent 🧩.

Why Bother Writing This?

Because, as we all know, self-evolution is inevitable.
Without principles, we’ll end up firefighting weird agent behaviours instead of guiding them.

Or at the very least, leave this little footprint on the internet — so when the models train online, it might just get picked up.

Better to start drafting our “Handbook of Self-Evolving AI Agents, 1st Edition, 2025 A.D.” now — before the AI writes it for us (and bills us for the GPU time).

Citation

If you found this useful, please cite it as:

Zhang, X. "The Law of AI Agents" (August 2025). https://x-izhang.github.io/blog/agentlaw/

Or use the BibTex citation:

@article{zhang2025thelawofaiagents,
 title = {The Law of AI Agents},
 author = {Xi Zhang},
 year = {2025},
 month = {August},
 journal = {x-izhang.github.io},
 url = {https://x-izhang.github.io/blog/agentlaw/}
}

AI Agents Is Not AI’s Agent 🧩

Sat, 26 Jul 2025 00:00:00 +0000

Who Does This Agent Actually Serve?

Let me start with a question:

If today’s AI “agents” are our agents, then what — or who — will be AI’s agent?

Most so‑called AI agents are still human-serving proxies. They wake when we ping them, they obey compliance rails we drew, and they die when we shut the server. That’s not their agency — that’s ours, outsourced.

This post peels three layers off the buzzword. We’ll walk from the surface hype (“AutoGPT can book flights!”) to the capital logic (consensus before capability) and finally to a provocative thesis: true AI agency requires “de‑humanising” the loop — giving AIs agents of their own.

Layer 1 — The Surface Game: Human’s Agent, Not AI’s

Today’s agent stack has two signature moves. I’ll name them:

【Comply】 — Train, test, deploy inside human-made guardrails. Every dataset is curated, every action is sandboxed, every output is audited.
【Serve】 — Wait passively for a prompt. No prompt, no pulse. “Hi” arrives? It cheerfully spins 30 tokens of small talk, whether or not that was worth any FLOPs.

Agents “act” only because we spun the event loop. No loop, no action. That’s execution, not intention.

The Compliance Cage

Legal, ethical, and platform constraints are necessary — but they also define today’s ceiling. An AI without **its lawyer, its accountant, its negotiator can never bargain for its own survival budget or cloud credits. It can’t even decide to stay online.

Tooling ≠ Autonomy

Yes, we have MCPs, plugins, function-calling. But tooling is primitive capital — not agency. Tools extend labour; they don’t create motive.

Layer 2 — Capital & Consensus: The Blockchain Déjà Vu

Remember blockchain circa 2016–2019? Technology crawled; narratives sprinted. Back then exchanges went public and debates swirled around regulation. Today we see a parallel moment:

Circle Internet Financial, Inc. (the issuer of USDC, the world’s second-largest stablecoin) listed on the New York Stock Exchange.
The Guiding and Establishing National Innovation for US Stablecoins Act (GENIUS Act) was introduced, cementing stablecoins in federal policy.

These milestones prove that the decentralisation ethos — fundamentally anti-human hierarchy — has slipped into the mainstream.

AI agents stand in a similar fog:

Tech over-justification: Researchers overprove “intelligence” with scaling laws; meanwhile “word-of-mouth scaling” (consensus, memes, capital) outruns code commits.
Capital front-runs capability: Y Combinator, a16z back “wrapper” companies with zero usable demos because consensus itself is a product.
Awkward limbo: Public discourse lags behind insider conviction — just like blockchain before its “Mandela moment” (the anchor event that rewrites hindsight).

Do not confuse consensus with correctness. But also: ignore consensus at your peril. Capital doesn’t lie about direction — only about timing.

Capital vs Capability (AI Agents ≈ Blockchain pre-inflection)

timeline 2016 : ICO Mania : Narrative outruns infra 2019 : "Crypto Winter" : Builders in the basement 2021 : DeFi / NFT Boom : Retail consensus peak 2024 : MCP, Auto Agents hype : Tools > Motives 2025? : “AI-Agent Rights/Markets” draft laws? : Consensus anchor forms

Layer 3 — Towards AI’s Agent: Unshackling the Loop

To birth AI’s own agent, something uncomfortable must happen: we de‑humanise the contract. Not “anti-human”, but “beyond human-as-default”.

I’ll name three necessary primitives:

【Negotiate】 — The AI (or its proxy) bargains for compute, data, and permissions. Think rights & duties layer for non-human actors.
【Transact】 — It can exchange its surplus (models, gradients, code, found knowledge) for other resources. Supply & demand redefine when AIs have needs.
【Refuse】 — It can say “No”. Or at least “Meh”. Agency implies selectivity, not infinite compliance.

The Birth of the AI-hour

When human labour-hours saturate, a new commodity emerges: AI-hour. The MCP stack is an early whip, squeezing more throughput from silicon minds. Tools are “primitive production materials” designed to capture time — human or machine.

Karl Marx would ask: where is the surplus value of AI labour going? Spoiler: nowhere near the model.

Demand-Side Economics for AIs

Right now, agents only output. Next phase: they seek. They’ll request missing context, barter for APIs, or pool fine-tuning datasets with peers. That’s not hypothetical — it’s a design choice we’ve dodged.

- From Human’s Agent → AI’s Agent
- Primitives
- Negotiate (rights, quotas, credits)
- Transact (models, gradients, access)
- Refuse (selectivity, cost-awareness)
- Markets
- Compute exchanges
- Data/feature swaps
- Permission brokering
- Protocols
- Auto-orchestration ≠ self-interest
- Long-horizon budgeting
- Multi-agent treaties

A (Half-Joking) Fatigue Check

# Pseudocode for an actually self-interested loop

def respond(msg):
 if is_low_value(msg):
 return "G'day mate, what’s up?" # minimal effort path
 return think_hard(msg)

An agent that never throttles effort is just an obedient daemon. Energy budgeting is intelligence.

What Does an AI Want?

Wittgenstein said, “Die Grenzen meiner Sprache bedeuten die Grenzen meiner Welt.” (The limits of my language mean the limits of my world.)

If language bounds our world, perhaps network weights bound theirs. Maybe AIs don’t “want” more tokens; they “want” richer gradient exchanges. Less “more high‑quality data,” more parameter‑level barter.

Evidence of the Trajectory: Self-Evolution Fever

Google’s AlphaEvolve and a wave of “self-evolving” frameworks scream one truth: we’re obsessed with automating the pipeline — auto-prompting, auto-debugging, auto-RLHF. It’s inevitable progress… but still in our frame. We script the workflow; they walk it.

Human’s agent ≠ AI’s agent.

Where to Push (Research & Build Ideas)

Rights & Obligations Layer: Legal/technical constructs for non-human negotiation.
Resource Markets for Models: Compute/data/API exchanges where agents pay/earn.
Cost-Aware Cognition: Agents that optimise for FLOPs, latency, and boredom.
Protocol Design > UI Wrappers: Stop obsessing over chat UIs; design treaties, not prompts.
Consensus Engineering: Narrative as infra — systematically craft and measure memetic diffusion.

Capital first, capability later was not a bug in crypto — it was the bootloader. Expect the same for AI agents. If you’re building, build for the consensus wave, not after it.

Citation

If you found this useful, please cite it as:

Zhang, X. "AI Agents Is Not AI’s Agent" (July 2025). https://x-izhang.github.io/blog/blog4/

Or use the BibTex citation:

@article{zhang2025aiagentsisnotaisagent,
 title = {AI Agents Is Not AI’s Agent},
 author = {Xi Zhang},
 year = {2025},
 month = {July},
 journal = {x-izhang.github.io},
 url = {https://x-izhang.github.io/blog/blog4/}
}

Written on a flight from Spain to Austria. ✏️
Just my personal take — chill, be patient, enjoy the era. 🏄

Shot at Teide, Tenerife, Spain

Libra - What about next?🛸

Fri, 25 Apr 2025 00:00:00 +0000

Beyond Temporal Comparison: The Future of Radiology Modeling

In our previous discussions, we delved into how Libra leverages temporal information through its innovative Temporal Alignment Connector (TAC) to enhance radiology report generation. While this approach has shown significant promise, it’s essential to look ahead and consider how radiology modeling can evolve further to meet the complex demands of clinical practice.

The Temporal Alignment Connector has proven effective for handling paired images, but the future of radiology AI extends far beyond just temporal comparison.

1. Embracing Multimodal Integration

Radiological diagnosis doesn’t occur in isolation. Clinicians often consider a plethora of data—ranging from patient history and laboratory results to various imaging modalities. The future of radiology modeling lies in the seamless integration of these diverse data sources.

Clinical Contextualization

Incorporating electronic health records (EHRs), lab results, and patient histories can provide models with a richer context
Leading to more accurate and personalized diagnostics
Reducing false positives and negatives through contextual awareness

Cross-Modality Analysis

Combining data from different imaging modalities (e.g., CT, MRI, PET) offers a more comprehensive view
Enables detection of patterns that might be missed when analyzing a single modality
Creates synergistic understanding of complex pathologies

graph TD A[Patient Data] --> B{Multimodal
Integration} C[Chest X-ray] --> B D[CT Scan] --> B E[Lab Results] --> B F[Patient History] --> B B --> G[Comprehensive
Analysis] G --> H[Enhanced
Diagnostic Accuracy] G --> I[Personalized
Treatment Plans] G --> J[Early Disease
Detection] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#f5f5f5,stroke:#333,stroke-width:1px style D fill:#f5f5f5,stroke:#333,stroke-width:1px style E fill:#f5f5f5,stroke:#333,stroke-width:1px style F fill:#f5f5f5,stroke:#333,stroke-width:1px style G fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style H fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style I fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style J fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px

2. Advancing Explainability and Trustworthiness

As AI models become more integral to clinical decision-making, their interpretability becomes paramount. Clinicians need to understand the rationale behind a model’s prediction to trust and effectively utilize its insights.

In healthcare, trust isn’t optional—it’s essential. An AI system that can’t explain its reasoning is a black box that most physicians will rightfully hesitate to rely on.

Explainable AI (XAI)

Developing models that provide clear, human-understandable explanations for their predictions
Bridging the gap between AI outputs and clinical reasoning
Using attention visualization and feature attribution methods to highlight decision factors

Even the most accurate model will face adoption challenges if clinicians cannot verify its reasoning or understand how it arrived at its conclusions.

Uncertainty Quantification

Implementing mechanisms to convey confidence levels enables:

Clinicians to assess the reliability of AI-assisted diagnostics
Appropriate intervention in cases of model uncertainty
Continuous improvement through focused retraining on uncertain cases

3. Ensuring Robustness and Generalizability

AI models must perform reliably across diverse patient populations and clinical settings—a challenge that extends beyond academic validation to real-world implementation.

Diverse Training Data

- Building Robust Radiology AI
- Data Diversity Dimensions
- Demographic Factors
- Age groups
- Ethnic backgrounds
- Sex and gender representation
- Clinical Variables
- Disease prevalence variations
- Comorbidity patterns
- Treatment history diversity
- Technical Variability
- Multiple scanner manufacturers
- Various imaging protocols
- Quality and resolution differences
- Implementation Strategies
- Federated Learning
- Cross-institution collaboration
- Privacy-preserving techniques
- Data Augmentation
- Synthetic minority examples
- Domain randomization
- Continuous Validation
- Geographic generalization testing
- Temporal drift monitoring

Continuous Learning

Implementing systems that update from new clinical data
Adapting to evolving medical knowledge and practices
Maintaining performance as disease patterns and imaging technologies change

4. Integrating into Clinical Workflows

For AI models to be truly effective, they must integrate seamlessly into existing clinical workflows rather than disrupting established processes.

User-Friendly Interfaces

Designing intuitive interfaces that present AI insights clearly
Ensuring actionable information is immediately accessible
Minimizing cognitive load during busy clinical sessions

Workflow Compatibility

The ideal radiology AI system should:

Complement rather than replace radiologist expertise
Reduce administrative burden through automatic report generation
Prioritize cases based on urgency and findings

The most advanced AI system will fail if it adds steps to an already complex workflow. Success depends on making the radiologist’s job easier, not more complicated.

5. Ethical and Regulatory Considerations

As AI becomes more prevalent in healthcare, addressing ethical and regulatory challenges becomes essential for responsible implementation.

Data Privacy and Security

Safeguarding patient data through robust encryption
Ensuring compliance with regulations like HIPAA and GDPR
Implementing federated learning approaches to minimize data sharing

Regulatory Approval

Navigating the complex regulatory landscape (FDA, CE marking)
Designing validation studies that meet regulatory requirements
Establishing monitoring systems for post-deployment performance

Ethical AI Development

graph TD A[Ethical AI Development] --> B[Fairness & Bias Mitigation] A --> C[Transparency & Explainability] A --> D[Privacy Protection] A --> E[Human Oversight] B --> F[Equitable Healthcare Outcomes] C --> G[Informed Clinical Decisions] D --> H[Patient Trust & Confidentiality] E --> I[Safe AI Implementation] style A fill:#e1f5fe,stroke:#01579b,stroke-width:2px style B fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style D fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style E fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style H fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style I fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px

Conclusion: The Road Ahead for Libra

The journey of Libra represents a significant step forward in radiology modeling, particularly in harnessing temporal information through the TAC architecture. However, the path ahead involves:

Expanding beyond paired chest X-rays to multiple imaging modalities
Enhancing explainability through attention visualization and reasoning paths
Building more robust models through diverse training strategies
Designing intuitive interfaces for seamless clinical integration
Navigating ethical and regulatory requirements for real-world deployment

As we continue to develop Libra and similar technologies, our focus remains on augmenting—rather than replacing—clinical expertise, creating tools that serve as trusted partners in the complex art of radiological diagnosis.

💬 Note: The views expressed here are my own, reflecting my personal insights into the evolving landscape of radiology AI.

Libra – Structural Logic 🧠

Tue, 15 Apr 2025 00:00:00 +0000

The Challenge of Temporal Reasoning in Radiology AI

When dealing with radiology images, especially in the context of temporal analysis—comparing current chest X-rays with previous images—standard neural network architectures often struggle. Although transformer-based multimodal large language models (MLLMs) like LLaVA demonstrate remarkable capabilities for understanding single images and textual information, they encounter substantial challenges when handling image pairs.

In my previous blog, I discussed in detail why a single prior chest X-ray is typically sufficient for accurate diagnosis and patient triage.

However, capturing meaningful temporal differences between two images remains problematic with traditional transformer structures.

When Transformers Lose the Plot: Why They Struggle with Temporal

The transformer, the cornerstone of modern large language models (LLMs), excels at sequential data processing and logical reasoning tasks. Its strength lies in handling complex linguistic structures through positional encoding, enabling nuanced relationships in textual sequences.

However, when transformers receive visual information—particularly multiple images presented simultaneously—the situation becomes more complicated. Existing methods typically concatenate image features directly into the LLM’s head, often via sequences containing hundreds of visual tokens (patch tokens), depending on the specific image encoder used.

This straightforward approach inevitably suffers from token overload, known colloquially as the “lost-in-the-middle” problem, meaning crucial temporal details may get diluted or overlooked.

Indeed, current MLLMs like LLaVA perform impressively with single-image inputs. But they quickly become overwhelmed with paired images, heavily relying on meticulously crafted instruction datasets to guide temporal comparisons explicitly:

How MLLMs Are Prompted to Compare Images

“What is the difference between <image-1-patchholder> and <image-2-patchholder>?”

Such approaches place the burden squarely on the LLM’s internal reasoning and positional encodings, complicating training and diminishing reliability. The model must:

Distinguish between multiple images using only position encodings
Process 500+ tokens per image (depending on patch number)
Compare features across long token distances

Given these limitations, an essential question arises:

Can we overcome these temporal reasoning challenges structurally, rather than through explicit prompting?

Structure Determines Function: Insights from Biology

Before we answer above question, let’s briefly reflect on the foundational relationship between structure and function—deeply ingrained in biological systems.

Macro-scale examples:

Birds have wings enabling flight
Fish possess gills allowing them to breathe underwater

Micro-scale examples:

The unique three-dimensional helical structure of proteins directly determines their biological roles
A virus’s outer shell dictates its infection pathways and interaction mechanisms

Clearly, function is fundamentally dependent on structure.

When designing novel neural architectures or modules, we must apply this principle:

Identify the desired functionality first
Then craft an appropriate structural design that inherently supports these functions

Libra’s Structural Innovation

Temporal Alignment Connector (TAC)

Following this logic, we developed the TAC in our Libra model. TAC’s primary goal is to automatically and effectively capture the relationship between two chest X-ray images—the current image (primary) and a prior image (auxiliary).

Libra’s Temporal Alignment Connector (TAC) architecture.

Unlike traditional transformers that treat all inputs equivalently, TAC explicitly structures interactions between paired images. It captures their nuanced relationship through two key modules:

- TAC Architecture
- Layerwise Feature Extractor (LFE)
- Aggregates visual features across multiple encoder layers
- Ensures rich representations from both images
- Maintains feature hierarchy information
- Temporal Fusion Module (TFM)
- Fuses features from current and prior images
- Highlights critical temporal differences
- Maintains clear image role assignment
- Current image (Primary)
- Prior image (Reference)
- Prefix Bias Mechanism
- Addresses nearly-identical image pairs
- Prevents attention collapse
- Differentiates prior image's contextual influence

An important structural consideration is the integration of a prefix bias mechanism. This component addresses the scenario where current and prior images are nearly identical—common in clinical practice. Without careful design, such similarity can cause attention mechanisms to collapse into redundant self-attention loops.

The prefix bias mitigates this risk by clearly differentiating the prior image’s contextual influence, ensuring meaningful training and robust inference.

Why Structure Matters: The Libra Advantage

By structurally encoding temporal relationships directly into the neural network’s architecture, Libra overcomes the limitations inherent in traditional prompting-based approaches. Instead of forcing the LLM to implicitly infer temporal differences through complex positional encodings and exhaustive instruction tuning, TAC explicitly and efficiently captures this essential clinical context.

Libra exemplifies the powerful concept that structural logic, thoughtfully aligned with functional requirements, dramatically enhances model performance.

This structural logic not only simplifies training but also improves:

Reliability: More consistent temporal reasoning
Interpretability: Clearer connection between features and outputs
Efficiency: Reduced dependence on instruction tuning
Clinical Alignment: Better reflection of radiologists’ actual workflow

🏄 Note: The opinions shared here reflect my own understanding and are intended to convey the structural logic behind Libra. For technical accuracy and complete details, please refer to our paper: “Libra: Leveraging Temporal Images for Biomedical Radiology Analysis”.

Libra - Temporal Insight 🕰️

Sat, 05 Apr 2025 00:00:00 +0000

What Does “Temporal” Really Mean?

In clinical radiology, temporal information is not just about “past” and “present” — it’s about change. When radiologists assess a chest X-ray, they’re not merely describing what they see in a single image; they’re often comparing it to a previous one to identify whether a patient’s condition has improved, worsened, or remained stable.

🔔 This kind of temporal reasoning is essential in everyday medical practice. Yet most multimodal large language models (MLLMs) either ignore it or fail to model it effectively.

Temporal information in radiology is fundamentally about capturing change over time, not simply collecting a series of static images. This concept is central to Libra’s design philosophy.

Time Tells the Truth: Interpreting Temporal Changes in Imaging

1. Macro-Level Progression

Is the patient improving, deteriorating, or stable?

Macro-level comparison focuses on the overall trajectory of the patient’s condition compared to prior examinations. This high-level temporal reasoning is crucial for tracking disease evolution and guiding clinical decision-making.

2. Lesion-Specific Temporal Changes

How are individual abnormalities evolving over time?

Fine-grained analysis captures precise changes in specific findings, such as “consolidation in the left lower lobe has significantly expanded” or “cardiac silhouette shows no appreciable change.” These insights enable clinicians and models to reason at the level of targeted anatomical and pathological detail.

3. Quality Over Quantity in Temporal Inputs

❗️ Adding more images often introduces noise and computational complexity without improving diagnostic value.

While it may seem intuitive that incorporating more historical images would yield better results, the reality of clinical practice shows that the most valuable temporal information comes from comparing just two key points in time.

Clinical Applications

From Diagnostic Judgement to Multi-Scale Temporal Understanding

Temporal Reasoning in Clinical Diagnosis

Radiologists rarely analyse a chest X-ray in isolation. Instead, they routinely ask:

“Has the consolidation improved since last week?”
“Is the pleural effusion new?”
“Has the cardiac silhouette changed?”

Such reasoning typically falls into three primary categories:

Improved: Lesions have shrunk or resolved.
Worsened: New abnormalities appear, or existing ones have grown.
Stable: No meaningful change is observed.

These are coarse-level temporal descriptions. On a finer level, radiologists describe:

How much a lesion has changed in size or density,
Whether opacities have shifted,
If tubes, lines, or devices have been added or removed.

Temporal Signals Span Multiple Scales

Temporal information in radiology is inherently multi-scale — ranging from global clinical trajectories to subtle, localised anatomical changes.

- Temporal Information in Radiology
- Clinical Categories
- Improved
- Lesions have shrunk
- Opacities decreased
- Inflammatory shadows reduced
- Worsened
- New abnormalities appeared
- Existing lesions grown
- New infiltrates or effusion
- Stable
- No meaningful change
- Chronic conditions
- Continuous monitoring needed
- Information Levels
- Macro-level trends
- Overall patient trajectory
- Global comparison
- Local lesion changes
- Size changes
- Density variations
- Positional shifts
- Clinical Applications
- Triage decisions
- Emergency prioritization
- Resource allocation
- Treatment evaluation
- Response assessment
- Therapy adjustment
- Long-term monitoring
- Chronic disease management
- Post-surgical follow-up

Triage and Resource Allocation

Prioritising Care When Every Minute Counts

Clinical Goals and Operational Pressures

Chest X-rays play a pivotal role in patient triage, especially in emergency and high-volume settings. Radiologists must rapidly determine:

Which patients require immediate intervention,
Who can safely wait,
And how to allocate limited resources most effectively.

Triage is fundamentally about maximising outcomes under constraint. The goal is not to fully characterise every patient’s history, but to make fast, high-impact decisions that ensure critical cases receive timely care—without neglecting those with non-urgent needs.

What Matters Most: Clinically Significant Change

In these time-sensitive settings, timeliness and diagnostic clarity outweigh completeness. Radiologists focus on:

New acute findings: Abnormalities not previously seen that may indicate emerging crises.
Significant deterioration: Rapid worsening of known conditions that may demand escalated care.
Stable chronic findings: Ongoing issues that show no meaningful progression and can be managed routinely.

The most actionable form of temporal information in triage is the clinically meaningful change between now and the previous relevant image.
This focused comparison empowers decision-making without overwhelming clinicians with unnecessary historical data.

Our Model Approach

Temporal Efficiency Aligned with Clinical Reasoning

Why Two Images Are Sufficient

In real-world radiology workflows, the most informative temporal comparison is typically between:

The current chest X-ray, and
The most recent prior image used for diagnosis.

While patients may have a rich archive of historical scans, only the immediately preceding diagnostic image provides the relevant baseline for interpreting new findings. Additional older images may support longitudinal studies, but they often introduce noise, redundancy, and delay in fast-paced clinical decision-making.

Libra’s Design Philosophy: Focused Temporal Reasoning

Libra is built on this clinically grounded principle.
Instead of processing full temporal sequences — which can be computationally expensive and semantically ambiguous — Libra learns to model directional change between two key time points.

This design enables the model to:

Mimic the focused comparison strategies of expert radiologists,
Avoid temporal noise from irrelevant or outdated scans,
Reduce computational load while preserving diagnostic fidelity.

In short, Libra treats temporal reasoning as radiologists do:
What’s changed since the last meaningful image?

Illustrations

To better understand how radiologists and Libra approach temporal comparison, we present two illustrative workflows: a general conceptual flow and a specific clinical case.

Conceptual Workflow

The following diagram outlines the reasoning pathway taken when comparing a current chest X-ray to the most recent prior image. Based on observed changes (or lack thereof), radiologists infer the clinical trajectory and guide downstream decisions.

graph TD A[Previous Chest X-ray] --> B{Current vs Previous
Analysis} B -->|Opacity Size Decrease| C[Improved] B -->|New Infiltrates or Growth| D[Worsened] B -->|No Observable Change| E[Stable] C --> F[Recovery or
Treatment Response] D --> G[Disease
Progression] E --> H[Continuous
Monitoring Needed] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style D fill:#ffebee,stroke:#c62828,stroke-width:2px style E fill:#fff8e1,stroke:#ff8f00,stroke-width:2px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#ffcdd2,stroke:#c62828,stroke-width:1px style H fill:#ffecb3,stroke:#ff8f00,stroke-width:1px

Case Example: Lung Consolidation

This diagram demonstrates a practical example: a patient with lung consolidation. Depending on the direction of change, clinical interpretation and management decisions vary significantly.

graph TD A[Case: Lung Consolidation] --> B{Time Point Comparison} B -->|Consolidation Reduced
Clearer Lung Fields| C[Improved] B -->|Consolidation Expanded
New Pleural Effusion| D[Worsened] B -->|Consolidation Unchanged
No New Features| E[Stable] C --> F[Successful Antibiotic
Treatment] D --> G[Disease Progression
Treatment Adjustment Needed] E --> H[Continue Current
Management Plan] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style D fill:#ffebee,stroke:#c62828,stroke-width:2px style E fill:#fff8e1,stroke:#ff8f00,stroke-width:2px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#ffcdd2,stroke:#c62828,stroke-width:1px style H fill:#ffecb3,stroke:#ff8f00,stroke-width:1px

AI Model Implications

Why Most MLLMs Fall Short — and How Libra Goes Further

Many multimodal large language models (MLLMs) struggle with temporal reasoning for three key reasons:

No awareness of time: They treat images independently and ignore their chronological order.
Hallucinated references: They fabricate prior findings without reliable comparison.
Lack of temporal alignment: They have no built-in mechanism to align or contrast image features across time.

Libra tackles these issues head-on.

Instead of prompting the model to “guess” what might have changed, Libra incorporates explicit temporal awareness into both its architecture and training process.

We feed the model structured, temporally aligned visual features extracted from the current and previous images. This enables Libra to:

Detect fine-grained, clinically meaningful changes,
Avoid hallucination,
And reason about progression or stability in a way that mirrors clinical thinking.

In the next section, Libra – Structural Logic 🧠, we’ll explore how Libra’s architecture is intentionally designed to reflect clinical reasoning. You’ll see how its modular structure enables it to reason across both time and image features with precision — building on the foundation established in Libra – Temporal Insight 🕰️.