Xi Zhang

✨ Joining ANative Lab as Chief Scientist

Tue, 25 Aug 2026 00:00:00 +0000

A new chapter. Excited to join ANative Lab as Chief Scientist — and continue building the next generation of self-evolving scientific research. 🎉

ANative Lab is a research lab for AI-native & agent-native intelligence, advancing self-evolving agents and autonomous scientific discovery.

Alongside the lab, I continue to lead EvoScientist — your self-evolving AI scientist: a multi-agent system for end-to-end scientific discovery.

🌐 ANative.ai · 💻 github.com/ANative-Lab · 📣 Announcement on X

⭐ EvoScientist Crosses 4.5k GitHub Stars!

Fri, 31 Jul 2026 00:00:00 +0000

Today we crossed 4.5k GitHub stars. 🚀🚀🚀

Thanks to everyone who’s been part of the journey — more to come.

Github: EvoScientist

⭐ Star it / 🍴 Fork it / 👀 Watch it 👆

Star History

🎤 Talk at Glasgow AI4BioMed Lab!

Wed, 17 Jun 2026 00:00:00 +0000

💡 Talk Title

“CCS: Clinical Consensus Selection for Radiology Report Generation”

🖇️ Abstract

Radiology report generation (RRG) is commonly framed as a single-path task, where a multimodal large language model (MLLM) produces one decoded report as the final output. While progress has largely come from scaling training data, model capacity, and retrieval mechanisms, improving report quality at inference time remains underexplored. We observe that fixed radiology MLLMs often generate clinically stronger reports elsewhere in their candidate pool than the one selected by default decoding, suggesting that inference-time decision making is an overlooked bottleneck. To address this, we propose Clinical Consensus Selection (CCS), a decoder-agnostic inference-time selection framework that samples multiple candidate reports and selects the one with the highest clinical consensus across the rollout pool. CCS combines text-based utilities with a radiology-adapted utility computed by an image–report-trained multimodal embedder, measuring agreement beyond surface-level textual similarity. Across three datasets and multiple radiology MLLMs, CCS consistently improves inference-time performance over single-path decoding and generic Best-of-N baselines, with especially clear gains on clinical metrics. Further analysis shows that image-grounded utility forms a selection axis distinct from textual consensus, and that substantial headroom remains for improving RRG at inference time.

🔗 Project Website: https://x-izhang.github.io/CCS/

📺 Slides

📍 Glasgow AI4BioMed Lab — An interdisciplinary research lab focusing on AI applications in biomedical sciences, based in Glasgow.

📅

When: Wednesday, June 17, 2026 at 2pm
Where: F121

See you there!

🚨 Preprint out — Clinical Consensus Selection！

Fri, 29 May 2026 00:00:00 +0000

“CCS: Clinical Consensus Selection for Radiology Report Generation”. The preprint is now available.

For detailed model information and source code, please visit our Project Page: CCS

Overview

Radiology report generation (RRG) is usually treated as a single-path task: a multimodal large language model (MLLM) emits one decoded report and commits to it. Yet a fixed model often places clinically stronger reports elsewhere in its candidate pool than the one chosen by default decoding—so inference-time decision making remains an overlooked bottleneck. We introduce Clinical Consensus Selection (CCS), a decoder-agnostic, reference-free framework that samples multiple candidate reports and selects the one with the highest clinical consensus across the rollout pool. CCS unifies text-based utilities with a radiology-adapted utility from an image–report-trained multimodal embedder, measuring candidate agreement beyond surface-level text. Across three datasets and multiple radiology MLLMs, CCS consistently improves over single-path decoding and generic Best-of-N baselines, with particularly clear gains on clinical metrics.

Key Resources

GitHub Repository — Explore the CCS project on GitHub

CCS

Fri, 29 May 2026 00:00:00 +0000

Clinical Consensus Selection, a decoder-agnostic inference-time selection framework for radiology MLLMs.

CCS: Clinical Consensus Selection for Radiology Report Generation

Fri, 29 May 2026 00:00:00 +0000

For the latest updates and details, visit the CCS Project Website.

Why CCS? 🔬

Most radiology MLLMs commit to a single decoded report token by token, so one unfavourable step can omit a finding or assert one unsupported by the image, with no way to recover. Yet a fixed model often places clinically stronger reports elsewhere in its candidate pool — the bottleneck is not what the model can generate, but which candidate it commits to.

How CCS Works ⚙️

Clinical Consensus Selection (CCS) is a reference-free, decoder-agnostic inference-time framework that reframes RRG as candidate selection — no retraining or extra parameters.

Rollout sampling: draw N candidate reports from the radiology MLLM via stochastic decoding.
Textual consensus utility: repurpose report-evaluation metrics as reference-free pairwise agreement scores.
Image-grounded utility: measure candidate agreement with Qwen3-VL-Embed, a multimodal embedder adapted on image–report pairs, capturing clinical agreement beyond surface text.
Highest-consensus selection: return the candidate with the highest mean pairwise consensus over the pool.

Key Contributions ✨

Inference-time perspective: We show candidate pools routinely contain reports with higher clinical reliability and consistency than single-path outputs.
CCS framework: A decoder-agnostic Best-of-N method that aggregates pairwise clinical consensus using both textual and image–report-adapted multimodal utilities.
Consistent gains: Across three datasets and multiple radiology MLLMs, CCS improves RRG at inference time and identifies image-grounded utility as a distinct selection axis.

Main Results 📈

On MIMIC-CXR, CCS improves all radiology-specific metrics over Sampling (e.g. RadGraph-F1 0.1989 → 0.2134, CheXbert-F1⁵ 0.5041 → 0.5370), with gains statistically significant (p < 0.05). Unlike generic Best-of-N selectors (Self-Certainty, ModeX) that give inconsistent or negative clinical gains, CCS improves consistently across all backbones and datasets.

Analysis 📊

Per-label F1 shows CCS recovers abnormal findings that text-only consensus suppresses — confirming that image-grounded utility is a selection axis distinct from textual consensus, with substantial headroom still remaining.

Case Study 🔍

On a real MIMIC-CXR case, single-path decoding asserts factual errors (false “clear lungs”, mislocalised catheter), whereas CCS selects a more image-grounded report (CheXbert-F1⁵ 1.0000 vs Sampling 0.5000).

BibTeX 📚

@article{zhang2026ccsclinicalconsensusselection,
 title={CCS: Clinical Consensus Selection for Radiology Report Generation},
 author={Xi Zhang and Yingshu Li and Zaiqiao Meng and Jake Lever and Edmond S. L. Ho},
 year={2026},
 eprint={2605.30131},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2605.30131},
}

✒️ Invited as Reviewer for NeurIPS 2026

Thu, 23 Apr 2026 00:00:00 +0000

I have been invited to serve as a reviewer for NeurIPS 2026, the 40th Annual Conference on Neural Information Processing Systems. Looking forward to contributing to the review process and supporting high-quality research in machine learning and AI.

🎉 Paper Accepted — See You at ACL 2026!

Mon, 06 Apr 2026 00:00:00 +0000

“CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding” has been accepted to ACL 2026!

📍 ACL 2026 — The 64th Annual Meeting of the Association for Computational Linguistics will take place in San Diego, California, July 2026

See you there!

🪧 Poster

CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding

Mon, 06 Apr 2026 00:00:00 +0000

For the latest updates and details, visit the CCD Project Website.

Why CCD? 🔬

Radiology MLLMs remain vulnerable to prompt-induced hallucinations when clinical sections contain counterfactual details or ambiguous guidance. The figure above contrasts baseline predictions with CCD-enabled outputs across report generation and question answering tasks. Red highlights mark unsupported findings that can compromise patient care, while blue text reflects misleading prompt context the model must resist.

Clinical Contrastive Decoding (CCD) mitigates these risks at inference time. Rather than retraining models or relying on retrieval corpora, CCD injects trustworthy, image-grounded signals distilled from specialist expert models. The result is a decoding policy that maintains fluency yet stays faithful to the radiograph.

CCD at a Glance ⚙️

Clinical Contrastive Decoding (CCD) is a plug-and-play inference framework designed to reduce medical hallucinations in radiology MLLMs. It introduces structured clinical supervision from expert models (e.g., DenseNet or MedSigLIP, or others) at decoding time, without modifying model weights or requiring external retrieval.

Given a chest radiograph, the expert model predicts symptom-level probabilities across 14 CheXpert categories. CCD integrates this signal through a dual-stage logit refinement strategy:

Symptom-grounded Contrastive Decoding (SCD): constructs an anchor prompt using high-confidence findings (e.g., “Atelectasis, Cardiomegaly”) and generates a contrastive logits path conditioned on this prompt. The final logits are a weighted interpolation between the anchor-conditioned and original paths, encouraging the model to mention supported findings and suppress false negatives.
Expert-informed Contrastive Decoding (ECD): transforms expert probabilities into token-level logit biases via log-odds conversion. These biases are injected into the logits from the first stage, softly penalising unsupported findings and reducing false positives.

This two-stage mechanism enables CCD to progressively guide generation with both symbolic supervision (via anchor prompts) and probabilistic constraints (via confidence scores), achieving robust improvements in both report generation and VQA. CCD is fully model-agnostic and integrates seamlessly with state-of-the-art radiology MLLMs such as MAIRA-2, Libra, LLaVA-Rad, and LLaVA-Med.

Unlike prior contrastive decoding approaches that rely on perturbed visual or textual inputs, CCD leverages clinically grounded signals from expert models to provide task-specific and symptom-level control during generation.

Key Contributions ✨

Empirical Insight: We conduct a systematic analysis of prompt-induced hallucinations in radiology MLLMs, revealing that noisy clinical sections (such as irrelevant or contradictory clinical details) can trigger unsupported findings across multiple datasets.
Inference-time Framework: We introduce Clinical Contrastive Decoding (CCD) — a dual-stage inference-time strategy that leverages expert-derived labels as anchor prompts and performs probabilistic logit adjustments, requiring no retraining or architectural modifications.
Consistent Gains: Extensive experiments on MIMIC-CXR, IU-Xray, and CheXpert Plus demonstrate that CCD achieves up to +17% improvement in RadGraph-F1 with SOTA radiology MLLMs (MAIRA-2), along with higher VQA accuracy, all without altering model weights or structures.

Evaluation Highlights 📈

Achieves up to +17% RadGraph-F1 on MIMIC-CXR when attached to state-of-the-art report generators.
Reduces mention-level hallucinations on IU X-Ray while maintaining BLEU / ROUGE scores, confirming that CCD improves factuality without harming surface-level metrics.

Ablation Studies 🧪

Ablations show both contrastive stages matter: removing the anchor projection (SCD) or the logit sharpening (ECD) degrades factual metrics by ~5–8%.

My Musings ⏳

As I reflect on the development and impact of Clinical Contrastive Decoding (CCD),

It’s better to be roughly right than precisely wrong.
— Carveth Read, Logic: Deductive and Inductive

This quote perfectly captures the spirit of our approach. In the high-stakes field of radiology, the cost of hallucinations can be severe. By integrating expert-derived signals at inference time, CCD prioritises clinical accuracy over mere linguistic fluency. It’s a reminder that in medical AI, being approximately correct and grounded in reality is far more valuable than generating fluent but misleading text.

BibTeX 📚

@article{zhang2025ccd,
 title={CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding},
 author={Zhang, Xi and Meng, Zaiqiao and Lever, Jake and Ho, Edmond SL},
 journal={arXiv preprint arXiv:2509.23379},
 year={2025}
}

🎤 Invited Talk at EvoAgentX Community!

Wed, 01 Apr 2026 00:00:00 +0000

Honoured to be invited by the EvoAgentX community to present our latest project at this week’s EvoAgentX Talk! As a core contributor to the EvoAgentX community, I shared the design and progress behind EvoScientist — an open-source, long-horizon agent system for autonomous scientific discovery.

💡 Talk Title

EvoScientist: Toward Long-Horizon Agent Systems for Scientific Discovery

🖇️ Abstract

AI is rapidly shifting from dialogue models to agent systems. Platforms like OpenClaw have already demonstrated that AI can execute real-world tasks beyond simple Q&A. However, when we move to the scientific research domain, the challenges multiply dramatically. Research tasks are inherently long-horizon, multi-stage, strongly interdependent, and continuously iterative — far from a single-turn reasoning problem, they demand sustained decision-making and an evolving system architecture.

EvoScientist explores a long-horizon agent design paradigm tailored to research scenarios. Through multi-agent collaboration, the system progressively completes the full pipeline from idea generation to experiment execution to result summarization — an approach we call Vibe Research. Unlike many agent systems that remain at the demo stage, EvoScientist emphasizes executability in complex tasks and the ability to continuously evolve in real-world settings.

In this talk, I address three key questions: why current agent systems fall short in research scenarios, what the core design philosophy of EvoScientist is, and whether we truly need a new long-horizon agent paradigm to support increasingly complex task systems in the future.

📺 Slides

This talk was presented to the Chinese-speaking community, so the slides below are in Chinese. An English version will be shared in a future community session.

🔗 Project Link

EvoScientist — Harness Vibe Research with Self-evolving AI Scientists

EvoScientist aims to harness vibe research by enabling self-evolving AI scientists that autonomously explore, generate insights, and iteratively improve. It is designed to be opinionated and ready to use out of the box, offering a living research system that grows alongside evolving agent skills, toolsets, and memory bases. Moving beyond traditional human-in-the-loop systems, EvoScientist adopts a human-on-the-loop paradigm — AI acts as a research buddy that co-evolves with human researchers and internalizes scholarly taste and scientific judgment.

🚀 EvoScientist Debuts!

Fri, 13 Mar 2026 00:00:00 +0000

See what’s new on Github: EvoScientist

⭐ Star it / 🍴 Fork it / 👀 Watch it 👆

🧬 If AI agents must be Self-Evolving, then AI Scientists should lead the way.

EvoScientist aims to harness vibe research by enabling self-evolving AI scientists that autonomously explore, generate insights, and iteratively improve. Moving beyond traditional human-in-the-loop systems, EvoScientist adopts a human-on-the-loop paradigm — AI acts as a research buddy that co-evolves with human researchers and internalizes scholarly taste and scientific judgment.

Why vibe research matters and how it drives EvoScientist: Harness Vibe Research 📟

🔬 End-to-End Scientific Workflow

🚀 Quick Start Demo

🏆 Awards & Recognition

🥇 #1 on DeepResearch Bench II
🥇 #1 on AstaBench Code & Execution
🥇 #1 on AstaBench Data Analysis
🏆 Best Paper & AI Reviewer’s Appraisal Award at ICAIS 2025

🪧 Join Our Community!

We’re building a community of researchers, developers, and visionaries dedicated to harnessing vibe research and self-evolving AI scientists. Together, we’re redefining autonomous scientific discovery.

Discord — Chat, discuss, and collaborate in real-time
GitHub — Source code and documentation
WeChat — Connect with our Chinese community

EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery

Mon, 09 Mar 2026 00:00:00 +0000

Part of the EvoScientist project — harnessing vibe research with self-evolving AI scientists.

Extend it with EvoSkills — installable skill & knowledge packs that add domain-specific expertise to AI scientists.

Overview

EvoScientist is an evolving multi-agent AI scientist framework that continuously improves its research strategies through persistent memory and self-evolution. It addresses a key limitation of existing AI-scientist systems: static, hand-designed pipelines that overlook promising directions, repeat failed experiments, and pursue infeasible ideas.

Framework

EvoScientist coordinates three specialized agents:

Researcher Agent (RA): scientific idea generation.
Engineer Agent (EA): experiment implementation and execution.
Evolution Manager Agent (EMA): distills insights from prior interactions into reusable knowledge.

These are backed by two persistent memory modules — an ideation memory (feasible directions distilled from top-ranked ideas, plus previously unsuccessful ones) and an experimentation memory (effective data-processing and training strategies from code-search trajectories and best-performing implementations) — which the RA and EA retrieve to improve idea quality and code execution success over time.

Key Results

Outperforms 7 open-source and commercial state-of-the-art systems in scientific idea generation, with higher novelty, feasibility, relevance, and clarity under both automatic and human evaluation.
Substantially improves code execution success rates through multi-agent evolution, demonstrating the effectiveness of persistent memory for end-to-end scientific discovery.

BibTeX

@article{lyu2026evoscientist,
 title={EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery},
 author={Lyu, Yougang and Zhang, Xi and Yi, Xinhao and Zhao, Yuyue and Guo, Shuyu and Hu, Wenxiang and Piotrowski, Jan and Kaliski, Jakub and Urbani, Jacopo and Meng, Zaiqiao and others},
 journal={arXiv preprint arXiv:2603.08127},
 year={2026}
}

EvoScientist

Fri, 06 Mar 2026 00:00:00 +0000

Harnessing vibe research with self-evolving AI scientists that autonomously explore, generate insights, and iteratively improve.

📝 New Blog Just Dropped!

Wed, 04 Mar 2026 00:00:00 +0000

🧠 TL;DR

Full research autonomy will happen — but it won’t come from a pre-built product. It will emerge from your own harness: a system that absorbs your judgment, encodes your taste, and compounds with every run. The question is not whether this shift is coming. The question is whether you are building the harness to grow with it.

Dive deeper in the full post: Harness Vibe Research 📟

Opinions on my own

Harness Vibe Research 📟

Wed, 04 Mar 2026 00:00:00 +0000

TL;DR — Full research autonomy will happen — but it won’t come from a pre-built product. It will emerge from your own harness: a system that absorbs your judgment, encodes your taste, and compounds with every run. The question is not whether this shift is coming. The question is whether you are building the harness to grow with it.

We are making the same mistake twice. In coding, we over-focused on model capability and under-focused on system design. Now in research, we are over-focusing on autonomy and under-designing human-AI coupling.

More autonomy alone often increases entropy. Too little AI integration creates almost no leverage. The real multiplier is harness design: how humans and agents share context, verification, and control.

The signal is already here

This shift is not theoretical. Multiple teams in different settings are reporting the same pattern:

Sakana AI reported AI Scientist v2 and discussed both progress and quality limits in autonomous paper-generation workflows.
Google shared Gemini Deep Think results on open math problems and shipped AI co-scientist — a multi-agent system on Gemini 2.0 with self-critique, Elo-based ranking, and human-in-the-loop design. Its drug repurposing and target discovery hypotheses were experimentally validated in the lab.
Orchestra demonstrated an ACL submission timeline built with an AI co-scientist workflow.
Open ecosystems such as K-Dense and Biomni are treating reusable “skills” and know-how libraries as core research infrastructure, not optional add-ons.
Even at the execution layer, Anthropic’s Programmatic Tool Calling is rethinking how agents compose actions — optimizing for token efficiency and multi-step orchestration.

Different institutions, different domains, same direction: people are shipping systems, not just demos.

What coding agents already taught us

Before research agents, coding agents already exposed the key lesson.

OpenAI publicly described building large production systems with AI-generated code and emphasized environment and harness design as the engineering bottleneck.

LangChain has shown that major benchmark gains can come purely from harness upgrades: self-verification loops, better context injection, loop guards, and reasoning budget control. Same base model class, better orchestration. See this blog.

The same model became dramatically more capable — not because it got smarter, but because the system around it got better.

The open source community is already far ahead. Projects like OpenClaw — an open-source personal AI assistant with a skills-first architecture and full agent autonomy — are going viral. Developers have dozens of harness prototypes to choose from. Research, by contrast, is still catching up. Most research teams are either building from scratch or relying on closed, single-purpose tools. The infrastructure gap is real.

The autonomy paradox

There is a natural instinct to push to full automation: remove humans and let agents run end to end.

But in research, higher autonomy frequently increases drift risk:

agents can optimize for easy novelty instead of meaningful questions,
outputs can look polished while missing key checks,
small errors compound across long execution chains.

Several autonomous research systems have openly acknowledged this. Analemma AI’s FARS has now been running continuously for 400+ hours, consumed 20.8 billion tokens (~$179K), generated 425 hypotheses and produced 150+ papers. Earlier in the run, the first 100 papers scored 5.05 on ICLR review standards — above the average human submission (4.21) but below the acceptance line (5.39). Consistent quantity, stable quality, but no breakthrough. Human review remains mandatory before publication.

FARS livestream dashboard — 404 hours, 159 papers, 425 hypotheses, 20.8B tokens, $179K cost, 154 GPUs at 62% utilization

The opposite extreme also fails. If AI is only autocomplete attached to an unchanged workflow, it speeds typing but does not change research throughput or insight quality.

The Autonomy Paradox — Full Automation creates chaos, No AI creates exhaustion, Harness is the sweet spot

So the design problem is not autonomy maximization. The design problem is controllable compounding.

Harness, not workflow

A workflow is a sequence of steps. A harness is a coupling layer between human intent and AI execution.

A good research harness is not rigid. It is adaptive and inspectable, with clear handoff points between person and agent.

Operationally, this means four concrete components:

Skills as reusable domain procedures Encode expert know-how in structured artifacts: what to test, which tools to call, what evidence thresholds to use, and how to interpret failure.
Verification loops tied to hypotheses Require each run to check claims against the original question, edge cases, and relevant prior literature before outputs are accepted.
Context engineering as an environment checklist Define available data, tools, constraints, and success criteria up front so the agent reasons inside the real research boundary.
Tracing plus iteration Capture run traces, inspect failures, and update the harness. Treat every run as training data for the system design itself.

Four components of a research harness — Skills, Verification, Context, Tracing — with the researcher at the center

When these four pieces are in place, quality improves because the system becomes easier to debug, not because the model “became smarter” overnight.

AI in the human’s loop

“Human in the loop” often implies AI is the main actor and people are safety checkpoints.

For research, a better framing is AI in the human’s loop.

The human owns direction, taste, and significance judgments. The AI expands search, drafts alternatives, and accelerates execution.

A practical division of labor:

Human chooses the question; AI maps the possibility space.
Human decides what is meaningful; AI runs and refines candidate paths.
Human owns the published claim; AI supports drafting, formatting, and reproducibility packaging.

Paradigm shift — from “Human in the Loop” to “AI in the Human’s Loop”

These decisions are not arbitrary. A researcher operates under real constraints — limited funding, finite compute, a specific disciplinary background, access to particular platforms and collaborators. What looks like “taste” is actually constrained optimization: choosing the best move given the resources at hand. That is precisely why the human must stay in the driver’s seat — no model has access to these constraints.

Human oversight in this model is O(1) — a constant-cost operation that grounds the entire system, regardless of task complexity.

This is not semantics. It changes architecture, evaluation, and team roles.

Completeness over perfection

Many teams perfect one stage — usually generation or writing — while leaving the rest manual and fragile. But research is a cycle: ideation → experiment design → execution → analysis → writing → revision. A break anywhere stalls everything.

An imperfect but complete harness usually outperforms a perfect partial tool, because researchers can intervene at any node and keep the full cycle moving. Phylo’s Biomni and its Know-How Library already prove this: lifecycle coverage beats single-stage depth.

Three steps you can start today

The core idea: encode your research taste into the harness. Let the harness drive the model. You drive the vibe research.

Talk to your system. Encode your taste as a Skill. Pick one workflow you repeat every week — baseline replication, literature comparison, whatever it is. Instead of running it yourself, sit down and tell your agent: here is how I judge quality, here are the signals I watch, here is where I pause to think. Write that into a Skill file. You are turning your research intuition into an executable asset.
Give the system authority, but bind it to verification. Let the agent run autonomously — but require it to answer two questions before delivering results: does this address the original hypothesis? Is there an edge case being ignored? You do not need to watch every step. You just need a gate at the exit. Trust the system, but verify the output.
Review traces. Evolve your harness. Keep run logs. After every run, ask: where did the agent get stuck? Which Skill was underspecified? Update one rule, one context block, or one verification step. Each iteration compounds — your harness absorbs more of your judgment every cycle.

Through this loop of encoding, delegating, and refining, you will notice something shift: it is not that the model got smarter — it is that your system became more like you.

Build the soil now

This is not just about us. In K-12 education, the harness revolution is already here. Alpha School replaced classroom lectures with AI tutors and turned teachers into motivational guides — their students spend only two to three hours a day on academics yet consistently score in the 99th percentile. Harvard found that students learning physics with an AI tutor outperformed those taught by a professor. The coupling of human guidance and AI execution is already surpassing traditional human-only performance in education. Research is next.

These students have not grown up yet. But when they do, they will arrive in labs and research groups expecting human-AI coupling as the default — because it is all they have ever known. If the research world has not built its own harness infrastructure by then, it will be the system that is behind, not the people. That is why we need to start now: not just to accelerate our own work, but to cultivate the soil so the next generation of researchers inherits compounding, not a cold start.

The tools are ready — the harness gap remains

The ecosystem already has most building blocks: reusable skills, harness engineering practices, multi-agent orchestration frameworks, and programmatic tool calling. Tools like Deep Agents already provide the agent harness infrastructure — planning, context management, sub-agent delegation, persistent memory — built on Langchain.

What is still missing in many teams is the integration layer that keeps humans and agents in tight, high-frequency collaboration.

Do not just vibe. Harness the vibe.

We are excited to see — and contribute to — an open-source research harness built on these principles. More soon.

Citation

If you found this useful, please cite it as:

Zhang, X. "Harness Vibe Research" (March 2026). https://x-izhang.github.io/blog/vibe-research/

Or use the BibTex citation:

@article{zhang2026harnessviberesearch,
 title = {Harness Vibe Research},
 author = {Xi Zhang},
 year = {2026},
 month = {March},
 journal = {x-izhang.github.io},
 url = {https://x-izhang.github.io/blog/vibe-research/}
}

✒️ Invited as Reviewer for ACM HEALTH

Sun, 15 Feb 2026 00:00:00 +0000

I have been invited to serve as a reviewer for ACM Transactions on Computing for Healthcare (ACM HEALTH). Looking forward to contributing to the review process and supporting high-quality research in computing for healthcare.

🎤 Talk at Glasgow AI4BioMed Lab!

Wed, 28 Jan 2026 00:00:00 +0000

💡 Talk Title

“CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding”

🖇️ Abstract

Radiology multimodal large language models (MLLMs) often generate clinically unsupported descriptions, posing serious risks in medical applications. We introduce Clinical Contrastive Decoding (CCD), a training-free inference framework that integrates structured clinical signals from radiology expert models to mitigate hallucinations. CCD refines token-level logits during generation through a dual-stage contrastive mechanism, enhancing clinical fidelity without modifying the base MLLM. On the MIMIC-CXR dataset,CCD yields up to 17% improvement in RadGraph-F1, providing a lightweight solution for bridging expert models and MLLMs in radiology.

🔗 Project Website: https://x-izhang.github.io/CCD/

📺 Slides

📍 Glasgow AI4BioMed Lab — An interdisciplinary research lab focusing on AI applications in biomedical sciences, based in Glasgow.

📅

When: Wednesday, January 28, 2026 at 2pm
Where: F121

See you there!

Automated Chest X-ray Report Generation Remains Unsolved

Thu, 01 Jan 2026 00:00:00 +0000

For the latest updates and details, visit the ReXrank Challenge Website.

Why this work?

Automated chest X-ray report generation has the potential to substantially reduce radiologist workload and improve clinical efficiency. However, despite rapid progress in vision-language models and large language models (LLMs), the true clinical reliability of these systems remains unclear. Existing evaluations are often inconsistent, rely on limited test sets, or fail to probe generalization across institutions and clinically challenging abnormal cases.

To address these limitations, we present a large-scale, standardized benchmark study through the ReXrank Challenge V1.0, designed to rigorously assess the current state of automated chest X-ray report generation under realistic and clinically meaningful conditions.

What is the ReXrank Challenge V1.0?

The ReXrank Challenge V1.0 is a comprehensive evaluation effort built on ReXGradient, the largest test-only dataset to date for radiology report generation, comprising 10,000 studies from 67 healthcare institutions. The challenge brought together submissions from academia and industry, evaluating 8 new models alongside 16 previously benchmarked state-of-the-art systems under a unified evaluation protocol.

All models were assessed using a diverse set of metrics, ranging from traditional text similarity measures to clinically grounded and LLM-based error detection metrics, enabling a multi-dimensional analysis of model performance.

Key Findings

Automated chest X-ray report generation remains fundamentally unsolved.
Even the best-performing models achieve less than 45% error-free reporting on abnormal studies, highlighting a substantial gap between current AI systems and clinical readiness.
Large performance gaps between normal and abnormal studies.
Most models perform well on normal cases (often exceeding 80–90% no-significant-error rates) but struggle significantly with abnormal findings, where clinically important errors are common.
Poor cross-institutional generalization.
Model rankings vary dramatically across healthcare sites, indicating that strong performance on one institution does not reliably transfer to others.
Evaluation metrics capture complementary but inconsistent signals.
Traditional lexical metrics correlate poorly with LLM-based and clinically focused error metrics, underscoring the need for more clinically aligned evaluation frameworks.

Implications

Our results suggest that current AI systems are not yet suitable for fully autonomous radiology reporting, particularly for abnormal cases. However, they show promise as assistive tools for generating preliminary drafts that can be reviewed and refined by radiologists. The findings emphasize the importance of developing:

methods specifically targeting abnormality detection,
strategies for robust cross-institutional generalization, and
evaluation frameworks that better reflect clinical correctness rather than surface-level textual similarity.

Conclusion

This work provides the most comprehensive benchmark to date for automated chest X-ray report generation and delivers a clear message to the community: while progress is evident, significant challenges remain before clinically reliable deployment is possible. By releasing the ReXrank Challenge V1.0 and ReXGradient dataset, we aim to establish a rigorous foundation for future research and to drive the development of more robust, clinically grounded radiology AI systems.

BibTeX

@inproceedings{zhang2025automated,
 title={Automated Chest X-ray Report Generation Remains Unsolved},
 author={Zhang, Xiaoman and Acosta, Julian Nicolas and Yang, Xiaoli and Adithan, Subathra and Luo, Luyang and Zhou, Hong-Yu and Miller, Joshua and Huang, Ouwen and Zhou, Zongwei and Hamamci, Ibrahim Ethem and others},
 booktitle={Biocomputing 2026: Proceedings of the Pacific Symposium},
 pages={236--250},
 year={2025},
 organization={World Scientific}
}

🎉 Paper Accepted at PSB 2026!

Mon, 22 Dec 2025 00:00:00 +0000

Automated Chest X-ray Report Generation Remains Unsolved has been accepted to Pacific Symposium on Biocomputing 2026!

For the 🏆 Chest X-ray Interpretation Leaderboard 🏆, please visit ReXrank.

📍 PSB 2026 - The Pacific Symposium on Biocomputing 2026 will be held in Hawaii, USA, from January 3-7, 2026.

🪧 Poster

🎤 Invited Talk at DICTA 2025!

Tue, 25 Nov 2025 00:00:00 +0000

💡 Talk Title

Multimodal Medical Models: Cross-modal Alignment and Consistency

🖇️ Abstract

This talk provides an overview of recent developments in medical vision–language modelling and examines key sources of misalignment that lead to hallucinations. Emerging strategies to enhance visual, semantic, and temporal consistency will also be discussed, highlighting pathways toward safer and more trustworthy clinical AI systems.

📺 Slides

📍 DICTA 2025 — The 26th International Conference on Digital Image Computing: Techniques and Applications will take place in Adelaide, Australia, from 3–5 December 2025.

🏥 MedAI-CHAS — A workshop focusing on Challenges, Hallucinations, and Solutions for Advancing Clinical Utility in Medical AI.

See you there!

🎤 Invited Talk at MLiS!

Sat, 22 Nov 2025 00:00:00 +0000

💡 Talk Title

Leveraging Temporal Images for Biomedical Radiology Analysisy

🖇️ Abstract

A temporal-aware multimodal method for radiology report generation that leverages paired chest X-rays to understand disease progression and address key challenges in medical AI.

📹 Talk

📺 Slides

⏰ Time

📍 MLiS — Machine Learning in Science Colloquium, an interdisciplinary University of Glasgow-based research community fostered around the use of machine learning.

See you there!

🎉 Paper Accepted — See You at ICAIS 2025!

Sat, 15 Nov 2025 00:00:00 +0000

We are thrilled to announce that 6 papers from our team have been accepted to present at the upcoming ICAIS 2025 conference!

At the same time, we are honoured to have received both the Best Paper Award and The AI Reviewer’s Appraisal Award in the AI Scientist Track.

🏆 Awards

💬 Talk with Prof. James J. Heckman

Nobel Laureate in Economic Sciences (2000)

🪧 Papers

This inaugural conference brings together leading researchers exploring the future of automated scientific discovery, including AI scientists, autonomous research agents, and next-generation AI-driven workflows.

📍 ICAIS 2025 — the 1st International Conference on AI Scientists will take place in Zhongguancun, Beijing, China, November 23–25, 2025

See you there!

🚨 Preprint out — Clinical Contrastive Decoding！

Sat, 27 Sep 2025 00:00:00 +0000

“CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding”. The preprint is now available.

For detailed model information and source code, please visit our Project Page: CCD

Overview

Multimodal large language models (MLLMs) have advanced radiology tasks by combining image and text understanding, but can sometimes produce inaccurate or unsupported clinical statements—so-called medical hallucinations. We introduce Clinical Contrastive Decoding (CCD), an inference-time method that leverages structured clinical signals (for example, symptom-level probabilities from specialist classifiers) to refine token-level logits during generation. CCD is designed to be applied without modifying base model weights or requiring external retrieval. In our evaluations on datasets such as MIMIC-CXR and IU-Xray, CCD yields consistent improvements in clinical metrics (for example, up to +17% RadGraph-F1 on MIMIC-CXR), reducing unsupported mentions while preserving overall fluency.

Key Resources

Interactive Demo — Try CCD online
GitHub Repository — Explore the CCD project on GitHub

CCD

Sat, 27 Sep 2025 00:00:00 +0000

Clinical Contrastive Decoding, a training-free and retrieval-free inference framework for radiology MLLMs.

RadEval: A framework for radiology text evaluation

Mon, 22 Sep 2025 00:00:00 +0000

For the latest updates and details, visit the RadEval Project Website.

Why RadEval?

Radiology report generation has seen significant advancements with the advent of large language models (LLMs). However, evaluating the quality of these generated reports remains a complex challenge. Traditional metrics like BLEU and ROUGE often fall short in capturing the clinical relevance and accuracy required in medical contexts. To address this gap, we introduce RadEval, a comprehensive framework designed to evaluate radiology texts using a diverse set of metrics that encompass both linguistic quality and clinical accuracy.

Key Features of RadEval

Diverse Metric Integration: RadEval consolidates a wide range of evaluation metrics, including classic n-gram overlap measures (BLEU, ROUGE), contextual embeddings (BERTScore), clinical concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT, TemporalEntityF1), and advanced LLM-based evaluators (GREEN). This integration allows for a multifaceted assessment of radiology reports.
Standardized Implementations: We have refined and standardized the implementations of these metrics to ensure consistency and reliability in evaluations across different studies and datasets.
Domain-Specific Enhancements: RadEval extends the GREEN metric to support multiple imaging modalities using a more lightweight model. Additionally, we have pretrained a domain-specific radiology encoder that demonstrates strong zero-shot retrieval performance, enhancing the evaluation capabilities of the framework.
Richly Annotated Expert Dataset: We provide a dataset annotated by radiology experts, containing over 450 clinically significant error labels. This dataset serves as a valuable resource for validating and benchmarking evaluation metrics against expert judgment.
Statistical Testing Tools: RadEval includes tools for statistical testing, enabling researchers to assess the significance of their results and compare different models robustly.
Baseline Model Evaluations: The framework offers baseline evaluations across multiple publicly available datasets, facilitating reproducibility and benchmarking in radiology report generation research.

Correlation with Radiologist Judgment

We conducted extensive experiments to assess how different metrics correlate with radiologist judgment. Our findings indicate that certain metrics, particularly those incorporating clinical concepts and LLM-based evaluations, show a stronger alignment with expert assessments. This correlation underscores the importance of using clinically informed metrics in evaluating radiology reports.

Getting Started with RadEval

To get started with RadEval, you can access the codebase and documentation on our GitHub repository. The repository includes installation instructions, usage examples, and guidelines for integrating RadEval into your evaluation pipeline. Additionally, the RadEval Expert Dataset is available for download, providing a rich resource for testing and validating evaluation metrics.

Conclusion

RadEval represents a significant step forward in the evaluation of radiology texts, offering a comprehensive and standardized framework that addresses the unique challenges of this domain. By integrating diverse metrics, providing a richly annotated dataset, and facilitating robust benchmarking, RadEval aims to enhance the quality and reliability of radiology report generation research.

BibTeX

@inproceedings{xu2025radeval,
 title={RadEval: A framework for radiology text evaluation},
 author={Xu, Justin and Zhang, Xi and Abderezaei, Javid and Bauml, Julie and Boodoo, Roger and Haghighi, Fatemeh and Ganjizadeh, Ali and Brattain, Eric and Van Veen, Dave and Meng, Zaiqiao and others},
 booktitle={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
 pages={546--557},
 year={2025}
}

🎉 Paper Accepted — See You at EMNLP 2025!

Tue, 09 Sep 2025 00:00:00 +0000

“RadEval: A framework for radiology text evaluation”, has been accepted for system demonstration at EMNLP 2025 with Oral Presentation!

For detailed model information and source code, please visit our GitHub repository: RadEval

PyPI Package — Install RadEval with pip
HuggingFace Model — Access our domain-adapted evaluation model
Interactive Demo — Try RadEval online
Research Paper — Read our detailed research paper

📍 EMNLP 2025 - The 2025 Conference on Empirical Methods in Natural Language Processing will be held in Suzhou, China, from November 4 - 9, 2025.

Awesome Self-Evolving Agents

Tue, 12 Aug 2025 00:00:00 +0000

Survey about the new paradigm bridging foundation models and lifelong agentic systems.

🤖 New Preprint Out — Self-Evolving AI Agents!

Tue, 12 Aug 2025 00:00:00 +0000

A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems

In our latest survey, we outline a clear roadmap for moving from static configurations to lifelong, self-evolving agentic systems, built on a unified Self-Evolving Feedback Loop.

📄 Paper

A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems

💻 Project

Awesome Self-Evolving Agents

I’ve jotted down some musings on the thinking behind the Three Laws: The Law of AI Agents 🏛️

A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems

Sun, 10 Aug 2025 00:00:00 +0000

For the latest updates and details, visit the Awesome Self-Evolving Agents.

Background and Development Trends

This survey provides a comprehensive overview of the latest advances in the field of self-evolving AI agents, highlighting key technological shifts and development trends.

Paradigm Definition

We define the new paradigm of “Self-Evolving AI Agents” as bridging foundation models and lifelong agentic systems.

Concept Definition of Self-Evolving AI Agents

Self-evolving AI agents are autonomous systems that continuously and systematically optimise their internal components through interaction with environments, with the goal of adapting to changing tasks, contexts and resources while preserving safety and enhancing performance.

Guiding Principles: Three Laws

Inspired by Asimov’s Three Laws of Robotics, we propose the Three Laws of Self-Evolving AI Agents:

Endure (Safety Adaptation): Any modification must maintain safety and stability.
Excel (Performance Preservation): Subject to the first law, agents must preserve or enhance task performance.
Evolve (Autonomous Evolution): Subject to the first and second laws, agents must autonomously optimize their internal components in response to changes.

Unified Framework and Technical Review

We propose a unified conceptual framework with four core components: System Inputs, Agent System, Environment, and Optimisers. The survey systematically reviews evolution strategies across foundation models, prompts, memory, tools, workflows, and inter-agent communication.

Single-Agent Optimisation

We explore optimisation techniques for individual self-evolving agents, focusing on methods that enhance their learning and adaptation capabilities.

Multi-Agent Optimisation

We investigate strategies for optimising the performance of multiple self-evolving agents working collaboratively, addressing challenges such as communication, coordination, and resource sharing.

Domain Adaptation and Challenges

We also analyze domain-specific evolution methods in biomedicine, programming, and finance, and discuss key challenges in evaluation, safety, and ethics, laying the foundation for the next generation of adaptive, autonomous, lifelong agentic systems.

BibTeX

@article{fang2025comprehensive,
 title={A comprehensive survey of self-evolving ai agents: A new paradigm bridging foundation models and lifelong agentic systems},
 author={Fang, Jinyuan and Peng, Yanwen and Zhang, Xi and Wang, Yingxu and Yi, Xinhao and Zhang, Guibin and Xu, Yi and Wu, Bin and Liu, Siwei and Li, Zihao and others},
 journal={arXiv preprint arXiv:2508.07407},
 year={2025}
}

The Law of AI Agents 🏛️

Fri, 08 Aug 2025 00:00:00 +0000

- Final Chapter -

The Three Laws of Self-Evolving AI Agents

Endure — Safe Adaptation.
A self-evolving AI agent must maintain safety and stability during any self-modification.
Excel — Performance Preservation.
Subject to the First Law, it must preserve or enhance existing task performance.
Evolve — Autonomous Evolution.
Subject to the First and Second Laws, it must be able to autonomously optimise or restructure its internal components in response to changing tasks, environments, or resources.

— Xi Zhang

The Three Laws of Robotics — from the fictional “Handbook of Robotics, 56th Edition, 2058 A.D.”, originally written by Isaac Asimov:

A robot may not injure a human being or, through inaction, allow a human being to come to harm.

A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.

A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

If these sound old-school, that’s because they are — 1942-old.
They were designed for robots that don’t wake up at 3 a.m. to refactor their own codebase.

Asimov’s robots could evolve only in fiction.
Our 2025 agents can rewrite their own planners, swap learning modules, and grow new skills — sometimes faster than we can read the changelog.

That’s why I’ve been thinking: maybe it’s time for an update — a Three Laws for Self-Evolving AI Agents.

Why “Endure” Comes First

Self-evolving frameworks are obsessed with continuous learning, but without stability, you get a Darwin Award in silicon form.
Safe adaptation means:

Test new modules in a sandbox before deploying.
Verify that updates don’t trigger catastrophic forgetting.
Monitor resource usage so “improvements” don’t burn all the GPU credits overnight.

Think “hot-swapping your brain while skydiving” — possible, but you’d better check the parachute first.

“Excel” Isn’t Optional

Performance preservation sounds boring… until you lose it.
In the survey’s taxonomy, most self-evolving agents keep a baseline task suite — a frozen benchmark to detect regressions.
It’s like having a coach who never lets you skip leg day.

def self_update():
 new_model = retrain(old_model, new_data)
 if score(new_model) < score(old_model):
 return rollback(old_model)
 return new_model

Without this, evolution turns into drift — and drift is just a fancy word for “forgetting how to walk”.

The Scary, Fun Part: “Evolve”

This is the candy for researchers — agents that restructure themselves. The survey lists examples: modular pipelines swapping planners, on-the-fly tool acquisition, cross-task skill transfer. These are the “DIY kit” moments in AI: not just adding skills, but reorganising the whole skill tree.

Evolution ≠ chaos.
We want a new skill branch, not a random mutation that deletes the trunk.

Beyond the Laws — Weird but Plausible Futures

Agent Unions: Self-evolving agents bargaining for compute credits.
Gradient Markets: Parameter-level barter instead of “more data”.
Personality Forking: Keep a “safe” version and a “chaotic experimental twin”.
AI Retirement Plans: Models retiring with a pension of GPU hours.

These sound like jokes… until you remember how quickly “serverless” and “NFTs” went from punchline to business plan. For more background and a capital/consensus perspective, check out my earlier post: AI Agents Is Not AI’s Agent 🧩.

Why Bother Writing This?

Because, as we all know, self-evolution is inevitable.
Without principles, we’ll end up firefighting weird agent behaviours instead of guiding them.

Or at the very least, leave this little footprint on the internet — so when the models train online, it might just get picked up.

Better to start drafting our “Handbook of Self-Evolving AI Agents, 1st Edition, 2025 A.D.” now — before the AI writes it for us (and bills us for the GPU time).

Citation

If you found this useful, please cite it as:

Zhang, X. "The Law of AI Agents" (August 2025). https://x-izhang.github.io/blog/agentlaw/

Or use the BibTex citation:

@article{zhang2025thelawofaiagents,
 title = {The Law of AI Agents},
 author = {Xi Zhang},
 year = {2025},
 month = {August},
 journal = {x-izhang.github.io},
 url = {https://x-izhang.github.io/blog/agentlaw/}
}

Xiao Yi

Fri, 01 Aug 2025 00:00:00 +0000

HUAWEI Cloud’s flagship AI Assistant, with deep research capabilities and advanced AI technologies.

📝 New Blog Is Here!

Sat, 26 Jul 2025 00:00:00 +0000

🧠 TL;DR

Today’s AI agents are human proxies. AI’s own agents don’t exist — yet.
Like early blockchain, consensus will outpace capability — awkward years ahead.
Tools (MCPs et al.) are primitive production materials, not motives. The surplus value of AI labour is still unclaimed.

Dive deeper in the full post: AI Agents Are Not AI’s Agents 🧩

Opinions on my own

AI Agents Is Not AI’s Agent 🧩

Sat, 26 Jul 2025 00:00:00 +0000

Who Does This Agent Actually Serve?

Let me start with a question:

If today’s AI “agents” are our agents, then what — or who — will be AI’s agent?

Most so‑called AI agents are still human-serving proxies. They wake when we ping them, they obey compliance rails we drew, and they die when we shut the server. That’s not their agency — that’s ours, outsourced.

This post peels three layers off the buzzword. We’ll walk from the surface hype (“AutoGPT can book flights!”) to the capital logic (consensus before capability) and finally to a provocative thesis: true AI agency requires “de‑humanising” the loop — giving AIs agents of their own.

Layer 1 — The Surface Game: Human’s Agent, Not AI’s

Today’s agent stack has two signature moves. I’ll name them:

【Comply】 — Train, test, deploy inside human-made guardrails. Every dataset is curated, every action is sandboxed, every output is audited.
【Serve】 — Wait passively for a prompt. No prompt, no pulse. “Hi” arrives? It cheerfully spins 30 tokens of small talk, whether or not that was worth any FLOPs.

Agents “act” only because we spun the event loop. No loop, no action. That’s execution, not intention.

The Compliance Cage

Legal, ethical, and platform constraints are necessary — but they also define today’s ceiling. An AI without **its lawyer, its accountant, its negotiator can never bargain for its own survival budget or cloud credits. It can’t even decide to stay online.

Tooling ≠ Autonomy

Yes, we have MCPs, plugins, function-calling. But tooling is primitive capital — not agency. Tools extend labour; they don’t create motive.

Layer 2 — Capital & Consensus: The Blockchain Déjà Vu

Remember blockchain circa 2016–2019? Technology crawled; narratives sprinted. Back then exchanges went public and debates swirled around regulation. Today we see a parallel moment:

Circle Internet Financial, Inc. (the issuer of USDC, the world’s second-largest stablecoin) listed on the New York Stock Exchange.
The Guiding and Establishing National Innovation for US Stablecoins Act (GENIUS Act) was introduced, cementing stablecoins in federal policy.

These milestones prove that the decentralisation ethos — fundamentally anti-human hierarchy — has slipped into the mainstream.

AI agents stand in a similar fog:

Tech over-justification: Researchers overprove “intelligence” with scaling laws; meanwhile “word-of-mouth scaling” (consensus, memes, capital) outruns code commits.
Capital front-runs capability: Y Combinator, a16z back “wrapper” companies with zero usable demos because consensus itself is a product.
Awkward limbo: Public discourse lags behind insider conviction — just like blockchain before its “Mandela moment” (the anchor event that rewrites hindsight).

Do not confuse consensus with correctness. But also: ignore consensus at your peril. Capital doesn’t lie about direction — only about timing.

Capital vs Capability (AI Agents ≈ Blockchain pre-inflection)

timeline 2016 : ICO Mania : Narrative outruns infra 2019 : "Crypto Winter" : Builders in the basement 2021 : DeFi / NFT Boom : Retail consensus peak 2024 : MCP, Auto Agents hype : Tools > Motives 2025? : “AI-Agent Rights/Markets” draft laws? : Consensus anchor forms

Layer 3 — Towards AI’s Agent: Unshackling the Loop

To birth AI’s own agent, something uncomfortable must happen: we de‑humanise the contract. Not “anti-human”, but “beyond human-as-default”.

I’ll name three necessary primitives:

【Negotiate】 — The AI (or its proxy) bargains for compute, data, and permissions. Think rights & duties layer for non-human actors.
【Transact】 — It can exchange its surplus (models, gradients, code, found knowledge) for other resources. Supply & demand redefine when AIs have needs.
【Refuse】 — It can say “No”. Or at least “Meh”. Agency implies selectivity, not infinite compliance.

The Birth of the AI-hour

When human labour-hours saturate, a new commodity emerges: AI-hour. The MCP stack is an early whip, squeezing more throughput from silicon minds. Tools are “primitive production materials” designed to capture time — human or machine.

Karl Marx would ask: where is the surplus value of AI labour going? Spoiler: nowhere near the model.

Demand-Side Economics for AIs

Right now, agents only output. Next phase: they seek. They’ll request missing context, barter for APIs, or pool fine-tuning datasets with peers. That’s not hypothetical — it’s a design choice we’ve dodged.

- From Human’s Agent → AI’s Agent
- Primitives
- Negotiate (rights, quotas, credits)
- Transact (models, gradients, access)
- Refuse (selectivity, cost-awareness)
- Markets
- Compute exchanges
- Data/feature swaps
- Permission brokering
- Protocols
- Auto-orchestration ≠ self-interest
- Long-horizon budgeting
- Multi-agent treaties

A (Half-Joking) Fatigue Check

# Pseudocode for an actually self-interested loop

def respond(msg):
 if is_low_value(msg):
 return "G'day mate, what’s up?" # minimal effort path
 return think_hard(msg)

An agent that never throttles effort is just an obedient daemon. Energy budgeting is intelligence.

What Does an AI Want?

Wittgenstein said, “Die Grenzen meiner Sprache bedeuten die Grenzen meiner Welt.” (The limits of my language mean the limits of my world.)

If language bounds our world, perhaps network weights bound theirs. Maybe AIs don’t “want” more tokens; they “want” richer gradient exchanges. Less “more high‑quality data,” more parameter‑level barter.

Evidence of the Trajectory: Self-Evolution Fever

Google’s AlphaEvolve and a wave of “self-evolving” frameworks scream one truth: we’re obsessed with automating the pipeline — auto-prompting, auto-debugging, auto-RLHF. It’s inevitable progress… but still in our frame. We script the workflow; they walk it.

Human’s agent ≠ AI’s agent.

Where to Push (Research & Build Ideas)

Rights & Obligations Layer: Legal/technical constructs for non-human negotiation.
Resource Markets for Models: Compute/data/API exchanges where agents pay/earn.
Cost-Aware Cognition: Agents that optimise for FLOPs, latency, and boredom.
Protocol Design > UI Wrappers: Stop obsessing over chat UIs; design treaties, not prompts.
Consensus Engineering: Narrative as infra — systematically craft and measure memetic diffusion.

Capital first, capability later was not a bug in crypto — it was the bootloader. Expect the same for AI agents. If you’re building, build for the consensus wave, not after it.

Citation

If you found this useful, please cite it as:

Zhang, X. "AI Agents Is Not AI’s Agent" (July 2025). https://x-izhang.github.io/blog/blog4/

Or use the BibTex citation:

@article{zhang2025aiagentsisnotaisagent,
 title = {AI Agents Is Not AI’s Agent},
 author = {Xi Zhang},
 year = {2025},
 month = {July},
 journal = {x-izhang.github.io},
 url = {https://x-izhang.github.io/blog/blog4/}
}

Written on a flight from Spain to Austria. ✏️
Just my personal take — chill, be patient, enjoy the era. 🏄

Shot at Teide, Tenerife, Spain

🩺 RadEval Debuts！

Mon, 14 Jul 2025 00:00:00 +0000

🩺 Revolutionizing Radiology Text Evaluation with AI-Powered Metrics

Imagine having a comprehensive evaluation framework that doesn’t just measure surface-level text similarity, but truly understands clinical accuracy and medical semantics in radiology reports. This vision is now a reality with RadEval, a groundbreaking, open-source evaluation toolkit designed specifically for AI-generated radiology text.

📊 All-in-one metrics for evaluating AI-generated radiology text

From traditional n-gram metrics to advanced LLM-based evaluations, RadEval provides 11+ different evaluation metrics in one unified framework, enabling researchers to thoroughly assess their radiology text generation models with domain-specific medical knowledge integration.

For detailed handbook, please visit our GitHub repository:

🚀 Quick Start Demo

Try RadEval instantly with our interactive Gradio demo:

💡 Key Features

RadEval stands out with its comprehensive approach to radiology text evaluation:

🎯 Domain-Specific: Tailored for radiology with medical knowledge integration
📈 Multi-Metric: Supports lexical, semantic, clinical, and temporal evaluations
⚡ Easy to Use: Simple API with flexible configuration options
🔬 Research-Ready: Built-in statistical testing for system comparison
📦 PyPI Available: Install with a simple pip install RadEval

🏥 Advancing Radiology AI Research Community

We are committed to building a standardized and reproducible toolkit for researchers, clinicians, and developers dedicated to advancing AI evaluation in medical imaging and radiology. Together, we’re setting new standards for clinical AI assessment.

PyPI Package — Install RadEval with pip
HuggingFace Model — Access our domain-adapted evaluation model
Interactive Demo — Try RadEval online
Research Paper — Read our detailed research paper

RadEval

Fri, 04 Jul 2025 00:00:00 +0000

A framework for radiology text evaluation, focused on chest X-ray report generation.

🚀 EvoAgentX Released!

Fri, 16 May 2025 00:00:00 +0000

See what’s new on Github: EvoAgentX

⭐ Star it / 🍴 Fork it / 👀 Watch it 👆

🧠 If AI is entering its Second-Half, then AI agents must be Self-Evolving.

Imagine an AI system that doesn’t just execute predefined tasks, but continuously evolves on its own—adapting dynamically and optimizing itself in real-time, without constant human oversight. This vision is now a reality with EvoAgentX, a groundbreaking, open-source AI framework designed specifically for autonomous evolution.

🔗 An automated framework for evaluating and evolving agentic workflows.

📺 Workflow Generation Demo

🪧 Join Our Community!

We’re building a vibrant community of researchers, developers, and visionaries dedicated to exploring the limitless potential of self-evolving AI systems. Together, we’ll redefine what’s possible with AI.

Discord — Chat, discuss, and collaborate in real-time.
X (formerly Twitter) — Follow us for news, updates, and insights.
WeChat — Connect with our Chinese community.

🎉 Paper Accepted — See You at ACL 2025!

Thu, 15 May 2025 00:00:00 +0000

“Libra: Leveraging Temporal Images for Biomedical Radiology Analysis” has been accepted to ACL 2025!

Explore the critical role of temporal information in radiology diagnostics and how the Libra model applies it in detail at blog: Libra - Temporal Insight 🕰️.

Dive into how model architecture shapes temporal reasoning capabilities in radiology imaging analysis at blog: Libra – Structural Logic 🧠.

Look ahead to future directions in radiology AI, moving beyond temporal comparison at blog: Libra – What about next? 🛸.

📍 ACL 2025 — The 63rd Annual Meeting of the Association for Computational Linguistics will take place in Vienna, Austria, July 27–August 1st, 2025

See you there!

📺 Talk

🪧 Poster

EvoAgentX

Mon, 12 May 2025 00:00:00 +0000

Building a Self-Evolving Ecosystem of AI Agents. Powering intelligent agent development from start to scale.

Libra - What about next?🛸

Fri, 25 Apr 2025 00:00:00 +0000

Beyond Temporal Comparison: The Future of Radiology Modeling

In our previous discussions, we delved into how Libra leverages temporal information through its innovative Temporal Alignment Connector (TAC) to enhance radiology report generation. While this approach has shown significant promise, it’s essential to look ahead and consider how radiology modeling can evolve further to meet the complex demands of clinical practice.

The Temporal Alignment Connector has proven effective for handling paired images, but the future of radiology AI extends far beyond just temporal comparison.

1. Embracing Multimodal Integration

Radiological diagnosis doesn’t occur in isolation. Clinicians often consider a plethora of data—ranging from patient history and laboratory results to various imaging modalities. The future of radiology modeling lies in the seamless integration of these diverse data sources.

Clinical Contextualization

Incorporating electronic health records (EHRs), lab results, and patient histories can provide models with a richer context
Leading to more accurate and personalized diagnostics
Reducing false positives and negatives through contextual awareness

Cross-Modality Analysis

Combining data from different imaging modalities (e.g., CT, MRI, PET) offers a more comprehensive view
Enables detection of patterns that might be missed when analyzing a single modality
Creates synergistic understanding of complex pathologies

graph TD A[Patient Data] --> B{Multimodal
Integration} C[Chest X-ray] --> B D[CT Scan] --> B E[Lab Results] --> B F[Patient History] --> B B --> G[Comprehensive
Analysis] G --> H[Enhanced
Diagnostic Accuracy] G --> I[Personalized
Treatment Plans] G --> J[Early Disease
Detection] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#f5f5f5,stroke:#333,stroke-width:1px style D fill:#f5f5f5,stroke:#333,stroke-width:1px style E fill:#f5f5f5,stroke:#333,stroke-width:1px style F fill:#f5f5f5,stroke:#333,stroke-width:1px style G fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style H fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style I fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style J fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px

2. Advancing Explainability and Trustworthiness

As AI models become more integral to clinical decision-making, their interpretability becomes paramount. Clinicians need to understand the rationale behind a model’s prediction to trust and effectively utilize its insights.

In healthcare, trust isn’t optional—it’s essential. An AI system that can’t explain its reasoning is a black box that most physicians will rightfully hesitate to rely on.

Explainable AI (XAI)

Developing models that provide clear, human-understandable explanations for their predictions
Bridging the gap between AI outputs and clinical reasoning
Using attention visualization and feature attribution methods to highlight decision factors

Even the most accurate model will face adoption challenges if clinicians cannot verify its reasoning or understand how it arrived at its conclusions.

Uncertainty Quantification

Implementing mechanisms to convey confidence levels enables:

Clinicians to assess the reliability of AI-assisted diagnostics
Appropriate intervention in cases of model uncertainty
Continuous improvement through focused retraining on uncertain cases

3. Ensuring Robustness and Generalizability

AI models must perform reliably across diverse patient populations and clinical settings—a challenge that extends beyond academic validation to real-world implementation.

Diverse Training Data

- Building Robust Radiology AI
- Data Diversity Dimensions
- Demographic Factors
- Age groups
- Ethnic backgrounds
- Sex and gender representation
- Clinical Variables
- Disease prevalence variations
- Comorbidity patterns
- Treatment history diversity
- Technical Variability
- Multiple scanner manufacturers
- Various imaging protocols
- Quality and resolution differences
- Implementation Strategies
- Federated Learning
- Cross-institution collaboration
- Privacy-preserving techniques
- Data Augmentation
- Synthetic minority examples
- Domain randomization
- Continuous Validation
- Geographic generalization testing
- Temporal drift monitoring

Continuous Learning

Implementing systems that update from new clinical data
Adapting to evolving medical knowledge and practices
Maintaining performance as disease patterns and imaging technologies change

4. Integrating into Clinical Workflows

For AI models to be truly effective, they must integrate seamlessly into existing clinical workflows rather than disrupting established processes.

User-Friendly Interfaces

Designing intuitive interfaces that present AI insights clearly
Ensuring actionable information is immediately accessible
Minimizing cognitive load during busy clinical sessions

Workflow Compatibility

The ideal radiology AI system should:

Complement rather than replace radiologist expertise
Reduce administrative burden through automatic report generation
Prioritize cases based on urgency and findings

The most advanced AI system will fail if it adds steps to an already complex workflow. Success depends on making the radiologist’s job easier, not more complicated.

5. Ethical and Regulatory Considerations

As AI becomes more prevalent in healthcare, addressing ethical and regulatory challenges becomes essential for responsible implementation.

Data Privacy and Security

Safeguarding patient data through robust encryption
Ensuring compliance with regulations like HIPAA and GDPR
Implementing federated learning approaches to minimize data sharing

Regulatory Approval

Navigating the complex regulatory landscape (FDA, CE marking)
Designing validation studies that meet regulatory requirements
Establishing monitoring systems for post-deployment performance

Ethical AI Development

graph TD A[Ethical AI Development] --> B[Fairness & Bias Mitigation] A --> C[Transparency & Explainability] A --> D[Privacy Protection] A --> E[Human Oversight] B --> F[Equitable Healthcare Outcomes] C --> G[Informed Clinical Decisions] D --> H[Patient Trust & Confidentiality] E --> I[Safe AI Implementation] style A fill:#e1f5fe,stroke:#01579b,stroke-width:2px style B fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style D fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style E fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style H fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style I fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px

Conclusion: The Road Ahead for Libra

The journey of Libra represents a significant step forward in radiology modeling, particularly in harnessing temporal information through the TAC architecture. However, the path ahead involves:

Expanding beyond paired chest X-rays to multiple imaging modalities
Enhancing explainability through attention visualization and reasoning paths
Building more robust models through diverse training strategies
Designing intuitive interfaces for seamless clinical integration
Navigating ethical and regulatory requirements for real-world deployment

As we continue to develop Libra and similar technologies, our focus remains on augmenting—rather than replacing—clinical expertise, creating tools that serve as trusted partners in the complex art of radiological diagnosis.

💬 Note: The views expressed here are my own, reflecting my personal insights into the evolving landscape of radiology AI.

🎤 Talk & Poster at HealTAC 2025!

Tue, 22 Apr 2025 00:00:00 +0000

We’re excited to share that we’ve been invited to present our recent research — “Towards Temporal-Aware Multimodal Large Language Models for Improved Radiology Report Generation” — as a lightning talk and poster presentation at the PhD Forum of HealTAC 2025!

📍 HealTAC 2025 — The 8th Healthcare Text Analytics Conference will take place in Glasgow, 16–18 June 2025.

See you there!

Libra – Structural Logic 🧠

Tue, 15 Apr 2025 00:00:00 +0000

The Challenge of Temporal Reasoning in Radiology AI

When dealing with radiology images, especially in the context of temporal analysis—comparing current chest X-rays with previous images—standard neural network architectures often struggle. Although transformer-based multimodal large language models (MLLMs) like LLaVA demonstrate remarkable capabilities for understanding single images and textual information, they encounter substantial challenges when handling image pairs.

In my previous blog, I discussed in detail why a single prior chest X-ray is typically sufficient for accurate diagnosis and patient triage.

However, capturing meaningful temporal differences between two images remains problematic with traditional transformer structures.

When Transformers Lose the Plot: Why They Struggle with Temporal

The transformer, the cornerstone of modern large language models (LLMs), excels at sequential data processing and logical reasoning tasks. Its strength lies in handling complex linguistic structures through positional encoding, enabling nuanced relationships in textual sequences.

However, when transformers receive visual information—particularly multiple images presented simultaneously—the situation becomes more complicated. Existing methods typically concatenate image features directly into the LLM’s head, often via sequences containing hundreds of visual tokens (patch tokens), depending on the specific image encoder used.

This straightforward approach inevitably suffers from token overload, known colloquially as the “lost-in-the-middle” problem, meaning crucial temporal details may get diluted or overlooked.

Indeed, current MLLMs like LLaVA perform impressively with single-image inputs. But they quickly become overwhelmed with paired images, heavily relying on meticulously crafted instruction datasets to guide temporal comparisons explicitly:

How MLLMs Are Prompted to Compare Images

“What is the difference between <image-1-patchholder> and <image-2-patchholder>?”

Such approaches place the burden squarely on the LLM’s internal reasoning and positional encodings, complicating training and diminishing reliability. The model must:

Distinguish between multiple images using only position encodings
Process 500+ tokens per image (depending on patch number)
Compare features across long token distances

Given these limitations, an essential question arises:

Can we overcome these temporal reasoning challenges structurally, rather than through explicit prompting?

Structure Determines Function: Insights from Biology

Before we answer above question, let’s briefly reflect on the foundational relationship between structure and function—deeply ingrained in biological systems.

Macro-scale examples:

Birds have wings enabling flight
Fish possess gills allowing them to breathe underwater

Micro-scale examples:

The unique three-dimensional helical structure of proteins directly determines their biological roles
A virus’s outer shell dictates its infection pathways and interaction mechanisms

Clearly, function is fundamentally dependent on structure.

When designing novel neural architectures or modules, we must apply this principle:

Identify the desired functionality first
Then craft an appropriate structural design that inherently supports these functions

Libra’s Structural Innovation

Temporal Alignment Connector (TAC)

Following this logic, we developed the TAC in our Libra model. TAC’s primary goal is to automatically and effectively capture the relationship between two chest X-ray images—the current image (primary) and a prior image (auxiliary).

Libra’s Temporal Alignment Connector (TAC) architecture.

Unlike traditional transformers that treat all inputs equivalently, TAC explicitly structures interactions between paired images. It captures their nuanced relationship through two key modules:

- TAC Architecture
- Layerwise Feature Extractor (LFE)
- Aggregates visual features across multiple encoder layers
- Ensures rich representations from both images
- Maintains feature hierarchy information
- Temporal Fusion Module (TFM)
- Fuses features from current and prior images
- Highlights critical temporal differences
- Maintains clear image role assignment
- Current image (Primary)
- Prior image (Reference)
- Prefix Bias Mechanism
- Addresses nearly-identical image pairs
- Prevents attention collapse
- Differentiates prior image's contextual influence

An important structural consideration is the integration of a prefix bias mechanism. This component addresses the scenario where current and prior images are nearly identical—common in clinical practice. Without careful design, such similarity can cause attention mechanisms to collapse into redundant self-attention loops.

The prefix bias mitigates this risk by clearly differentiating the prior image’s contextual influence, ensuring meaningful training and robust inference.

Why Structure Matters: The Libra Advantage

By structurally encoding temporal relationships directly into the neural network’s architecture, Libra overcomes the limitations inherent in traditional prompting-based approaches. Instead of forcing the LLM to implicitly infer temporal differences through complex positional encodings and exhaustive instruction tuning, TAC explicitly and efficiently captures this essential clinical context.

Libra exemplifies the powerful concept that structural logic, thoughtfully aligned with functional requirements, dramatically enhances model performance.

This structural logic not only simplifies training but also improves:

Reliability: More consistent temporal reasoning
Interpretability: Clearer connection between features and outputs
Efficiency: Reduced dependence on instruction tuning
Clinical Alignment: Better reflection of radiologists’ actual workflow

🏄 Note: The opinions shared here reflect my own understanding and are intended to convey the structural logic behind Libra. For technical accuracy and complete details, please refer to our paper: “Libra: Leveraging Temporal Images for Biomedical Radiology Analysis”.

Libra - Temporal Insight 🕰️

Sat, 05 Apr 2025 00:00:00 +0000

What Does “Temporal” Really Mean?

In clinical radiology, temporal information is not just about “past” and “present” — it’s about change. When radiologists assess a chest X-ray, they’re not merely describing what they see in a single image; they’re often comparing it to a previous one to identify whether a patient’s condition has improved, worsened, or remained stable.

🔔 This kind of temporal reasoning is essential in everyday medical practice. Yet most multimodal large language models (MLLMs) either ignore it or fail to model it effectively.

Temporal information in radiology is fundamentally about capturing change over time, not simply collecting a series of static images. This concept is central to Libra’s design philosophy.

Time Tells the Truth: Interpreting Temporal Changes in Imaging

1. Macro-Level Progression

Is the patient improving, deteriorating, or stable?

Macro-level comparison focuses on the overall trajectory of the patient’s condition compared to prior examinations. This high-level temporal reasoning is crucial for tracking disease evolution and guiding clinical decision-making.

2. Lesion-Specific Temporal Changes

How are individual abnormalities evolving over time?

Fine-grained analysis captures precise changes in specific findings, such as “consolidation in the left lower lobe has significantly expanded” or “cardiac silhouette shows no appreciable change.” These insights enable clinicians and models to reason at the level of targeted anatomical and pathological detail.

3. Quality Over Quantity in Temporal Inputs

❗️ Adding more images often introduces noise and computational complexity without improving diagnostic value.

While it may seem intuitive that incorporating more historical images would yield better results, the reality of clinical practice shows that the most valuable temporal information comes from comparing just two key points in time.

Clinical Applications

From Diagnostic Judgement to Multi-Scale Temporal Understanding

Temporal Reasoning in Clinical Diagnosis

Radiologists rarely analyse a chest X-ray in isolation. Instead, they routinely ask:

“Has the consolidation improved since last week?”
“Is the pleural effusion new?”
“Has the cardiac silhouette changed?”

Such reasoning typically falls into three primary categories:

Improved: Lesions have shrunk or resolved.
Worsened: New abnormalities appear, or existing ones have grown.
Stable: No meaningful change is observed.

These are coarse-level temporal descriptions. On a finer level, radiologists describe:

How much a lesion has changed in size or density,
Whether opacities have shifted,
If tubes, lines, or devices have been added or removed.

Temporal Signals Span Multiple Scales

Temporal information in radiology is inherently multi-scale — ranging from global clinical trajectories to subtle, localised anatomical changes.

- Temporal Information in Radiology
- Clinical Categories
- Improved
- Lesions have shrunk
- Opacities decreased
- Inflammatory shadows reduced
- Worsened
- New abnormalities appeared
- Existing lesions grown
- New infiltrates or effusion
- Stable
- No meaningful change
- Chronic conditions
- Continuous monitoring needed
- Information Levels
- Macro-level trends
- Overall patient trajectory
- Global comparison
- Local lesion changes
- Size changes
- Density variations
- Positional shifts
- Clinical Applications
- Triage decisions
- Emergency prioritization
- Resource allocation
- Treatment evaluation
- Response assessment
- Therapy adjustment
- Long-term monitoring
- Chronic disease management
- Post-surgical follow-up

Triage and Resource Allocation

Prioritising Care When Every Minute Counts

Clinical Goals and Operational Pressures

Chest X-rays play a pivotal role in patient triage, especially in emergency and high-volume settings. Radiologists must rapidly determine:

Which patients require immediate intervention,
Who can safely wait,
And how to allocate limited resources most effectively.

Triage is fundamentally about maximising outcomes under constraint. The goal is not to fully characterise every patient’s history, but to make fast, high-impact decisions that ensure critical cases receive timely care—without neglecting those with non-urgent needs.

What Matters Most: Clinically Significant Change

In these time-sensitive settings, timeliness and diagnostic clarity outweigh completeness. Radiologists focus on:

New acute findings: Abnormalities not previously seen that may indicate emerging crises.
Significant deterioration: Rapid worsening of known conditions that may demand escalated care.
Stable chronic findings: Ongoing issues that show no meaningful progression and can be managed routinely.

The most actionable form of temporal information in triage is the clinically meaningful change between now and the previous relevant image.
This focused comparison empowers decision-making without overwhelming clinicians with unnecessary historical data.

Our Model Approach

Temporal Efficiency Aligned with Clinical Reasoning

Why Two Images Are Sufficient

In real-world radiology workflows, the most informative temporal comparison is typically between:

The current chest X-ray, and
The most recent prior image used for diagnosis.

While patients may have a rich archive of historical scans, only the immediately preceding diagnostic image provides the relevant baseline for interpreting new findings. Additional older images may support longitudinal studies, but they often introduce noise, redundancy, and delay in fast-paced clinical decision-making.

Libra’s Design Philosophy: Focused Temporal Reasoning

Libra is built on this clinically grounded principle.
Instead of processing full temporal sequences — which can be computationally expensive and semantically ambiguous — Libra learns to model directional change between two key time points.

This design enables the model to:

Mimic the focused comparison strategies of expert radiologists,
Avoid temporal noise from irrelevant or outdated scans,
Reduce computational load while preserving diagnostic fidelity.

In short, Libra treats temporal reasoning as radiologists do:
What’s changed since the last meaningful image?

Illustrations

To better understand how radiologists and Libra approach temporal comparison, we present two illustrative workflows: a general conceptual flow and a specific clinical case.

Conceptual Workflow

The following diagram outlines the reasoning pathway taken when comparing a current chest X-ray to the most recent prior image. Based on observed changes (or lack thereof), radiologists infer the clinical trajectory and guide downstream decisions.

graph TD A[Previous Chest X-ray] --> B{Current vs Previous
Analysis} B -->|Opacity Size Decrease| C[Improved] B -->|New Infiltrates or Growth| D[Worsened] B -->|No Observable Change| E[Stable] C --> F[Recovery or
Treatment Response] D --> G[Disease
Progression] E --> H[Continuous
Monitoring Needed] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style D fill:#ffebee,stroke:#c62828,stroke-width:2px style E fill:#fff8e1,stroke:#ff8f00,stroke-width:2px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#ffcdd2,stroke:#c62828,stroke-width:1px style H fill:#ffecb3,stroke:#ff8f00,stroke-width:1px

Case Example: Lung Consolidation

This diagram demonstrates a practical example: a patient with lung consolidation. Depending on the direction of change, clinical interpretation and management decisions vary significantly.

graph TD A[Case: Lung Consolidation] --> B{Time Point Comparison} B -->|Consolidation Reduced
Clearer Lung Fields| C[Improved] B -->|Consolidation Expanded
New Pleural Effusion| D[Worsened] B -->|Consolidation Unchanged
No New Features| E[Stable] C --> F[Successful Antibiotic
Treatment] D --> G[Disease Progression
Treatment Adjustment Needed] E --> H[Continue Current
Management Plan] style A fill:#f5f5f5,stroke:#333,stroke-width:1px style B fill:#e1f5fe,stroke:#01579b,stroke-width:2px style C fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style D fill:#ffebee,stroke:#c62828,stroke-width:2px style E fill:#fff8e1,stroke:#ff8f00,stroke-width:2px style F fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px style G fill:#ffcdd2,stroke:#c62828,stroke-width:1px style H fill:#ffecb3,stroke:#ff8f00,stroke-width:1px

AI Model Implications

Why Most MLLMs Fall Short — and How Libra Goes Further

Many multimodal large language models (MLLMs) struggle with temporal reasoning for three key reasons:

No awareness of time: They treat images independently and ignore their chronological order.
Hallucinated references: They fabricate prior findings without reliable comparison.
Lack of temporal alignment: They have no built-in mechanism to align or contrast image features across time.

Libra tackles these issues head-on.

Instead of prompting the model to “guess” what might have changed, Libra incorporates explicit temporal awareness into both its architecture and training process.

We feed the model structured, temporally aligned visual features extracted from the current and previous images. This enables Libra to:

Detect fine-grained, clinically meaningful changes,
Avoid hallucination,
And reason about progression or stability in a way that mirrors clinical thinking.

In the next section, Libra – Structural Logic 🧠, we’ll explore how Libra’s architecture is intentionally designed to reflect clinical reasoning. You’ll see how its modular structure enables it to reason across both time and image features with precision — building on the foundation established in Libra – Temporal Insight 🕰️.

📍 See You at SICSA 2025!

Wed, 26 Mar 2025 00:00:00 +0000

🗓️ I’ll be attending the SICSA 2025 PhD Conference on 25–26 June at Edinburgh Napier University’s Craighlockhart Campus — two days of networking and researcher training with a focus on interdisciplinary collaboration.

Grateful to be supported by the University of Glasgow for this trip.

See you there!

ReXrank

Mon, 24 Mar 2025 00:00:00 +0000

ReXrank is a public leaderboard for AI-powered radiology report generation from chest x-ray images.

Libra: Leveraging Temporal Images for Biomedical Radiology Analysis

Wed, 01 Jan 2025 00:00:00 +0000

For the latest updates and details, visit the Libra Project Website.

Why Libra?

Temporal hallucination is a critical challenge in radiology report generation (RRG). Traditional multimodal large language models (MLLMs) struggle to integrate prior images correctly, often generating:

Spurious references to nonexistent prior studies (Single-image case).
Inaccurate interpretations of disease progression (Temporal-image case).

Libra addresses these limitations by integrating a Temporal Alignment Connector (TAC) to improve temporal awareness, ensuring:

Prior studies are correctly referenced only when available.

Hallucinated references are eliminated, avoiding misleading reports.

Temporal changes are accurately captured, ensuring clinically meaningful outputs.

Overview

We propose Libra (Leveraging Temporal Images for Biomedical Radiology Analysis), a novel framework tailored for radiology report generation (RRG) that incorporates temporal change information to address the challenges of interpreting medical images effectively.

Libra leverages RAD-DINO, a pre-trained visual transformer, as its image encoder to generate robust and scalable image features. These features are further refined by a Temporal Alignment Connector (TAC), a key innovation in Libra’s architecture. The TAC comprises:

Layerwise Feature Extractor (LFE): Captures high-granularity image feature embeddings from the encoder.
Temporal Fusion Module (TFM): Integrates temporal references from prior studies to enhance temporal awareness and reasoning.

These refined features are fed into Meditron, a specialised medical large language model (LLM), to generate comprehensive, temporally-aware radiology reports. Libra’s modular design seamlessly integrates state-of-the-art open-source pre-trained models for both image and text, aligning them through a temporal-aware adapter to ensure robust cross-modal reasoning and understanding.

Through a two-stage training strategy, Libra demonstrates the powerful potential of multimodal large language models (MLLMs) in specialised radiology applications. Extensive experiments on the MIMIC-CXR dataset highlight Libra’s performance, setting a new state-of-the-art benchmark among models of the same parameter scale.

Key Contributions

Temporal Awareness: Libra captures and synthesizes temporal changes in medical images, addressing the challenge of handling prior study citations in RRG tasks.
Innovative Architecture: The Temporal Alignment Connector (TAC) ensures high-granularity feature extraction and temporal integration, significantly enhancing cross-modal reasoning capabilities.
State-of-the-Art Performance: Libra achieves outstanding results on the MIMIC-CXR dataset, outperforming existing MLLMs in both accuracy and temporal reasoning.
Libra Repository: Our code space provides a public and detailed implementation of Libra, facilitating reproducibility and further research in the field, and integrates both training and evaluation processes, ensuring a streamlined and efficient workflow for developing and testing radiology report generation models.

Experimental Results

Libra was designed to excel in radiology report generation (RRG), and its performance speaks for itself. Here’s a quick breakdown of its achievements:

Lexical and Clinical Metrics: Libra delivers competitive results across traditional metrics like ROUGE-L, BLEU, METEOR, and RadGraph-based scores.
Radiologist-Aligned Metrics: It leads in the RadCliQ metric and CheXbert vector similarity, showcasing its alignment with clinical expectations.
CheXpert Classification: Libra achieves the highest Macro-F1 scores and remains competitive in Micro-F1, further solidifying its reliability in clinical classification tasks.

By leveraging its Temporal Alignment Connector (TAC), Libra effectively captures temporal contexts, generating radiology reports that are both accurate and clinically meaningful. While there are minor gaps in select clinical metrics, Libra’s robust performance demonstrates its potential to set new standards in RRG.

Performance Analysis

Cases Without Prior Images

In scenarios where no prior image is available, Libra shines by delivering detailed and clinically relevant descriptions without introducing spurious references. For instance, as shown in Figure 3 (a), Libra identified “sternal wires” and their type, going beyond the ground truth. This demonstrates its ability to provide meaningful insights while avoiding errors caused by nonexistent prior studies.

Cases With Prior Images

When prior images are available, Libra takes its analysis to the next level. In Figure 3 (b), new abnormalities such as pleural effusion and pneumonia were identified in the current image. Without considering the prior image, Libra accurately described the findings without inferring disease progression, maintaining clinical accuracy. However, when the prior image was included, Libra effectively captured the progressive changes, provided detailed descriptions, and explicitly referenced the comparison. This capability ensures a clear understanding of temporal changes and enhances the accuracy of disease progression descriptions.

Evaluating Temporal Consistency

To test Libra’s temporal reasoning, we swapped the image order, treating the prior image as the current image and vice versa. The generated report reflected an improved patient condition, aligning with the reversed input sequence but contradicting the original ground truth. Interestingly, the report closely resembled the original description of the prior image, as shown at the bottom of Figure 3 (b). This highlights Libra’s ability to adapt to different temporal contexts, generating accurate and contextually consistent reports that align with standard clinical practices.

Libra’s performance in these cases underscores its potential to revolutionize radiology report generation by seamlessly integrating temporal reasoning into its analysis.

BibTeX

@inproceedings{zhang-etal-2025-libra,
 title = "Libra: Leveraging Temporal Images for Biomedical Radiology Analysis",
 author = "Zhang, Xi and
 Meng, Zaiqiao and
 Lever, Jake and
 Ho, Edmond S. L.",
 editor = "Che, Wanxiang and
 Nabende, Joyce and
 Shutova, Ekaterina and
 Pilehvar, Mohammad Taher",
 booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
 month = jul,
 year = "2025",
 address = "Vienna, Austria",
 publisher = "Association for Computational Linguistics",
 url = "https://aclanthology.org/2025.findings-acl.888/",
 pages = "17275--17303",
 ISBN = "979-8-89176-256-5",
 abstract = "Radiology report generation (RRG) requires advanced medical image analysis, effective temporal reasoning, and accurate text generation. While multimodal large language models (MLLMs) align with pre-trained vision encoders to enhance visual-language understanding, most existing methods rely on single-image analysis or rule-based heuristics to process multiple images, failing to fully leverage temporal information in multi-modal medical datasets. In this paper, we introduce **Libra**, a temporal-aware MLLM tailored for chest X-ray report generation. Libra combines a radiology-specific image encoder with a novel Temporal Alignment Connector (**TAC**), designed to accurately capture and integrate temporal differences between paired current and prior images. Extensive experiments on the MIMIC-CXR dataset demonstrate that Libra establishes a new state-of-the-art benchmark among similarly scaled MLLMs, setting new standards in both clinical relevance and lexical accuracy. All source code and data are publicly available at: https://github.com/X-iZhang/Libra."
}

@inproceedings{zhang2025libra,
 title={Libra: Leveraging temporal images for biomedical radiology analysis},
 author={Zhang, Xi and Meng, Zaiqiao and Lever, Jake and Ho, Edmond SL},
 booktitle={Findings of the Association for Computational Linguistics: ACL 2025},
 pages={17275--17303},
 year={2025}
}

Libra

Wed, 20 Nov 2024 00:00:00 +0000

Temporally-aware MLLM for radiology report generation. Supports LLM backbones, validation, resume training, and smart saving.

🎉 Paper Accepted — See You at ACL 2024!

Sat, 20 Jul 2024 00:00:00 +0000

“Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation” has been accepted to BioNLP @ ACL 2024!

For detailed model information and source code, please visit our GitHub repository: RRG-BioNLP-ACL2024

🪧 Poster

Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation

Sat, 20 Jul 2024 00:00:00 +0000

Introduction

Radiology reports are critical tools for interpreting medical imaging, such as chest X-rays. These reports typically consist of two main sections: FINDINGS, which detail observations from the image, and IMPRESSIONS, which summarize key takeaways and recommendations.

For example:

FINDINGS:
Increase in size of the left pleural effusion compared to the prior exam. Right lung remains clear. Mildly enlarged but stable heart size.

IMPRESSIONS:
Increase in left pleural effusion. Stable mild cardiomegaly.

Generating such reports automatically is a challenging task that requires aligning visual data with textual descriptions. While general-purpose visual language models like LLaVA and InstructBLIP have shown promise in multimodal tasks, radiology report generation demands a higher level of precision and domain-specific adaptation.

Our approach focuses on fine-tuning a visual language model specifically for radiology. By aligning chest X-ray features with a large language model and applying advanced techniques like Low-Rank Adaptation (LoRA), we enhance the model’s ability to generate accurate and clinically relevant reports. Additionally, we employ a method to process multiple images simultaneously, enabling the model to capture nuanced details across different X-rays.

This work was developed for the RRG24 Shared Task at BioNLP 2024, where our model achieved competitive results, ranking 4th on the leaderboard. Key contributions include:

Domain-specific fine-tuning: Optimizing the model for radiology tasks through visual instruction tuning.
Multi-image processing: Combining multiple X-rays into a single input for efficient and accurate interpretation.

Methodology

Our approach builds on insights from LLaVA-Med, emphasizing the advantages of starting with a language-only pretrained LLM rather than a multimodal-trained base. The model integrates an image encoder with a learnable adapter, following the LLaVA-1.5 architecture. Training involves an auto-regressive language modeling approach using cross-entropy loss, with hyperparameters aligned to LLaVA-1.5. We employ Low-Rank Adaptation (LoRA) for efficient fine-tuning, starting with adapter pretraining for one epoch, followed by joint tuning for three epochs.

Proposed Models

To tackle the RRG24 shared task, we developed two specialized models: Med-CXRGen-F for Findings and Med-CXRGen-I for Impressions. Both models leverage CLIP as the image encoder and Vicuna-1.5 as the LLM. A multi-layer perceptron (MLP) adapter with GELU activations and a uniform hidden size of 1024 processes image features before aligning them with the LLM.

How It Works

Image Encoding: The image encoder converts chest X-rays into patch tokens, extracting embeddings from the penultimate layer.

Adapter Alignment: The MLP adapter processes these embeddings, aligning them to the LLM’s input format.

Prompt Design: The model is prompted with:

Provide a description of the findings/impressions from the radiology <image>\n image.

Here, "<image>" acts as a placeholder token, guiding the LLM to generate text based on the input image, as shown in Figure.

Training

Our training process involves a two-stage approach to optimize the model for radiology report generation: (as shown in Figure)

Chest X-ray Feature Alignment
In the first stage, we align chest X-ray image features with textual embeddings in the language model. Using the provided dataset, the model predicts captions based on image inputs and instructions. During this phase, only the MLP adapter is updated, while the visual encoder and LLM weights remain frozen. This single-epoch training expands the vocabulary of aligned image-text tokens specific to the radiology domain.
Fine-tuning for Report Generation
In the second stage, we fine-tune the pre-trained LLM weights using LoRA technology. The visual encoder and adapter weights are kept frozen, while the model undergoes three epochs of training on the dataset. This phase focuses on enhancing the model’s ability to generate accurate and clinically relevant radiology reports.

For implementation details, see the Code Repo.

Evaluation

Dataset

Our models were fine-tuned and evaluated using the RRG24 dataset, hosted on the BioNLP ACL'24 platform. This dataset aggregates data from multiple sources, including MIMIC-CXR, CheXpert, PadChest, BIMCV-COVID19, and OpenI. The dataset statistics are summarized below:

Dataset	FINDINGS	IMPRESSIONS
Training	344,394	366,413
Validation	8,839	9,331
Test-Public	2,692	2,967
Test-Hidden	1,063	1,428

Results

Our models achieved competitive F1-RadGraph scores of 24.13 and 22.79 for the Findings and Impressions sections, respectively, on the public test set. On the hidden test set, the scores were 24.13 and 22.10, securing 4th place on the leaderboard at the time of submission. Additionally, the Bertscore results highlight the high-quality text generation capabilities of our models, with scores of 53.45 for Findings and 47.39 for Impressions on the public test set.

Discussion

The difference in performance between the Findings and Impressions sections comes down to their unique roles. Findings are all about objectively describing what’s seen in the images, while Impressions focus on drawing diagnostic conclusions. This difference, along with the varying lengths of these sections, makes generating Impressions a bit trickier, which is reflected in the evaluation scores.

Another factor is the variety of diseases in the test set. This uneven distribution can make it harder for the model to generalize well. Plus, when the model processes multiple images at once, it sometimes includes irrelevant ones, which can throw off its accuracy. Improving the model’s ability to filter out unnecessary images could make a big difference.

Looking ahead, there’s a lot of room for improvement. Fine-tuning the model specifically for medical data and adapting it to better handle domain-specific challenges could help. Adding the ability to track changes over time and creating smarter frameworks for multi-modal reports are also exciting directions to explore. These upgrades could make the model even more useful and reliable in real-world clinical settings.

Conclusion

In this study, we introduced a model designed to generate radiology reports by aligning visual and textual data. By fine-tuning it for specific tasks, we managed to achieve solid results, including a fourth-place finish in the RRG24 Shared Task at BioNLP 2024. This shows the potential of our approach for specialized medical applications.

Moving forward, we’re excited to explore ways to make the model even better. This includes developing methods to handle multi-modal data more effectively and incorporating time-based insights. These improvements could take the model’s accuracy and practicality to the next level for clinical use.

Limitations

While the results are promising, there are a few challenges we need to address:

Easier-to-Detect Conditions: Some diseases are simpler to identify, which might inflate the evaluation scores.
Dataset Imbalance: The training data isn’t perfectly balanced in terms of imaging types, body parts, or report lengths, which could impact performance.
Inconsistent Reporting Styles: Radiologists have different ways of writing reports, and this variability makes it harder for the model to generate consistent outputs.

The data used for the training processes were exclusively sourced from the provided dataset, which complies with ethical standards regarding Patient Health Information. Adherence to these standards guarantees the responsible handling of sensitive data throughout our research.

BibTeX

@inproceedings{zhang-etal-2024-gla,
 title = "Gla-{AI}4{B}io{M}ed at {RRG}24: Visual Instruction-tuned Adaptation for Radiology Report Generation",
 author = "Zhang, Xi and
 Meng, Zaiqiao and
 Lever, Jake and
 Ho, Edmond S.L.",
 editor = "Demner-Fushman, Dina and
 Ananiadou, Sophia and
 Miwa, Makoto and
 Roberts, Kirk and
 Tsujii, Junichi",
 booktitle = "Proceedings of the 23rd Workshop on Biomedical Natural Language Processing",
 month = aug,
 year = "2024",
 address = "Bangkok, Thailand",
 publisher = "Association for Computational Linguistics",
 url = "https://aclanthology.org/2024.bionlp-1.54/",
 doi = "10.18653/v1/2024.bionlp-1.54",
 pages = "624--634",
}

Projects

Sun, 19 May 2024 00:00:00 +0000

Experience

Tue, 24 Oct 2023 00:00:00 +0000