RadEval | Xi Zhang

RadEval: A framework for radiology text evaluation

Mon, 22 Sep 2025 00:00:00 +0000

For the latest updates and details, visit the RadEval Project Website.

Why RadEval?

Radiology report generation has seen significant advancements with the advent of large language models (LLMs). However, evaluating the quality of these generated reports remains a complex challenge. Traditional metrics like BLEU and ROUGE often fall short in capturing the clinical relevance and accuracy required in medical contexts. To address this gap, we introduce RadEval, a comprehensive framework designed to evaluate radiology texts using a diverse set of metrics that encompass both linguistic quality and clinical accuracy.

Key Features of RadEval

Diverse Metric Integration: RadEval consolidates a wide range of evaluation metrics, including classic n-gram overlap measures (BLEU, ROUGE), contextual embeddings (BERTScore), clinical concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT, TemporalEntityF1), and advanced LLM-based evaluators (GREEN). This integration allows for a multifaceted assessment of radiology reports.
Standardized Implementations: We have refined and standardized the implementations of these metrics to ensure consistency and reliability in evaluations across different studies and datasets.
Domain-Specific Enhancements: RadEval extends the GREEN metric to support multiple imaging modalities using a more lightweight model. Additionally, we have pretrained a domain-specific radiology encoder that demonstrates strong zero-shot retrieval performance, enhancing the evaluation capabilities of the framework.
Richly Annotated Expert Dataset: We provide a dataset annotated by radiology experts, containing over 450 clinically significant error labels. This dataset serves as a valuable resource for validating and benchmarking evaluation metrics against expert judgment.
Statistical Testing Tools: RadEval includes tools for statistical testing, enabling researchers to assess the significance of their results and compare different models robustly.
Baseline Model Evaluations: The framework offers baseline evaluations across multiple publicly available datasets, facilitating reproducibility and benchmarking in radiology report generation research.

Correlation with Radiologist Judgment

We conducted extensive experiments to assess how different metrics correlate with radiologist judgment. Our findings indicate that certain metrics, particularly those incorporating clinical concepts and LLM-based evaluations, show a stronger alignment with expert assessments. This correlation underscores the importance of using clinically informed metrics in evaluating radiology reports.

Getting Started with RadEval

To get started with RadEval, you can access the codebase and documentation on our GitHub repository. The repository includes installation instructions, usage examples, and guidelines for integrating RadEval into your evaluation pipeline. Additionally, the RadEval Expert Dataset is available for download, providing a rich resource for testing and validating evaluation metrics.

Conclusion

RadEval represents a significant step forward in the evaluation of radiology texts, offering a comprehensive and standardized framework that addresses the unique challenges of this domain. By integrating diverse metrics, providing a richly annotated dataset, and facilitating robust benchmarking, RadEval aims to enhance the quality and reliability of radiology report generation research.

BibTeX

@inproceedings{xu2025radeval,
 title={RadEval: A framework for radiology text evaluation},
 author={Xu, Justin and Zhang, Xi and Abderezaei, Javid and Bauml, Julie and Boodoo, Roger and Haghighi, Fatemeh and Ganjizadeh, Ali and Brattain, Eric and Van Veen, Dave and Meng, Zaiqiao and others},
 booktitle={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
 pages={546--557},
 year={2025}
}

🩺 RadEval Debuts！

Mon, 14 Jul 2025 00:00:00 +0000

🩺 Revolutionizing Radiology Text Evaluation with AI-Powered Metrics

Imagine having a comprehensive evaluation framework that doesn’t just measure surface-level text similarity, but truly understands clinical accuracy and medical semantics in radiology reports. This vision is now a reality with RadEval, a groundbreaking, open-source evaluation toolkit designed specifically for AI-generated radiology text.

📊 All-in-one metrics for evaluating AI-generated radiology text

From traditional n-gram metrics to advanced LLM-based evaluations, RadEval provides 11+ different evaluation metrics in one unified framework, enabling researchers to thoroughly assess their radiology text generation models with domain-specific medical knowledge integration.

For detailed handbook, please visit our GitHub repository:

🚀 Quick Start Demo

Try RadEval instantly with our interactive Gradio demo:

💡 Key Features

RadEval stands out with its comprehensive approach to radiology text evaluation:

🎯 Domain-Specific: Tailored for radiology with medical knowledge integration
📈 Multi-Metric: Supports lexical, semantic, clinical, and temporal evaluations
⚡ Easy to Use: Simple API with flexible configuration options
🔬 Research-Ready: Built-in statistical testing for system comparison
📦 PyPI Available: Install with a simple pip install RadEval

🏥 Advancing Radiology AI Research Community

We are committed to building a standardized and reproducible toolkit for researchers, clinicians, and developers dedicated to advancing AI evaluation in medical imaging and radiology. Together, we’re setting new standards for clinical AI assessment.

PyPI Package — Install RadEval with pip
HuggingFace Model — Access our domain-adapted evaluation model
Interactive Demo — Try RadEval online
Research Paper — Read our detailed research paper

RadEval

Fri, 04 Jul 2025 00:00:00 +0000

A framework for radiology text evaluation, focused on chest X-ray report generation.