ecoxial2007/CheX-Phi4MM-SFT

CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency

【GitHub Project】 | 【Arxiv Paper】

Introduction

CheXPO-v2 is a state-of-the-art multimodal vision-language model specialized for chest X-ray interpretation. It is built upon the Phi-4 multimodal architecture and refined using a novel alignment framework that shifts from outcome-based rewards to process supervision.

By integrating a Knowledge Graph (KG) Consistency Reward, CheXPO-v2 ensures that the model's reasoning process is clinically sound, significantly reducing hallucinations and improving factual accuracy in radiology report generation and visual question answering.

Key Innovations

• Process Supervision via KG Consistency: Unlike traditional RLHF (e.g., GRPO) that only rewards the final answer, CheXPO-v2 uses Entity-Relation Matching to penalize incoherent logic at the atomic level (Disease, Relation, Anatomy).
• Data Efficiency: Achieves new SOTA performance on benchmarks like MIMIC-CXR-VQA using only 5k preference samples.
• Verifiable Reasoning: Produces structured and clinically verifiable reasoning chains, mitigating the "overthinking" problem where models generate verbose but flawed logic.

How to Use

This model is compatible with the transformers library (version >= 4.48.2). You can use the standard AutoModelForCausalLM and AutoProcessor to run inference.

import torch
from transformers import AutoModelForCausalLM, AutoProcessor
from PIL import Image

# Load model and processor
model_id = "ecoxial2007/CheX-Phi4MM-SFT"
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
trust_remote_code=True,
torch_dtype=torch.bfloat16
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Prepare input
prompt = "<|user|>\n<|image_1|>\nWhat are the main findings in this chest X-ray? <|end|>\n<|assistant|>\n"
image = Image.open("path_to_your_cxr_image.jpg")

inputs = processor(text=prompt, images=[image], return_tensors="pt").to(model.device)

# Generate response
outputs = model.generate(**inputs, max_new_tokens=512)
response = processor.batch_decode(outputs, skip_special_tokens=True)
print(response[0])

Citation

If you find this work useful, please cite our paper:

@article
{liang2024chexpo2,
title={CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency},
author={Liang, Xiao and An, Yuxuan and Wang, Di and Hu, Jiawei and Jiao, Zhicheng and Jing, Bin and Wang, Quan},
journal={arXiv preprint arXiv:2512.17213},
year={2024}
}

Acknowledgement

This project is hosted at ecoxial2007/CheX-Phi4MM. We thank the community for their contributions to medical multimodal AI.