Re3Cap-CLIP-LLaVA1.5-7B

This model is fine-tuned from LLaVA-1.5-7B using GRPO with the CLIP image-text similarity (CLIP) reward and the Re³Cap retrieval-guided refinement strategy, as described in our EMNLP 2026 Main paper.

Overview

Re³Cap (Retrieval-Guided Refinement for Image Captioning) is a retrieval-guided reasoning strategy that improves image captioning without requiring additional annotations. It is instantiated by Caption Refinement Suggester (CRS) and Caption Quality Assessor (CQA), which together identify hallucinations and omissions in generated captions, leading to more accurate and detailed descriptions.

This checkpoint applies Re³Cap on top of the CLIP reward, which uses the CLIP image-text similarity score between the image and the generated caption as the reward signal (Cho et al., 2022).

Usage

from transformers import LlavaForConditionalGeneration, AutoProcessor
from PIL import Image

model = LlavaForConditionalGeneration.from_pretrained("kkk5/Re3Cap-CLIP-LLaVA1.5-7B", torch_dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained("kkk5/Re3Cap-CLIP-LLaVA1.5-7B")

image = Image.open("your_image.jpg").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Caption this image as accurately as possible, without speculation. Describe what you see."},
]}]

text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(images=image, text=text, return_tensors="pt").to(model.device)

output_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
print(output_text)

Citation

@inproceedings{jia2026re3cap,
    title     = {Re$^3$Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning},
    author    = {Jia, Haonan and Dong, Shichao and Sun, Zenghui and Zheng, Jiawen and Miao, Ziqi and Shi, Gege and Zhao, Qiuyu and Lan, Jinsong and Zhu, Xiaoyong and Zheng, Bo},
    booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
    year      = {2026}
}