--- language: - zh - en license: apache-2.0 base_model: Qwen/Qwen3-VL-8B-Instruct library_name: transformers pipeline_tag: image-text-to-text tags: - qwen3-vl - vision-language - text-to-image-evaluation - image-text-alignment - consistency-evaluation - chain-of-thought - rubric-grading - ms-swift --- # WeGen-Consistency-CoT > A sharp-eyed multimodal judge for text-to-image prompt consistency. > Give it a generated image and its source prompt; it scores like a reviewer, deducts like a quality inspector, and explains like an annotation expert. **WeGen-Consistency-CoT** is a text-to-image consistency evaluation model fully fine-tuned from **Qwen3-VL-8B-Instruct**. It is designed for long, detailed prompts and focuses on whether a generated image faithfully follows the prompt. The model returns a structured judgment with a **1-10 score, total deduction, overall assessment, and fine-grained deduction reasons**. ![WeGenBench teaser](teaser.png) This model is intended to be used as the consistency judge in the **WeGenBench** evaluation pipeline. For the benchmark, data format, and full evaluation workflow, please refer to the official repository: [WeChatCV/WeGenBench](https://github.com/WeChatCV/WeGenBench). Instead of producing only a black-box score, the model decomposes prompt-image mismatches into interpretable error categories, such as missing entities, wrong counts, incorrect attributes, reversed actions, mismatched materials, distorted text, or composition errors. It is useful for automatic text-to-image evaluation, badcase mining, data filtering, regression testing, and human review assistance. ## Highlights - **CoT-style deduction grading**: The model first gives an overall assessment, then lists each deduction category and reason. - **Prompt-faithfulness first**: It evaluates prompt-image consistency rather than aesthetic quality, realism, or artistic preference. - **Fine-grained error taxonomy**: It supports categories such as `entity`, `appearance`, `activity`, `counting`, `shape`, `material`, `text`, and `composition`. - **Long-prompt ready**: Training uses `max_length=32768`, making the model suitable for complex prompts with many visual constraints. - **Built with ms-swift**: The model is trained with `swift sft` on top of Qwen3-VL-8B-Instruct for reproducible fine-tuning and evaluation. ## What It Does Input: 1. A generated image to be evaluated 2. The original prompt used to generate the image Output: ```text Score: 7/10, Total deduction: 3 Overall assessment: The main subject, composition, and overall style are mostly aligned with the prompt, but the biological details of the rear body do not fully match the requested insect abdomen. Deduction details: 1. appearance: The prompt asks for a bee-like abdomen and tail, but the generated rear body looks more like a furry cat tail instead of a typical insect abdomen. Deduct 3 points. ``` The model follows a **deduction-based consistency evaluation** scheme. It starts from a maximum score of 10 and deducts points for explicit, observable prompt constraints that are not correctly reflected in the generated image. ## Recommended Prompt The recommended evaluation prompt is: ```text Your task is to evaluate text-to-image consistency. Based on the given image and prompt, judge how well the generated image matches the prompt. Give a score from 1 to 10. First provide an overall assessment, then list the deduction details and error reasons. The prompt is: ``` Append the original generation prompt after this instruction and pass the image together with the text. ## Output Format For automatic parsing and downstream evaluation, we recommend the following format: ```text Score: <1-10>/10, Total deduction: <0-10> Overall assessment: Deduction details: 1. : , . Deduct points. 2. : , . Deduct points. ``` Common deduction categories: | Category | Meaning | | --- | --- | | `entity` | Missing entity, wrong entity, or mismatched subject | | `appearance` | Incorrect appearance, color, clothing, or local visual attribute | | `activity` | Wrong action, pose, gesture, or interaction | | `counting` | Incorrect number of objects or people | | `shape` | Incorrect shape, structure, or geometry | | `material` | Incorrect material, texture, or surface quality | | `text` | Missing, distorted, unreadable, or incorrect text | | `composition` | Incorrect viewpoint, layout, subject scale, or framing | ## Inference with ms-swift We recommend using `TransformersEngine` from `ms-swift`: ```python from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest MODEL_PATH = "yinggzhang/WeGenBench-Consistency-COT" IMAGE_PATH = "example.png" PROMPT = "A classic BMW sedan is parked in an indoor environment, with smooth body lines..." judge_prompt = ( "Your task is to evaluate text-to-image consistency. Based on the given image and prompt, " "judge how well the generated image matches the prompt. Give a score from 1 to 10. " "First provide an overall assessment, then list the deduction details and error reasons. " "The prompt is:" ) engine = TransformersEngine(MODEL_PATH, max_batch_size=1) request_config = RequestConfig(max_tokens=1024, temperature=0) request = InferRequest( messages=[{"role": "user", "content": judge_prompt + PROMPT}], images=[IMAGE_PATH], ) response = engine.infer([request], request_config)[0] print(response.choices[0].message.content) ``` ## Best For - Automatic text-to-image consistency scoring - Cross-model comparison and leaderboard pre-screening - Constraint-following analysis for complex prompts - Badcase attribution and error distribution analysis - Image-text data cleaning, filtering, and quality regression - Human annotation assistance ## Limitations - This model evaluates **prompt consistency**, not general aesthetics. - It should not be used as the final authority for highly specialized domains such as medical, legal, or safety-critical imagery. - Scores may still be affected by visual understanding errors, prompt ambiguity, and annotation style. - CoT-style explanations are useful for review and debugging, but production systems should still parse structured fields and keep human spot checks. - If the input prompt contains subjective, vague, or conflicting constraints, the model may produce conservative judgments. ## Citation If you find this model useful, please cite: ```bibtex @misc{liang2026wegenbenchmultidimensionaldiagnosticbenchmark, title={WeGenBench: A Multidimensional Diagnostic Benchmark towards Text-to-Image Model Optimization}, author={Qian Liang and Xiaomin Li and Ying Zhang and Jia Xu and Lihao Ni and Hongrui Li and Jingjing Li and Jing Lyu and Chen Li}, year={2026}, eprint={2606.20100}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2606.20100}, } ``` ## Contact Us For questions, feedback, or collaboration, please contact `yinggzhang@tencent.com`.