Add HyperClick-3B model card with usage and evaluation

README.md

@@ -0,0 +1,135 @@
+---
+license: other
+license_name: qwen-research
+license_link: https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE
+base_model: Qwen/Qwen2.5-VL-3B-Instruct
+base_model_relation: finetune
+library_name: transformers
+pipeline_tag: image-text-to-text
+tags:
+- hyperclick
+- gui-grounding
+- multimodal
+- confidence-calibration
+- reinforcement-learning
+- arxiv:2510.27266
+---
+
+# HyperClick-3B
+
+[Paper](https://arxiv.org/pdf/2510.27266) · [Code](https://github.com/xiaomi-research/hyperclick) · [3B model](https://huggingface.co/SeerRay-Lab/Qwen2.5-VL-3B-HyperClick) · [7B model](https://huggingface.co/SeerRay-Lab/Qwen2.5-VL-7B-HyperClick)
+
+## Overview
+
+HyperClick grounds natural-language instructions in GUI screenshots and predicts a click point together with an explicit confidence score. This repository contains the **3B checkpoint**, based on [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), with full model weights in Safetensors format.
+
+The training framework combines supervised fine-tuning with reinforcement fine-tuning. Its rewards check output format, grounding correctness, and confidence alignment using a truncated Gaussian spatial target and the Brier score. See the [training code](https://github.com/xiaomi-research/hyperclick/blob/main/src/open-r1-multimodal/src/open_r1/hyperclick.py) for details.
+
+![HyperClick framework](https://raw.githubusercontent.com/xiaomi-research/hyperclick/main/assets/Framework.png)
+
+## Reported results
+
+Grounding accuracy (%) from the [project evaluation table](https://github.com/xiaomi-research/hyperclick#evaluation). These are the project's reported results, not a new evaluation of the uploaded files.
+
+| Model | ScreenSpot | ScreenSpot-v2 | ScreenSpot-Pro | MMBench-GUI | UI-I2E-Bench | CAGUI | UI-Vision |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| HyperClick-3B | 88.5 | 90.6 | 41.3 | 71.4 | 71.8 | 81.0 | 19.6 |
+| HyperClick-7B | 91.5 | 93.7 | 48.2 | 79.6 | 76.5 | 82.9 | 25.7 |
+
+## Quick start
+
+The example below uses Transformers on a CUDA GPU. Install PyTorch for your CUDA environment, then install the inference dependencies:
+
+```bash
+pip install "transformers==4.49.0" "accelerate==1.10.0" "qwen-vl-utils==0.0.11" pillow
+```
+
+Replace `screenshot.png` and the instruction with your own input. The prompt follows the HyperClick training template.
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
+from qwen_vl_utils import process_vision_info
+
+model_id = "SeerRay-Lab/Qwen2.5-VL-3B-HyperClick"
+model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+    model_id, torch_dtype=torch.bfloat16, device_map="auto"
+).eval()
+processor = AutoProcessor.from_pretrained(
+    model_id, min_pixels=3136, max_pixels=4390400, use_fast=False
+)
+
+screenshot = Image.open("screenshot.png").convert("RGB")
+instruction = "Click the search button"
+messages = [dict(role="user", content=[
+    dict(type="image", image=screenshot, min_pixels=3136, max_pixels=4390400),
+    dict(type="text", text=(
+        f'Point to the element related to the instruction "{instruction}" '
+        'on the screenshot with your confidence.'
+    )),
+])]
+prompt = processor.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True
+)
+images, _ = process_vision_info(messages)
+inputs = processor(text=[prompt], images=images, return_tensors="pt").to(model.device)
+with torch.inference_mode():
+    generated = model.generate(**inputs, max_new_tokens=128, do_sample=False)
+answer = processor.batch_decode(
+    generated[:, inputs.input_ids.shape[1]:], skip_special_tokens=True
+)[0]
+print(answer)
+
+# Dimensions of the image coordinate space used by the model.
+patch_size = processor.image_processor.patch_size
+_, grid_h, grid_w = inputs.image_grid_thw[0].tolist()
+input_width, input_height = grid_w * patch_size, grid_h * patch_size
+print("Model image size:", input_width, input_height)
+```
+
+### Output format and coordinates
+
+The expected output format is:
+
+```text
+<point>[x,y]</point><confidence>conf</confidence>
+```
+
+`x` and `y` are pixel coordinates in the processed image; `conf` is a confidence estimate between 0 and 1. To map a predicted point back to the original screenshot, use:
+
+```python
+# x and y are parsed from the model response.
+x_original = x * screenshot.width / input_width
+y_original = y * screenshot.height / input_height
+```
+
+Image resizing affects the coordinate system. Validate the response format and point bounds before using a prediction. Confidence is a learned estimate and can be incorrect, especially for unfamiliar interfaces or ambiguous instructions.
+
+## Training
+
+The [GitHub repository](https://github.com/xiaomi-research/hyperclick) provides training setup, example annotation formats, and the reinforcement fine-tuning entry points:
+
+```bash
+bash src/open-r1-multimodal/run_hyperclick_3b.sh
+```
+
+## License
+
+This model is derived from Qwen2.5-VL-3B-Instruct. See the upstream [Qwen Research License](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE) for the base model terms.
+
+## Citation
+
+The latest arXiv version is titled *Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning*.
+
+```bibtex
+@misc{zhang2025hyperclick,
+  title={Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning},
+  author={Shaojie Zhang and Pei Fu and Ruoceng Zhang and Jiahui Yang and Anan Du and Xiuwen Xi and Shaokang Wang and Ying Huang and Bin Qin and Zhenbo Luo and Jian Luan},
+  year={2025},
+  eprint={2510.27266},
+  archivePrefix={arXiv},
+  primaryClass={cs.CV},
+  url={https://arxiv.org/abs/2510.27266}
+}
+```