Update README.md

9e8a97c verified about 1 month ago

6.73 kB

	---
	license: apache-2.0
	language:
	- en
	base_model:
	- Roboflow/rf-detr-medium
	pipeline_tag: object-detection
	library_name: transformers
	tags:
	- text-generation-inference
	- mobile-gui-detection
	datasets:
	- mrtoy/mobile-ui-design
	---

	<div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; border: 1px solid #e2e8f0; border-radius: 8px; box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.05); overflow: hidden; background: #ffffff; margin-bottom: 30px;">
	<div style="background: #1e40af; padding: 20px; color: white;">
	<h1 style="margin: 0; font-size: 22px; font-weight: 700; color: white; border: none;">rf-detr-mobile-gui-detection</h1>
	<p style="margin: 8px 0 0 0; font-size: 13px; color: #bfdbfe; font-weight: 500;">Mobile GUI grounding model built on top of roboflow/rf-detr-medium</p>
	</div>
	<div style="display: flex; gap: 8px; flex-wrap: wrap; padding: 12px 20px; background: #f8fafc; border-bottom: 1px solid #e2e8f0;">
	<span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">Object Detection</span>
	<span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">DETR</span>
	<span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">Mobile GUI</span>
	<span style="background: #ffffff; color: #1e40af; font-size: 11px; font-weight: 700; padding: 4px 10px; border-radius: 4px; border: 1px solid #93c5fd; text-transform: uppercase; letter-spacing: 0.5px;">Grounding</span>
	</div>
	<div style="padding: 20px; display: flex; flex-direction: column; gap: 16px;">
	<p style="margin: 0; font-size: 14px; color: #334155; line-height: 1.6;"><b>rf-detr-mobile-gui-detection</b> is a mobile gui grounding model built on top of <b>roboflow/rf-detr-medium</b> using the <b>rfdetrforobjectdetection</b> architecture. rf-detr is an end-to-end object detection model that combines ideas from lw-detr and deformable detr: a dinov2-with-registers-style vit backbone, an rf-detr windowing pattern for efficient attention, a multi-scale projector between the encoder and decoder, and a multi-scale deformable detr decoder for fast convergence and strong accuracy-latency tradeoffs.</p>

	<div style="border: 1px solid #fde68a; padding: 14px; border-radius: 4px; background: #fffbeb;">
	<span style="font-weight: 700; color: #92400e; font-size: 12px; display: block; margin-bottom: 6px; text-transform: uppercase; letter-spacing: 0.5px;">Note</span>
	<p style="margin: 0; font-size: 13px; color: #475569; line-height: 1.5;">rf-detr: neural architecture search for real-time detection transformers: <a href="https://huggingface.co/papers/2511.09554" style="color: #92400e; text-decoration: underline;">https://huggingface.co/papers/2511.09554</a></p>
	</div>
	</div>
	</div>

	## Metrics Loss Map

	![metrics_loss_map](https://huggingface.co/prithivMLmods/rf-detr-mobile-gui-detection/resolve/main/metrics_loss_map.png)

	## Per Class Metrics

	![per_class_metrics](https://huggingface.co/prithivMLmods/rf-detr-mobile-gui-detection/resolve/main/per_class_metrics.png)

	## Quick Start with Transformers

	```
	pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
	pip install torchvision==0.23.0 transformers==5.9.0 accelerate gradio==6.19.0
	```

	```py
	import gradio as gr
	import torch
	from PIL import Image, ImageDraw

	from transformers import AutoImageProcessor, RfDetrForObjectDetection

	# Load model and processor
	model_name = "prithivMLmods/rf-detr-mobile-gui-detection"

	processor = AutoImageProcessor.from_pretrained(model_name)
	model = RfDetrForObjectDetection.from_pretrained(model_name)

	# Detection threshold
	THRESHOLD = 0.35


	def detect_gui(image):
	image = Image.fromarray(image).convert("RGB")

	inputs = processor(images=image, return_tensors="pt")

	with torch.no_grad():
	outputs = model(**inputs)

	target_sizes = torch.tensor([image.size[::-1]])
	results = processor.post_process_object_detection(
	outputs,
	target_sizes=target_sizes,
	threshold=THRESHOLD,
	)[0]

	draw = ImageDraw.Draw(image)

	detections = []

	for score, label, box in zip(
	results["scores"],
	results["labels"],
	results["boxes"],
	):
	box = [round(x, 2) for x in box.tolist()]
	label_name = model.config.id2label[label.item()]
	confidence = round(score.item(), 3)

	# Draw bounding box
	draw.rectangle(box, outline="red", width=3)

	# Draw label
	draw.text(
	(box[0] + 4, max(0, box[1] - 16)),
	f"{label_name} {confidence:.2f}",
	fill="red",
	)

	detections.append(
	{
	"Label": label_name,
	"Confidence": confidence,
	"Bounding Box": box,
	}
	)

	return image, detections


	demo = gr.Interface(
	fn=detect_gui,
	inputs=gr.Image(type="numpy", label="Upload Mobile UI Screenshot"),
	outputs=[
	gr.Image(type="pil", label="Detected GUI Elements"),
	gr.JSON(label="Detections"),
	],
	title="RF-DETR Mobile GUI Detection",
	description="Upload a mobile UI screenshot to detect GUI elements using RF-DETR.",
	)

	if __name__ == "__main__":
	demo.launch()
	```

	> e.g., demo screenshot

	![screencapture-c959e285e9de4018e9-gradio-live-2026-06-28-21_22_41](https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/I0LeSI-y43mKKy9tKMa-0.png)

	## Acknowledgements

	* [roboflow/rf-detr-medium](https://huggingface.co/Roboflow/rf-detr-medium): rf-detr is an end-to-end object detection model that combines ideas from lw-detr and deformable detr: a dinov2-with-registers-style vit backbone (with an rf-detr windowing pattern for efficient attention), a multi-scale projector between the encoder and decoder, and a multi-scale deformable detr decoder for fast convergence and strong accuracy-latency tradeoffs.

	* [mobile ui design detection[dataset]](https://huggingface.co/datasets/mrtoy/mobile-ui-design) by [mrtoy](https://huggingface.co/mrtoy): this dataset is designed for object detection tasks focused on detecting elements in mobile ui designs. the target objects include text, images, and groups. the dataset contains mobile ui images with object detection bounding boxes, class labels, and localization information.