xwk123 commited on
Commit
29f0e7b
·
verified ·
1 Parent(s): f0d3fef

Add prompt usage README

Browse files
Files changed (1) hide show
  1. README.md +86 -0
README.md ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ pipeline_tag: image-text-to-text
4
+ tags:
5
+ - gui-world-model
6
+ - mobile-gui
7
+ - text-generation
8
+ - qwen3-vl
9
+ - multimodal
10
+ language:
11
+ - en
12
+ ---
13
+
14
+ # MobileWorld-TEXT-8B
15
+
16
+ `MobileWorld-TEXT-8B` is a mobile GUI world model. Given the current screenshot and a candidate action, it predicts the next screen state as a complete text description.
17
+
18
+ Use this model when the downstream component needs a full textual description of the predicted next page rather than HTML code or only the changed region.
19
+
20
+ ## Input
21
+
22
+ The model is used as a multimodal chat model. Provide:
23
+
24
+ - One current GUI screenshot.
25
+ - A system prompt describing the full next-page prediction task.
26
+ - A user prompt containing the action description, action target, and relative coordinates.
27
+
28
+ ### System Prompt
29
+
30
+ ```text
31
+ You are a graphical user interface (GUI) agent. You are given an action, an action target, the relative coordinates of the object being manipulated, and a screenshot. Your task is to predict the state of the next page after performing this action. Note that you need to provide a complete and detailed description of the next page, not an incremental description based on changes to the current page.
32
+ ```
33
+
34
+ ### User Prompt Template
35
+
36
+ ```text
37
+ <image> Predict the next page state via text description from this current screenshot using action description "{action_description}" and action tartget "{action_target}" and relative coordinates "{relative_coordinates}".
38
+ ```
39
+
40
+ The word `tartget` is intentionally kept as in the training/inference template.
41
+
42
+ ### Fields
43
+
44
+ - `action_description`: natural-language action, for example `click`, `scroll down`, or `input text: hello`.
45
+ - `action_target`: target UI element or target point, for example `point(536, 1280)` or `search input field`.
46
+ - `relative_coordinates`: normalized coordinates in `(x, y)` format, for example `(0.496296, 0.533333)`. Use `N/A` for actions without a coordinate target.
47
+
48
+ ## Output
49
+
50
+ The expected output is a complete natural-language description of the predicted next screen. It should describe the resulting page state, not just the delta from the current screen.
51
+
52
+ ## Example OpenAI-Compatible Call
53
+
54
+ ```python
55
+ import base64
56
+ from openai import OpenAI
57
+
58
+ client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
59
+
60
+ def image_url(path):
61
+ payload = base64.b64encode(open(path, "rb").read()).decode("utf-8")
62
+ return f"data:image/png;base64,{payload}"
63
+
64
+ system_prompt = """You are a graphical user interface (GUI) agent. You are given an action, an action target, the relative coordinates of the object being manipulated, and a screenshot. Your task is to predict the state of the next page after performing this action. Note that you need to provide a complete and detailed description of the next page, not an incremental description based on changes to the current page."""
65
+
66
+ user_prompt = '<image> Predict the next page state via text description from this current screenshot using action description "input text: pizza" and action tartget "search input field" and relative coordinates "N/A".'
67
+
68
+ response = client.chat.completions.create(
69
+ model="xwk123/MobileWorld-TEXT-8B",
70
+ messages=[
71
+ {"role": "system", "content": system_prompt},
72
+ {
73
+ "role": "user",
74
+ "content": [
75
+ {"type": "image_url", "image_url": {"url": image_url("screenshot.png")}},
76
+ {"type": "text", "text": user_prompt},
77
+ ],
78
+ },
79
+ ],
80
+ max_tokens=4000,
81
+ temperature=0.0,
82
+ )
83
+
84
+ next_page_description = response.choices[0].message.content.strip()
85
+ ```
86
+