File size: 3,576 Bytes
29f0e7b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 | ---
license: other
pipeline_tag: image-text-to-text
tags:
- gui-world-model
- mobile-gui
- text-generation
- qwen3-vl
- multimodal
language:
- en
---
# MobileWorld-TEXT-8B
`MobileWorld-TEXT-8B` is a mobile GUI world model. Given the current screenshot and a candidate action, it predicts the next screen state as a complete text description.
Use this model when the downstream component needs a full textual description of the predicted next page rather than HTML code or only the changed region.
## Input
The model is used as a multimodal chat model. Provide:
- One current GUI screenshot.
- A system prompt describing the full next-page prediction task.
- A user prompt containing the action description, action target, and relative coordinates.
### System Prompt
```text
You are a graphical user interface (GUI) agent. You are given an action, an action target, the relative coordinates of the object being manipulated, and a screenshot. Your task is to predict the state of the next page after performing this action. Note that you need to provide a complete and detailed description of the next page, not an incremental description based on changes to the current page.
```
### User Prompt Template
```text
<image> Predict the next page state via text description from this current screenshot using action description "{action_description}" and action tartget "{action_target}" and relative coordinates "{relative_coordinates}".
```
The word `tartget` is intentionally kept as in the training/inference template.
### Fields
- `action_description`: natural-language action, for example `click`, `scroll down`, or `input text: hello`.
- `action_target`: target UI element or target point, for example `point(536, 1280)` or `search input field`.
- `relative_coordinates`: normalized coordinates in `(x, y)` format, for example `(0.496296, 0.533333)`. Use `N/A` for actions without a coordinate target.
## Output
The expected output is a complete natural-language description of the predicted next screen. It should describe the resulting page state, not just the delta from the current screen.
## Example OpenAI-Compatible Call
```python
import base64
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
def image_url(path):
payload = base64.b64encode(open(path, "rb").read()).decode("utf-8")
return f"data:image/png;base64,{payload}"
system_prompt = """You are a graphical user interface (GUI) agent. You are given an action, an action target, the relative coordinates of the object being manipulated, and a screenshot. Your task is to predict the state of the next page after performing this action. Note that you need to provide a complete and detailed description of the next page, not an incremental description based on changes to the current page."""
user_prompt = '<image> Predict the next page state via text description from this current screenshot using action description "input text: pizza" and action tartget "search input field" and relative coordinates "N/A".'
response = client.chat.completions.create(
model="xwk123/MobileWorld-TEXT-8B",
messages=[
{"role": "system", "content": system_prompt},
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": image_url("screenshot.png")}},
{"type": "text", "text": user_prompt},
],
},
],
max_tokens=4000,
temperature=0.0,
)
next_page_description = response.choices[0].message.content.strip()
```
|