README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-Omni-7B
library_name: transformers
tags:
- omni-modal
- multimodal
- agent
- tool-use
- vision-language
- audio-language
- video-language
- omnigaia
- qwen2.5-omni
- sft
language:
- en
- zh
pipeline_tag: any-to-any
---

# Qwen2.5-Omni-7B SFT (with observations) — OmniGAIA fine-tune

This is a fully fine-tuned version of [Qwen/Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) on the **OmniGAIA** training data, trained as a tool-integrated omni-modal agent that natively reasons over **video + audio + image** and calls external tools (`web_search`, `page_browser`, `code_executor`, `read_image/audio/video`).

The key differences from the recipe in the [OmniGAIA paper](https://huggingface.co/spaces/RUC-NLPIR/OmniGAIA-Leaderboard) are:

- **Training data source.** We use the **raw `with_obs` trajectories from the OmniGAIA train split as-is**, rather than re-synthesizing trajectories via hindsight-guided tree exploration (the paper's OmniAtlas-SFT recipe).
- **Long-context handling via sliding window.** Long trajectories are split via **sliding window (width=8192, overlap=4096)** at user/assistant pair boundaries, so all observation context still fits in the model context window during training instead of being truncated.
- **Role-level masking equivalent to the paper's masked SFT.** Original `role=tool` (observation) messages are normalized to `role=user` to fit the Qwen2.5-Omni chat template. Combined with `ms-swift`'s default `--loss_scale default` (which only computes loss on `assistant` turns), this means **only assistant tokens contribute to the loss; tool / observation content is fully masked**, functionally the same as the paper's trajectory-level masked SFT.

> The **training objective is functionally equivalent to OmniAtlas-SFT** (only assistant tokens are supervised; tool / observation tokens are masked). The improvement over the paper's 7B numbers is therefore attributable to the **training-data recipe** (raw `with_obs` trajectories + sliding-window long-context handling), not to a different loss target.

This is `checkpoint-150` (epoch ≈ 1.28), which gave the most stable validation performance during training.

---

## Evaluation on OmniGAIA (Pass@1, official protocol)

Evaluated using the **official OmniGAIA eval script** (`run_base_agent_original_paper.py`) with **DeepSeek-V3.2 as the LLM judge**, same tools/active-perception setup as published baselines.

| Seed | Pass@1 (LLM_Equal) | EM | Avg tool calls | Non-empty rate |
| :--- | :---: | :---: | :---: | :---: |
| 42   | 16.94 | 8.61 | 12.16 | 100% |
| 1337 | 15.28 | 5.28 | 12.27 | 100% |
| 2024 | 14.44 | 6.11 | 11.91 | 100% |
| 7    | 16.39 | 4.17 | 11.86 | 100% |
| **Mean ± std (4 seeds)** | **15.76 ± 1.12** | **6.04 ± 1.89** | 12.05 | 100% |

### Comparison to OmniGAIA paper baselines (Qwen2.5-Omni-7B family)

| Method | Pass@1 |
| :--- | :---: |
| Qwen2.5-Omni-7B (zero-shot)   | 3.6 |
| + OmniAtlas-SFT (paper)       | 11.4 |
| + OmniDPO (paper)             | 13.3 |
| **+ with_obs SFT (this model)** | **15.76** |

> The cross-seed variance comes mainly from (i) real-time `web_search` results drifting between runs, and (ii) DeepSeek-V3.2 judge non-determinism, not from the model itself (vLLM is run in greedy mode).

### Per-difficulty breakdown (averaged over 4 seeds)

| Level | Count | Pass@1 |
| :--- | :---: | :---: |
| Easy   | 122 | 25.41 |
| Medium | 160 | 12.97 |
| Hard   |  78 |  7.05 |

---

## Training

| Item | Value |
| :--- | :--- |
| Base model | Qwen/Qwen2.5-Omni-7B |
| Trainer | ms-swift (full-param SFT, bf16) |
| Train data | OmniGAIA train split, kept as `with_obs` cached dataset (~2,150 trajectories), split via sliding window (width=8192, overlap=4096) → 4,682 windows |
| Epochs | 2 |
| Effective batch | 1 × 8 GPUs × `gradient_accumulation_steps=5` |
| Learning rate | 1e-5, warmup ratio 0.03 |
| Max length | 8,192 |
| Precision | bf16 |
| Attention | flash_attention_2 |
| Other | `padding_free`, `use_liger_kernel`, gradient & vit_gradient checkpointing, DeepSpeed ZeRO-2 |

This checkpoint corresponds to **global step 150 / 234** (epoch ≈ 1.28). Later checkpoints (200, 234) showed slightly higher train loss reduction but no monotonic Pass@1 gain in our evaluation, so we use step 150 for the leaderboard submission.

### Data contamination self-check

We verified that **no test-set question text from `test_metadata.json` appears verbatim in any training sample** (0 / 2,151 overlap). The training set shares media with the test set (~23%) because the OmniGAIA dataset is constructed from a shared media pool — this is the same condition under which all OmniAtlas / Qwen baselines in the paper were evaluated, so the comparison is apples-to-apples.

---

## How to use

```python
from transformers import Qwen2_5OmniForConditionalGeneration, AutoProcessor

model_path = "ZhangYuchi/modelbest-Qwen-2.5-Omni-7B-SFT-only"

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    model_path,
    torch_dtype="bfloat16",
    device_map="auto",
    trust_remote_code=True,
)
```

For best results with OmniGAIA-style multi-turn tool-use evaluation, serve it with vLLM:

```bash
vllm serve <model_path> \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 32768
```

and drive it via the OmniGAIA agent loop (`run_base_agent_original_paper.py --enable-active-perception ...`) with `web_search`, `page_browser`, `code_executor`, and `read_image/audio/video` tools.

## Intended use & limitations

- Designed for **omni-modal agentic QA** with multi-turn tool use over video + audio + image.
- Inherits all limitations of `Qwen2.5-Omni-7B` (audio perception errors on long clips, vision recognition errors on rare entities, etc.).
- The training data is heavy on tool-call trajectories — direct one-shot answers without tool use are not what this checkpoint is optimized for.

## Caveats and known limitations (read this before comparing to the paper)

1. **Functionally equivalent loss masking, different role wrapper.**
   `tool` / observation messages are remapped to `role=user` (to satisfy the Qwen2.5-Omni chat template), and `ms-swift` then masks all non-`assistant` tokens via `--loss_scale default`. The end effect — supervising only `assistant` tokens (reasoning + `tool_call`) — matches the paper's trajectory-level masked SFT, but **the on-disk role schema differs**. Downstream tooling that special-cases `role=tool` may not recognize the observation segments in this dataset / format.

2. **Trajectories are not re-synthesized.**
   The paper's OmniAtlas-SFT uses hindsight-guided tree exploration with Gemini-3-Flash to synthesize new tool-integrated trajectories. This model trains directly on the raw `with_obs` trajectories shipped in the OmniGAIA train split, without any re-synthesis or trajectory cleanup.

3. **No DPO / RLHF stage.**
   The paper reports its strongest 7B number with `OmniAtlas-SFT + OmniDPO`. This model is SFT-only.

4. **Shared media pool between train and test.**
   The OmniGAIA train and test sets are constructed from a shared media pool (~23% of train trajectories share at least one media file with a test item, even though **no test-set question text appears verbatim in any train sample** — see "Data contamination self-check" above). This is a property of the benchmark itself, the same as all baselines in the paper.

What this implies for the reported numbers:

- The 15.76 Pass@1 figure is achieved under the **same evaluation protocol and the same effective training objective** (masked SFT) as the paper's 7B baselines. The headroom over the paper's `+OmniAtlas-SFT (11.4)` and `+OmniDPO (13.3)` therefore reflects differences in **training data recipe** (raw `with_obs` + sliding window) rather than a different loss mask.
- A clean apples-to-apples ablation (paper-style trajectory synthesis vs raw `with_obs`, holding everything else fixed) has not been run in this codebase, so the per-component contribution of "raw with_obs" vs "sliding window" vs "different trajectory pool" is not yet decomposed.
- Use this checkpoint as a **strong reproducible 7B starting point** for OmniGAIA. If you intend to extend this for a publication, please run the corresponding ablations.

## License

Apache-2.0, inherited from the base [Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B).

## Citation

If you use this model, please also cite the underlying base model and the OmniGAIA benchmark:

```bibtex
@misc{qwen25omni,
  title  = {Qwen2.5-Omni Technical Report},
  author = {Qwen Team},
  year   = {2025},
}

@misc{omnigaia,
  title  = {OmniGAIA: Towards Native Omni-Modal AI Agents},
  author = {RUC-NLPIR et al.},
  year   = {2026},
  url    = {https://huggingface.co/spaces/RUC-NLPIR/OmniGAIA-Leaderboard},
}
```