OmniFysics-Captioner: Grounding Omni-Modal Understanding in the Physical World for Better Captioning
🌐 Project • 🤗 Captioner • 🕵️ OmniFysics-Agent • 🧰 Tool: PPM • 📚 Citation
Introduction
Building omni-modal models with physical intelligence requires fine-grained supervision that captures physical evidence such as contact, support, deformation, and state transitions. Existing captioners provide detailed visual or event-level description, but do not always preserve the physical evidence behind object interactions and state changes together.
OmniFysics-Captioner is an end-to-end, tool-free omni-modal Captioner that reads raw audio and video in a single forward pass and directly generates physics-aware audiovisual captions. Trained with the active-perception agent OmniFysics-Agent, it amortizes evidence acquisition and organization without requiring external tools at inference. Within the Agent, a physical perception model (PPM) fine-tuned on approximately 2M image-level samples serves as a dedicated tool for extracting object-interaction and state-change cues.
Contents
Model
OmniFysics-Captioner
OmniFysics-Captioner is an end-to-end, tool-free omni-modal Captioner fine-tuned from Qwen3-Omni on Daily-Physics 50K. It reads raw audio and video in a single forward pass and directly generates physics-aware, detailed captions, amortizing the evidence-acquisition and organization capability of OmniFysics-Agent without requiring external tools at inference.
At deployment, it generates a caption from raw audio and video in one forward pass without external tools.
🤗 Fysics-AI/OmniFysics-Captioner
OmniFysics-Agent
OmniFysics-Agent forms a global event timeline from a low-cost audiovisual proxy, locates local intervals that require further inspection, and dynamically orchestrates modality-specific tools. Each observation batch is written to Evidence Memory and drives the next Plan–Execute–Observe–Reflect round, progressively refining modality choice, temporal scope, and question focus. The acquired evidence is spatiotemporally aligned and traceable, and a finalizer organizes it into a temporally coherent caption.
Tool: Physical Perception Model (PPM)
Within OmniFysics-Agent, the PPM focuses on objects and physical phenomena in representative frames, perceiving physical cues such as material, contact, and deformation, and analyzing object interactions, state changes, and their potential outcomes. It is fine-tuned on approximately 2M image-level physical-perception samples to output object-centric, physical-aware evidence that complements the audio and visual tools.
The released PPM checkpoint is available on Hugging Face:
🤗 Fysics-AI/OmniFysics-Captioner/PPM
Paper
Paper and supplementary material: Coming soon.
Citation
@article{qiu2026omnifysicscaptioner,
title = {OmniFysics-Captioner: Grounding Omni-Modal Understanding in the Physical World for Better Captioning},
author = {Qiu, Kaixiang and Han, Minghao and Liu, Keliang and Liu, Yizhou and Han, Jinghang and Jiang, Yue and Wang, Shunli and Zhang, Lihua and Yang, Dingkang},
journal = {arXiv preprint},
year = {2026}
}
License
The content of this repository is released under the Apache License 2.0 with an additional non-commercial restriction: it may be used, reproduced, and distributed for research and educational purposes only. Any commercial use is prohibited without prior written permission from the maintainers. Source videos remain subject to the licenses of their original datasets.
- Downloads last month
- 174