---
license: cc-by-nc-4.0
language:
- en
pipeline_tag: feature-extraction
tags:
- embeddings
- imu
- inertial
- motion
- accelerometer
- sensor
- robotics
- human-activity-recognition
- qwen3-vl
base_model: EximiusLabs/fusion-embedding-2-2b-preview
---
# fusion-embedding-2-tremor
[](https://github.com/Eximius-Labs/fusion-embedding) [](https://github.com/Eximius-Labs/fusion-embedding) [](#license) [](#) [](https://github.com/Eximius-Labs/fusion-embedding)
[](https://www.runpod.io/console/hub/Eximius-Labs/tremor)
**Tremor** is the inertial (IMU) sensor pack for Eximius Labs' fusion-embedding stack.
It maps a short window of body-worn motion (a 3-axis accelerometer stream) into the
[Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) text
embedding space, so a robot's or wearable's motion history becomes searchable in plain
language: "find when it was walking", "find when it picked something up".
Tremor is part of the **fusion-embedding family** and plugs into the same shared space as
[fusion-embedding-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview). It
targets the frozen Qwen3-VL-Embedding-2B text embeddings that the family is built on, and
because fusion-embedding-2's text, image, and video paths are byte-identical to that base,
Tremor's motion embeddings are **directly comparable to that model's audio, text, image, and
video** in one 2048-d space. Motion becomes a first-class, language-addressable modality
alongside the rest.
[GitHub](https://github.com/Eximius-Labs/fusion-embedding) | [fusion-embedding-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview) | Technical report: in preparation
## Model Overview
Tremor is a **frozen [UniMTS](https://github.com/xiyuanzh/UniMTS) motion encoder plus a
small trained projector**. A single accelerometer window is placed at one joint of
UniMTS's orientation-invariant skeleton-graph encoder (ST-GCN), which produces a 512-d
motion feature; a projector of roughly 2.6M parameters maps that into the frozen Qwen
base's 2048-d text space. Nothing is trained except the projector. Motion and text
embeddings are L2-normalized and compared by cosine similarity.
The design choice that matters is the encoder: rather than train an IMU encoder from
scratch, Tremor reuses a frozen, orientation-invariant pretrained one and learns only
the projection into the language space. The pretraining prior, not our training, is
what carries cross-subject and cross-mounting generalization.
| Feature | Value |
| --- | --- |
| Parameters | ~2.06B frozen Qwen base + frozen UniMTS encoder; **2.6M trained** |
| Modality | inertial motion (3-axis accelerometer) |
| Supported tasks | `zero-shot activity retrieval`, `zero-shot activity classification` |
| Input | one accelerometer window `[3, T]` (any length/rate; resampled to 200 samples @ 20 Hz) |
| Sensor placement | single IMU, mapped to one skeleton joint (robots/wearables have a known, fixed mount) |
| Embedding dimension | 2048 |
| Pooling strategy | last-token pooling (text side) |
| Base model | Qwen/Qwen3-VL-Embedding-2B (frozen) |
| Motion encoder | UniMTS ST-GCN, accelerometer-only (frozen) |
| Trained components | projector 2.6M `LayerNorm(512) -> Linear(512,1024) -> GELU -> Linear(1024,2048)` |
| Distribution | ~10 MB trained projector; the frozen encoder and base download from their own repos |
## See it in action
**A full activity sequence, read live.** A **Unitree G1** performs each activity in simulation — sitting, walking, running, climbing — while a window slides across a real, held-out body-worn accelerometer stream ([RealWorld](https://sensor.informatik.uni-mannheim.de/), out of training) and Tremor's scores update in real time, tracking the motion as it changes. The robot shows each activity full-body; the signal and the predictions are real, frame-synced data.
**On the robot's own IMU and camera.** On a real **Unitree G1 humanoid** ([Humanoid-Everyday](https://usc-psi-lab.github.io/humanoid-everyday/)), Tremor reads the robot's own IMU as language, live and frame-synced to its **egocentric camera** — zero simulation. As it switches from adjusting a phone stand to walking to a door, its IMU signature changes from flat to dynamic and Tremor's read tracks the activity in real time.
**Any body-worn sensor.** No robot is required — the same works on a wristband or wearable. Here the raw accelerometer stream slides by directly and Tremor's scores track sitting → walking → running → climbing on the held-out RealWorld set.
**Text → motion retrieval.** Query an activity in plain language and Tremor returns the nearest motion windows. Below, on the RealWorld set held out of training entirely (zero-shot), each thumbnail is a retrieved window's 3-axis accelerometer trace; correct retrievals (the window's true activity matches the query) are outlined in green.
## Training and Evaluation
Tremor is trained with InfoNCE against the frozen base's native chat-template text
embeddings, on a **diverse 16-source pool of real inertial data**: 13 standard
human-activity-recognition datasets, [Ego4D](https://ego4d-data.org/) (egocentric
human), Humanoid-Everyday (Unitree robot), and
[Capture24](https://ora.ox.ac.uk/objects/uuid:99d7c092-d865-4a19-b096-cc16440cd001)
(151 subjects of free-living wrist accelerometry). Only the projector learns.
The single most important lesson of the project is the **evaluation metric**. Instance
retrieval (finding the exact paired window) is misleading for this task and makes a
working activity recognizer look dead. The right metric is **k-way activity
classification on held-out subjects and held-out datasets**, and all numbers below use
it (chance = 1/k).
### Diversity is what makes the base general
Trained on one source (Ego4D alone), an IMU encoder generalizes poorly to sensor
streams that look nothing like it. Tremor's headline result is **zero-shot transfer to
datasets held out of training entirely**. On five held-out datasets, a single-source
baseline (the identical architecture trained on Ego4D only) sits at or below chance,
while Tremor generalizes.
| Held-out dataset (5-way, zero-shot) | Single-source baseline | **Tremor v0.1** |
| --- | --- | --- |
| RealWorld | 0.16 | **0.68** |
| USC-HAD | 0.13 | **0.59** |
| DSADS | 0.29 | **0.58** |
| UTD-MHAD | 0.21 | **0.33** |
| **held-out mean** | **0.20** (chance) | **0.545** |
On the two in-domain holds (held-out subjects/episodes of sources seen in training),
Tremor reaches 0.40 5-way on both Ego4D and the Humanoid-Everyday robot set. The
held-out datasets above are excluded from training entirely, so those numbers are
genuine cross-dataset generalization.
## Usage
Requirements
- `torch` (CUDA recommended), `numpy`, `transformers>=4.46`, `huggingface_hub`
- The frozen UniMTS encoder code and weights (Apache-2.0 code):
```bash
git clone https://github.com/xiyuanzh/UniMTS # provides model.py::ST_GCN_18
```
Point `UNIMTS_REPO` (or `unimts_repo=`) at the clone. The UniMTS weights download
automatically from the hub.
- The frozen Qwen3-VL-Embedding-2B base downloads from its original repository.
via inference.py (this repository)
```python
import numpy as np
from inference import TremorEmbedder
tr = TremorEmbedder.from_pretrained(
"EximiusLabs/fusion-embedding-2-tremor",
revision="v0.1-preview",
unimts_repo="UniMTS", # path to your git clone of xiyuanzh/UniMTS
)
# accel: a 3-axis accelerometer window, shape [3, T] (any length / sample rate)
accel = np.load("window.npy") # e.g. [3, 300]
# rank candidate activities by similarity to the motion
for text, score in tr.rank(accel, ["walking", "sitting", "running", "climbing stairs"]):
print(f"{score:+.3f} {text}")
# or embed motion and text into the shared space directly
m = tr.embed_motion(accel) # 2048-d, L2-normalized
t = tr.embed_text(["walking", "sitting"]) # 2048-d each
```
Embedding quality is sensitive to the base's chat-template formatting; `inference.py`
applies it for you, so match candidate activities against motion through this API rather
than embedding text yourself with a different template.
## Related models
Tremor comes in two tiers, both built on
[fusion-embedding-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview):
| Model | Best for | Accuracy |
| --- | --- | --- |
| **This model — general base** | any sensor / platform, unseen datasets (zero-shot) | **0.545** held-out 5-way (unseen datasets) |
| **[fusion-embedding-2-tremor-g1](https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor-g1)** | deployment on the **Unitree humanoid** (G1/H1) | **0.740** in-domain 5-way |
Start with this **general base** — it is the tier that generalizes to sensors it has never seen. Move
to the per-fleet **Tremor-G1** head only when you deploy on that platform and want maximum in-domain
accuracy. Specializing a head raises in-domain accuracy but does not improve cross-dataset transfer;
the two do different jobs. Deployment guidance (sensor contract, streaming, on-device inference,
per-fleet fine-tuning) is in the Tremor-G1 card.
## Deploy on RunPod
One-click from the [RunPod Hub](https://www.runpod.io/console/hub/Eximius-Labs/tremor).
```bash
curl -s https://api.runpod.ai/v2//runsync \n -H "Authorization: Bearer $RUNPOD_API_KEY" \n -H "Content-Type: application/json" \n -d '{"input": {"motion": [[/*x*/], [/*y*/], [/*z*/]]}}'
```
`motion` is a 3-axis accelerometer window shaped [3, T]. Use `text` to embed an activity query. The Unitree G1 head deploys separately at https://www.runpod.io/console/hub/Eximius-Labs/tremor-g1 . Returns 2048-d vectors.
## License
The trained projector weights in this repository are released under
**[CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/)** (non-commercial).
This posture is deliberate and reflects the model's lineage. Tremor's frozen encoder is
[UniMTS](https://github.com/xiyuanzh/UniMTS), which was pretrained on synthetic IMU
derived from the [AMASS](https://amass.is.tue.mpg.de/) / HumanML3D motion corpora. AMASS
is released under a non-commercial research license, so we ship Tremor non-commercial to
respect that lineage. A commercially-clean encoder (re-pretrained on permissively
licensed motion) is future work; a commercial license may follow.
## Limitations
- **Accelerometer only.** The released UniMTS encoder is accelerometer-only; gyroscope
is not used. Rotational-rate information is not currently modeled.
- **Cross-dataset ceiling.** Zero-shot cross-dataset accuracy plateaus around 0.55
(5-way). This is a research preview, not a production activity classifier. The right
use is language-addressable motion search, not high-stakes classification.
- **Single-sensor, single-joint.** One IMU mapped to one skeleton joint, matching a
fixed-mount deployment (a robot torso, a wristband). Multi-sensor fusion is not
modeled.
- **English text only**, and the base's chat-template formatting must be used.
- **Specialization vs. generalization.** Fine-tuning on a specific platform's data
raises accuracy on that platform substantially but does not improve cross-dataset
transfer. Tremor is the general base; per-fleet fine-tuning is a separate recipe.
## Citation
If you use Tremor, please cite this repository and acknowledge the frozen components it
builds on:
```bibtex
@misc{tremor2026,
title = {Tremor: an inertial-motion sensor pack for the fusion-embedding space},
author = {Tonmoy, Abdul Basit},
year = {2026},
note = {Eximius Labs. Model weights CC-BY-NC-4.0.},
url = {https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor}
}
```
Tremor's frozen motion encoder is **UniMTS** (Zhang et al., *UniMTS: Unified Pre-training
for Motion Time Series*, NeurIPS 2024), pretrained on synthetic IMU derived from
**AMASS** (Mahmood et al., ICCV 2019). The text space is **Qwen3-VL-Embedding-2B**.
Please cite those works as well.