# Deploying Tremor

Tremor turns a body-worn or robot-mounted **3-axis accelerometer** stream into the
`Qwen3-VL-Embedding-2B` text space, so motion becomes searchable and classifiable in plain
language. This guide covers choosing a variant, the sensor contract, the two runtime modes, and
on-device integration.

## 1. Pick a variant

Tremor ships in two tiers, both in this repository.

| Variant | File | When to use | In-domain acc. | Zero-shot transfer |
| --- | --- | --- | --- | --- |
| **General base** | `tremor_projector.pt` (root) | Any sensor / platform, unseen datasets, low-data fleets | moderate | **strong** (0.545 held-out 5-way, unseen datasets) |
| **Tremor-G1** | `g1/tremor_projector.pt` | Deployment **on the Unitree humanoid** (G1/H1), head-mounted IMU | **high** (0.740 in-domain 5-way) | not the goal |

Rule of thumb: **start with the general base.** It is the one that generalizes to sensors it has
never seen. Move to a per-fleet head (like Tremor-G1) only when you are deploying on that exact
platform and want maximum in-domain accuracy. Specializing a head raises in-domain accuracy but
does **not** improve cross-dataset transfer — the two tiers serve different jobs.

```python
from inference import TremorEmbedder

# general base (default) — any body-worn / robot sensor
tr = TremorEmbedder.from_pretrained("EximiusLabs/fusion-embedding-2-tremor",
                                    revision="v0.1-preview", unimts_repo="UniMTS")

# Unitree-humanoid head — same repo, the `g1` subfolder
g1 = TremorEmbedder.from_pretrained("EximiusLabs/fusion-embedding-2-tremor",
                                    subfolder="g1", unimts_repo="UniMTS")
```

Both variants share the identical pipeline and API; only the trained projector differs (the
per-fleet head omits the input LayerNorm — the loader selects the architecture from the checkpoint
config automatically).

## 2. Sensor contract

- **Signal:** a single 3-axis **accelerometer**. Gyroscope is not used by the released encoder.
- **Shape:** `accel` as `np.ndarray` of shape `[3, T]` (x, y, z over time). Any length, any sample
  rate — the pipeline resamples the time axis to **200 samples at 20 Hz** internally.
- **Window:** a few seconds of motion per embedding works well (the model was trained on short
  activity windows). For streaming, slide a window with overlap (see §4).
- **Mount:** one IMU mapped to one skeleton joint (default joint 15). Deployments have a **fixed,
  known mount** (a robot torso/head, a wristband), which is exactly the regime Tremor targets. The
  UniMTS encoder is orientation-invariant, so exact mounting orientation is forgiving, but keep the
  mount consistent between calibration and use.

## 3. Two runtime modes

The heavy component at inference is the frozen **Qwen3-VL-Embedding-2B** text base. How you deploy
depends on whether your activity vocabulary is fixed.

### A. Fixed vocabulary (recommended for on-device / classification)
If you rank motion against a **known set of activity phrases**, embed those phrases **once, offline**
and ship the resulting 2048-d vectors. At inference you then run only the tiny UniMTS encoder +
projector + a cosine dot-product — **the 2B text base is not needed on the device.**

```python
import torch, torch.nn.functional as F

VOCAB = ["standing still", "walking", "running", "climbing stairs", "picking something up"]
text_bank = tr.embed_text(VOCAB)                 # [V, 2048]; compute once, save to disk
torch.save({"vocab": VOCAB, "bank": text_bank}, "activity_bank.pt")

# on device (no Qwen base): motion -> nearest activity
m = tr.embed_motion(accel)                        # [2048]
scores = (text_bank @ m)                          # cosine (both are L2-normalized)
pred = VOCAB[int(scores.argmax())]
```

This keeps the runtime footprint to the ~10 MB projector + the small UniMTS ST-GCN encoder.

### B. Open vocabulary (language search over a motion log)
Embed each motion window into 2048-d and store it in any vector index. Query later with free-form
text via `embed_text` — "find when it was walking", "find when it picked something up". Here the
text base is needed at query time (or run queries on a server).

## 4. Streaming

- Slide a window (e.g. 5–10 s) with a stride (e.g. 0.3–0.5 s) across the incoming accelerometer
  stream; embed each window with `embed_motion`.
- **Smooth the read** over ~1 s (a short moving average of the per-window scores) to suppress
  flicker at activity boundaries — this is what the model-card demos do.
- Precompute the text/vocabulary bank once (mode A) so each step is just encode + cosine.

## 5. Performance notes

- **Precision:** the base runs in bf16; the encoder + projector are small and can run fp32 on CPU.
- **Batching:** `embed_motion` accepts one window; batch by stacking windows through the encoder if
  you need throughput. Text embedding is padding-sensitive — embed phrases individually (the API
  does this) or bucket by exact length.
- **Chat template:** motion is aligned to text embedded with the base's chat template. Always embed
  candidate phrases through `embed_text` (which applies it); do not hand-roll the template.

## 6. Per-fleet fine-tuning (optional)

To build your own platform head (like Tremor-G1) for a new robot or wearable: collect that
platform's accelerometer windows with activity labels, keep the UniMTS encoder frozen, and train a
fresh projector against the base's text embeddings (InfoNCE). This raises in-domain accuracy on that
platform substantially. It is a **specialization** recipe — it will not improve zero-shot transfer to
other sensors, so keep the general base for anything outside the trained platform.

## 7. Limits

- Accelerometer-only; gyroscope not modeled. Single IMU, single joint. English text only.
- Research preview: cross-dataset accuracy plateaus around 0.55 (5-way). Use it for
  language-addressable motion search and coarse activity readout, not high-stakes classification.

See the model card (`README.md`) for the method, evaluation, and live demos.