fernandotonon commited on
Commit
b272e71
·
verified ·
1 Parent(s): 9740fa9

Update card for v4 architecture + corrected 6D

Browse files
Files changed (1) hide show
  1. README.md +36 -23
README.md CHANGED
@@ -11,43 +11,56 @@ tags:
11
 
12
  A small, **experimental** from-scratch text-to-motion model for
13
  [QtMeshEditor](https://github.com/fernandotonon/QtMeshEditor). Given a text
14
- prompt (an action keyword), it generates a 40-frame, 22-joint **canonical**
15
- skeletal clip that QtMeshEditor retargets onto an arbitrary humanoid rig.
 
 
 
 
 
16
 
17
  ## Status: experimental
18
 
19
- Render-verified quality is **action-dependent**: locomotion (e.g. `walk`) is
20
- coherent; gestures (`wave`, `run`) can drift in the last few frames. The shipped
21
- default in QtMeshEditor is the deterministic **template-clip retarget**; this
22
- model is an opt-in (`--model` / GUI checkbox / MCP `model:true`) that **falls
23
- back to the template** automatically when unavailable or out of vocabulary.
 
 
24
 
25
  ## Training data — permissive only
26
 
27
- Trained from scratch on **clean, dynamic, single-action windows** mined from:
28
- - **CMU MoCap** (commercial-OK) the bulk.
29
- - **Quaternius Universal Animation Library** (CC0) supplementary.
30
-
31
- AMASS / HumanML3D / KIT-ML were **excluded** (non-commercial). Idle/near-static
32
- and multi-action-labelled windows were filtered out (they dominated the raw set
33
- and caused pose collapse).
34
 
35
- ## Architecture
36
 
37
- 6D-rotation representation, cross-attention decoder with a delta-integration
38
- (cumsum) head for temporal continuity, CVAE latent, balanced sampling. ~7.6M
39
- params. Exports to ONNX (one forward pass).
 
 
 
 
 
40
 
41
  ## I/O contract
42
 
43
  ```
44
  input "tokens" float32 [1, V] one-hot over the fixed action vocab (see t2m-vocab.json)
45
- input "seed" float32 [1, Z] latent (zeros = mean clip)
46
  output "motion" float32 [1, T, C] C = 22*10 per-joint [tx,ty,tz, qx,qy,qz,qw, sx,sy,sz]
47
  ```
48
 
49
- `t2m-vocab.json` ships the `{vocab, Z, T, C, J, joints}` the host needs to build
50
- the input and interpret the output.
 
 
 
 
51
 
52
- Vocabulary: walk, run, jog, jump, dance, march, climb, kick, punch, sit,
53
- stretch, throw, wave, boxing, turn, forward.
 
11
 
12
  A small, **experimental** from-scratch text-to-motion model for
13
  [QtMeshEditor](https://github.com/fernandotonon/QtMeshEditor). Given a text
14
+ prompt (an action keyword), it generates a 60-frame @30fps, 22-joint
15
+ **canonical WORLD-frame** skeletal clip that QtMeshEditor retargets onto an
16
+ arbitrary humanoid rig.
17
+
18
+ > The model QtMeshEditor actually downloads at runtime lives in the shared
19
+ > [`fernandotonon/QtMeshEditor-models`](https://huggingface.co/fernandotonon/QtMeshEditor-models)
20
+ > repo under `motion/`. This repo is the standalone model card + mirror.
21
 
22
  ## Status: experimental
23
 
24
+ The shipped **default** in QtMeshEditor is the deterministic **template-clip
25
+ retarget** (a curated library of 47 real CMU mocap clips across 15 actions,
26
+ with per-action variety) — that is the quality bar. This model is an opt-in
27
+ (`--model` / GUI checkbox / MCP `model:true`) that **falls back to the
28
+ template** automatically when unavailable or out of vocabulary. It produces
29
+ coherent, upright motion with per-generate variety, but is stylistically
30
+ gentler/less crisp than the real-mocap templates.
31
 
32
  ## Training data — permissive only
33
 
34
+ Trained from scratch on **clean, dynamic, single-action windows** mined from
35
+ the **CMU MoCap** database (commercial-OK). AMASS / HumanML3D / KIT-ML were
36
+ **excluded** (non-commercial). Windows are 30fps, 2s, selected for motion
37
+ energy and snapped to a calm near-neutral start frame; mirror-augmented.
 
 
 
38
 
39
+ ## Architecture (v4)
40
 
41
+ - **6D-rotation** representation (Zhou et al. 2019), correctly column-packed.
42
+ - Cross-attention transformer decoder with an **absolute** per-frame pose head
43
+ (self-attention models temporal coherence; no error-accumulating cumsum).
44
+ - CVAE latent with z=0 supervision + aggregate-posterior matching.
45
+ - Per-sample velocity/acceleration matching in both 6D and true rotation
46
+ (geodesic) space; derived-local supervision (the quantity the retarget
47
+ renders); 1-2-1 output smoothing baked into the ONNX graph.
48
+ - ~7.6M params, exports to ONNX (one forward pass).
49
 
50
  ## I/O contract
51
 
52
  ```
53
  input "tokens" float32 [1, V] one-hot over the fixed action vocab (see t2m-vocab.json)
54
+ input "seed" float32 [1, Z] latent noise (host samples ~N(0,0.5) and does best-of-N)
55
  output "motion" float32 [1, T, C] C = 22*10 per-joint [tx,ty,tz, qx,qy,qz,qw, sx,sy,sz]
56
  ```
57
 
58
+ `t2m-vocab.json` ships the `{vocab, Z, T, C, J, joints, fps, frame}` the host
59
+ needs `frame: "world"` marks the WORLD-frame convention (retarget takes a
60
+ world delta), `fps: 30`. Vocabulary: walk, run, jump, dance, march, kick,
61
+ punch, wave, climb, sit, throw, boxing, idle.
62
+
63
+ ## Reproducing
64
 
65
+ `scripts/prep-t2m-v4.py` + `scripts/train-t2m-onnx-v4.py` in the QtMeshEditor
66
+ repo (one-time, offline dev tools — the app never runs Python).