Live causal audio-visual emotion recognition
A causal audio-visual emotion recognizer: it predicts from past audio and video only, never future frames, so the same model runs offline on a clip and live on a webcam at 4 predictions/second. Recognises 6 emotions: angry, disgust, fear, happy, neutral, sad.
Everything needed to run is in this repo — it works fully offline after download.
Accuracy — measured, please read
CREMA-D fold01 test split: 18 speakers the model never trained on, near class-balanced 6-class, chance = 0.167. Selected on a held-out validation split (never on test).
| accuracy (WAR) | balanced accuracy (UAR) | |
|---|---|---|
| this model (causal, audio+visual) | 0.787 | 0.789 |
| its teacher (non-causal, sees the whole clip) | 0.826 | — |
| chance | 0.167 | 0.167 |
("WAR" in the source repo means weighted average recall = overall accuracy, not balanced accuracy; the two nearly coincide here only because this test split is near-balanced.)
Causality costs about 4 points against the teacher that gets to see the future.
Audio matters. Measured on a 72-clip slice of the same test split:
| mode | accuracy |
|---|---|
| audio + visual (as trained) | 0.81 |
| video only (no microphone) | 0.68 |
| audio only | 0.49 |
Turning the microphone off costs ~12 points; sad suffers most (0.58 → 0.25) — it is the
class the voice carries. The 72-clip slice is small, so treat those three as indicative.
Limitations — what this model is not
- Trained on acted emotion. CREMA-D is actors performing an emotion on command. This recognises acted expression, which is not the same thing as what a person feels. Do not use it to make judgements about real people, and especially not consequential ones.
- Never evaluated on webcam video. Every number here is CREMA-D clips: studio lighting, frontal faces. Your camera, lighting and framing are out of distribution — expect worse.
- Six classes. Anything else you feel gets forced into the nearest of these six.
- Not state of the art. CREMA-D SOTA is ~0.895 with a non-causal model.
- English-speaking North American adult actors. Performance across other demographics is unmeasured, and face models are known to vary across skin tone and age.
- The live-camera tab is video-only (a browser frame stream has no synchronised mic), so it runs at the 0.68 row above. The Record clip tab uses camera + microphone together and is the path the 0.787/0.81 numbers refer to.
Run it locally (fully offline)
git lfs install
git clone https://huggingface.co/mhussainahmad/averformer-causal-av-cremad
cd averformer-causal-av-cremad
pip install -r requirements.txt # plus ffmpeg from your system package manager
python run_local.py --check # verify the offline install
python run_local.py --video clip.mp4 # analyse a video file
python app.py # the full live camera + mic UI at localhost:7860
No network is used at any point after the clone: the checkpoint carries the fine-tuned
backbones, the encoder architectures are built locally, and the face detector is bundled.
Verified with HF_HUB_OFFLINE=1 and all HTTP proxies pointed at a dead port.
How it works
| stage | component | detail |
|---|---|---|
| face | YuNet (bundled) | largest face per frame, square crop, 224×224 |
| visual | HSEmotion EfficientNet-B0 (AffectNet) | per frame → 1280-d |
| audio | DistilHuBERT | per 0.5 s trailing window → 768-d |
| temporal | causal TSM head | past→present shift only |
| fusion | per-timestep late fusion | audio via causal cumulative mean |
| readout | mean over per-timestep logits | identical to the training/eval readout |
Causality is structural, not a training trick: the visual encoder sees one frame, the audio encoder one trailing window, and the temporal head only shifts past→present. The clip contract is 16 frames over 4 s → a 0.25 s hop (4 Hz) with a 0.5 s trailing audio window. Streaming encodes only the newest frame and window, reusing cached features for the rest.
weights/model_cremad_f01_s2024.pth (113 MB) is the whole model: both fine-tuned
backbones plus the head, 28.2 M parameters.
Training
CREMA-D fold01, seed 2024. Frozen-then-fine-tuned pretrained encoders, cross-entropy only, subject-disjoint validation split, weight EMA, epoch selected on validation accuracy.
There is no knowledge distillation in this model, despite the "distill" naming in the source repo — it was tried and it hurt (0.294 vs 0.42). Pretrained-encoder initialization is the entire lever. Reported as measured, not as designed.
Labels
Emitted in this order, verified by joining each CREMA-D filename's emotion code to its integer label across every row of every training fold (zero conflicts):
0 angry 1 disgust 2 fear 3 happy 4 neutral 5 sad
License & credits
MIT for this code. CREMA-D is released under the Open Database License. HSEmotion (Savchenko), DistilHuBERT (NTU SPML), YuNet (OpenCV Zoo) retain their own licenses.