voxsum-qwen35-0.8b-anchored

On-device meeting summarizer for VoxSumDroid. Reads a transcript format v1 meeting transcript and writes NOTES format v2 — title, summary, decisions, action items, open questions and topics, with a [m:ss] timestamp on every bullet so the app can jump from a note to the audio.

Built for ≥80k-token meetings in Traditional Chinese and English, under a 1 GB RAM ceiling, on Snapdragon-662-class hardware, via llama.cpp.

Files

file	size
voxsum-qwen35-0.8b-anchored-Q4_0.gguf	478 MB — recommended, matches the QAT numerics
voxsum-qwen35-0.8b-anchored-Q4_K_M.gguf	505 MB
harness/	the long-document pipeline (see below)

Quality

Judged by gemma-4-26B-A4B-it (no family relationship to this model or to any of its teachers), n=20 stratified long meetings, 10 zh-TW / 10 en, 12k–40k tokens, 8k windows, model served at its trained numerics.

	faith /5	inversions	cover /5
this model	4.60	5%	4.45
LFM2.5-350M	2.90	55%	3.05
LFM2.5-230M	2.65	45%	3.36
Gemma-3-270M	1.92	33%	2.42

Paired sign test on the same meetings: this model wins on 18–20 of 20 against every alternative, zero losses, p=0.000. The alternatives are indistinguishable from one another (p=0.23–1.00).

Inversions — notes stating the opposite of the transcript about a decision, approval, outcome or commitment — are the metric that matters most for meeting notes. 5% vs 45–55% is the gap that decided this model.

Honest limits

• Judge noise is roughly ±0.4 faith; the same judge scored identical output 2.90 and 3.30 on two passes. Treat differences under ~0.5 as ties.
• n=20 is a data ceiling, not a choice: the zh-TW evaluation pool holds 26 meetings, 10–11 of them above 12k tokens.
• Do not judge changes with a Qwen model. Qwen3.5-9B taught 42k rows of this model's training mix and rated its own distillate a uniform 5.00/5.00/0%. gemma-3-27b-it is also unusable — it returned 5.00/0% for every arm we tried, including ones Gemma-4 scored 2.50.
• Our zh evaluation set is largely monologic (lectures, interviews, livestreams) while the en set is multi-party contested meetings. Every model scores higher on zh, and that gap is partly dataset composition rather than language ability. Contested-Chinese performance is unmeasured.

Speed and memory (RPi4, Cortex-A72 ×4 @1.8 GHz, as an SD662 proxy)

context	prefill tok/s	peak RSS
2,048	13.02	705 MB
8,192	7.74	785 MB
16,384	5.70	892 MB
decode	4.13	—

8k windows is optimal on both axes. Peak RSS 785 MB fits the 1 GB budget.

Architecture note: this is a hybrid model — layer_types is 6 full_attention layers (indices 3, 7, 11, 15, 19, 23) interleaved with 18 linear_attention layers. Only the full-attention layers hold an O(context) KV cache, so KV is 6,144 values per token (6 x 2 kv_heads x head_dim 256 x 2), not the 24,576 a fully-dense reading of the config would suggest. At 8k that is roughly 100 MB at f16; --cache-type-k q4_0 --cache-type-v q4_0 takes it to ~25 MB and is the model's trained configuration. The saving is real but modest — most of the 785 MB is weights and activations, not KV.

An 80k transcript is ≈3.2 h by this proxy — arithmetic from prefill rates, not a timed end-to-end run. It is measured under SUSTAINED thermal throttling (the RPi4 sits at 84-85 °C, above its 80 °C threshold), which is the correct operating point for a multi-hour job: three-repeat runs settle at 7.51 ± 0.05 (Q4_K_M) and 7.71 ± 0.01 (Q4_0), while a single cold run reads 10.50 and is not sustainable. Q4_0 is marginally faster than Q4_K_M under load as well as smaller. The RPi4 is ARMv8.0 with no dotprod and 4 cores; SD662 has 8 cores and may expose dotprod, in which case real hardware will be materially faster. Measure on device.

Usage

llama-server -m voxsum-qwen35-0.8b-anchored-Q4_0.gguf -c 8192 --jinja \
  --cache-type-k q4_0 --cache-type-v q4_0

{
  "messages": [{ "role": "user", "content": "<NOTES prompt + transcript window>" }],
  "max_tokens": 640,
  "temperature": 0,
  "chat_template_kwargs": { "enable_thinking": false }   // REQUIRED
}

enable_thinking: false is mandatory. Without it the reply lands in reasoning_content and content comes back empty. The --reasoning-budget 0 server flag alone does not do it.

Use the prompts in harness/prompts.py — the deployed templates, which are a different string from the anchor-demanding variant used to build the training data.

The harness

A single window will not hold an 80k transcript. harness/longdoc.py implements the measured pipeline:

transcript → compact() → windows(8000, 2-line overlap)
           → per-window NOTES → parse_notes() → merge_deterministic()
           → reduce_sections()   (only sections over cap)
           → render()

All orchestration is deterministic; the model only summarizes a window and occasionally shrinks an over-cap section. Two details that matter:

• spread(), not items[:cap]. Taking the first N merged items keeps only the earliest window and discards the end of every meeting — measured: a section spanning 0–39m collapsed to six bullets all at [0:00].
• The model's own reduce can collapse anchors. reduce_sections() compares its output's time span against the deterministic pick and prefers the deterministic one when the model's span is under 60% of it.

Do not use a three-op (chunk/merge/title) agentic harness. Measured on a 350M model it scored faith 1.42 with 75% inversions, versus 2.09 with 18% single-pass: deterministic orchestration amplifies a weak model rather than compensating for it.

Training

Distilled from Qwen3.6-35B-A3B (anchored NOTES) and Qwen3.5-9B (the remaining task suite), 44,391 samples across 11 tasks, faithfulness-gated. QAT throughout: int4-block32 weights, int8 output head, q4 KV cache — so the deployed Q4 numerics are what the model trained against.

A same-family teacher was tried and failed: LFM2-24B-A2B anchored only 20% of transcripts, at a modal faithfulness of 2 with 27% inversions, yielding 87 usable rows.