cstr/fasttext-lid176-GGUF

fastText LID-176 — GGUF

GGUF build of Facebook's lid.176.bin fastText supervised LID classifier covering 176 languages with ISO 639-1 short codes (en, de, fr, ru, zh, ja, …).

Designed to drop into the crispasr runtime alongside the GlotLID-V3 GGUF for post-ASR text language identification.

⚠️ License — CC-BY-SA-3.0 (viral)

The upstream lid.176.bin is distributed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA-3.0). Anyone redistributing this GGUF — or anything derived from it that includes its weights — must keep the same license terms:

• Attribute Facebook (the upstream authors).
• Distribute under CC-BY-SA-3.0 or a compatible license.

This is more restrictive than most ASR/LID models on the Hub. If that does not match your project's license posture, use the GlotLID-V3 GGUF (Apache-2.0) instead — it covers a strict superset of LID-176's languages.

Files

Quant	Size	Notes
F16	63 MB	only quant shipped — the input matrix is small (dim=16) so K-quants degrade quality more than they save space

K-quants require 256-element row alignment; LID-176's dim=16 rows are below that, so K-quants would silently fall back to legacy Q4_0 / Q5_0 / Q8_0. The F16 build is already small enough that quantization is not worth the loss in precision — the embedding bag fits entirely in L1 cache regardless.

Architecture — hierarchical softmax

Unlike GlotLID (flat softmax), LID-176 uses fastText's hierarchical softmax (HS) loss with a deterministic Huffman tree built from training-frequency label counts. The output matrix's n_labels rows parameterize internal tree nodes; per-label log-probability is computed by walking from the leaf to the root and summing log_sigmoid((2*code - 1) * (output[node] · hidden)) along the path.

Tensors:

• lid_fasttext.embedding.weight [n_words + bucket, dim] = [2,040,010, 16]
• lid_fasttext.output.weight [n_labels, dim] = [176, 16]
• lid_fasttext.hs_path_offsets [n_labels + 1] i32 — CSR-style path offsets per label
• lid_fasttext.hs_paths [total_steps] i32 — flattened internal-node indices
• lid_fasttext.hs_codes [total_steps] i8 — flattened code bits (0/1) per step

Hyperparams:

• n_words = 40,010
• bucket = 2,000,000
• dim = 16
• n_labels = 176
• minn, maxn = 2, 4
• loss = hs
• wordNgrams = 1
• HS tree: 1,855 total path steps, average path length 10.54

Use with crispasr

Standalone text-LID CLI:

crispasr-lid -m fasttext-lid176-f16.gguf --text "Hallo Welt"
# de  0.999

crispasr-lid -m fasttext-lid176-f16.gguf -k 3 < transcript.txt

Post-ASR via main crispasr binary:

crispasr -m ggml-base.bin -f input.wav \
    --lid-on-transcript fasttext-lid176-f16.gguf
# lang=en  conf=0.934

Validation

Per-stage diff harness against a Python ground-truth dump (uses fasttext.predict() as the reference for HS top-1 labels):

[PASS] input_ids          shape=[163]   cos=1.000000  max_abs=0.00e+00
[PASS] embedding_bag_out  shape=[16]    cos=1.000000  max_abs=3.55e-05
[PASS] logits             shape=[176]   cos=1.000000  max_abs=1.48e-03
[PASS] softmax            shape=[176]   cos=1.000000  max_abs=6.54e-05
[PASS] top1_score         shape=[1]     cos=1.000000
[INFO] top1_label         ours='fr'  ref='fr'

8/8 multilingual smoke tests correctly identified (en/de/fr/ru/zh/ja/pt; mr/hi can swap on 2-word Devanagari inputs — this is upstream model behaviour, not a port artefact).

Conversion

Built via crispasr's models/convert-glotlid-to-gguf.py (the same converter handles both flat-softmax GlotLID and HS LID-176 — variant flag selects which):

python models/convert-glotlid-to-gguf.py \
    --variant fasttext-lid176 \
    --input  /path/to/lid.176.bin \
    --output fasttext-lid176-f16.gguf \
    --dtype  f16

The converter parses lid.176.bin directly to extract per-label training frequencies, deterministically rebuilds fastText's Huffman tree (port of Model::buildTree from the upstream source), and emits the path/code metadata alongside the weights.

Citation & license

Upstream:

• Joulin, A., Grave, E., Bojanowski, P., Mikolov, T. Bag of Tricks for Efficient Text Classification. EACL 2017.
• Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., Mikolov, T. FastText.zip: Compressing text classification models.

This GGUF rebuild inherits the upstream CC-BY-SA-3.0 license. Redistributors must preserve the SA terms.

Note on base_model (corrected 2026-08-02)

This repo's frontmatter pointed at facebook/fasttext-language-identification, which is a different model — the NLLB-based LID-201 classifier, published under CC-BY-NC-4.0. The weights here are Facebook's original lid.176.bin, distributed from fasttext.cc under CC-BY-SA-3.0, exactly as the licence section above describes.

The pointer has been removed rather than repaired, because lid.176.bin has no Hub repo to point at. The licence field was already correct and is unchanged — recording that here because an automated sweep flagged this repo as a licence mismatch on the strength of the wrong pointer, and relabelling it non-commercial would have imposed a restriction the upstream does not.