---
license: gpl-3.0
tags:
- onnx
- language-identification
- fasttext
library_name: onnx
pipeline_tag: text-classification
base_model: laurievb/OpenLID-v2
---

# OpenLID-v2 - ONNX

ONNX export of [`laurievb/OpenLID-v2`](https://huggingface.co/laurievb/OpenLID-v2)
`model.bin`, an improved fastText supervised language identifier covering
200 language varieties in `iso639-3_Script` label form (for example
`eng_Latn`, `por_Latn`, `glg_Latn`), built on the
[OpenLID-v2 dataset](https://huggingface.co/datasets/laurievb/OpenLID-v2), an
updated version of Burchell et al., *An Open Dataset and Model for Language
Identification* (ACL 2023).

The licence is **GPL-3.0**, inherited unchanged from the original model.

> [!NOTE]
> The upstream model card recommends normalising/cleaning text with
> `openlid_normer.clean_line` before classification for best results. This
> export does not vendor that normalizer - it hashes raw text exactly like
> `TigreGotico/openlid-onnx` and `TigreGotico/glotlid-onnx` do, and parity was
> measured against fastText's own `predict()` on raw (uncleaned) text, so the
> two stay comparable. Apply the same cleaning step yourself before calling
> the featurizer if you want to match the upstream-recommended pipeline.

## What the graph does and does not do

fastText inference has five steps. The ONNX graph holds only the last two:

| Step | Where |
|---|---|
| 1. Tokenize the text | Python (`openlid_v2_hash.py`) |
| 2. Character n-grams per word (`minn`..`maxn`) | Python |
| 3. Hash the n-grams into buckets | Python |
| 4. Average the embedding rows of all feature ids | **ONNX** |
| 5. Multiply by the output matrix, then softmax | **ONNX** |

Steps 1 to 3 are string processing. ONNX has no portable operator for
fastText's FNV-1a byte hash over UTF-8, so that work stays in Python. The
split keeps the graph a pure numeric pipeline:

```
input_ids -> Gather(input_matrix) -> ReduceMean(axis=0) -> MatMul(output_matrix) -> Softmax -> probs
```

`openlid_v2_hash.py` is the reference implementation of steps 1 to 3, ported
from the same recipe used for [`TigreGotico/glotlid-onnx`](https://huggingface.co/TigreGotico/glotlid-onnx).
Two details are easy to get wrong:

* fastText casts each byte to a **signed** `int8_t` before the FNV-1a XOR, so
  bytes >= 0x80 are sign-extended. Without this, every non-ASCII n-gram lands
  in the wrong bucket.
* Each line ends with the `</s>` end-of-sentence token, and that token
  contributes its own vocabulary row.

This model uses plain softmax loss (not hierarchical softmax), so it does
not need the Huffman-tree combination step that `TigreGotico/lid176-onnx`
(fastText's classic `lid.176.bin`, trained with `loss=hs`) requires.

## Files

| File | Size | Purpose |
|---|---|---|
| `openlid-v2.onnx` | 1.13 GB | fp32 graph |
| `openlid-v2.int8.onnx` | 290 MB | dynamic int8 graph |
| `labels.json` | ~4 KB | 200 labels, in output order |
| `vocab.txt` | 1.8 MB | 185286 vocabulary words, in id order |
| `config.json` | 122 B | `dim`, `minn`, `maxn`, `bucket`, `nwords`, `nlabels`, `loss` |
| `openlid_v2_hash.py` | 5.2 KB | reference feature extractor |

## Model arguments

```
dim=256  minn=2  maxn=5  bucket=1000000  wordNgrams=1  loss=softmax
nwords=185286  nlabels=200
input_matrix=(1185286, 256)   # nwords + bucket
output_matrix=(200, 256)
```

## Tensors

| Name | Direction | Type | Shape |
|---|---|---|---|
| `input_ids` | input | int64 | `[num_features]` |
| `probs` | output | float32 | `[200]` |

`probs` is indexed by the order of `labels.json`.

## Usage

```python
import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download

from openlid_v2_hash import FastTextFeaturizer

d = snapshot_download("TigreGotico/openlid-v2-onnx")
feat = FastTextFeaturizer.from_files(f"{d}/vocab.txt", f"{d}/config.json")
labels = json.load(open(f"{d}/labels.json", encoding="utf-8"))
sess = ort.InferenceSession(f"{d}/openlid-v2.onnx", providers=["CPUExecutionProvider"])

def detect(text, k=5):
    probs = sess.run(None, {"input_ids": feat(text)})[0]
    top = np.argsort(-probs)[:k]
    return [(labels[i], float(probs[i])) for i in top]

print(detect("O tempo está moi bo hoxe en Santiago"))
# [('__label__glg_Latn', 0.9...), ...]
```

## Parity with fastText

59 short samples spanning 59 languages, chosen for script and resource
diversity: Portuguese, Galician, Catalan, Basque, Spanish, English, French,
German, Italian, Dutch, Arabic, Chinese, Japanese, Russian, Hindi, Greek,
Ukrainian, Swahili, Turkish, Polish, Czech, Finnish, Hungarian, Romanian,
Swedish, Hebrew, Persian, Thai, Vietnamese, Indonesian, Tagalog, Amharic,
Hausa, Yoruba, Igbo, Zulu, Somali, Bengali, Tamil, Telugu, Malayalam, Nepali,
Georgian, Armenian, Icelandic, Welsh, Irish, Maltese, Esperanto, Quechua,
Guarani, Haitian Creole, Malagasy, Kinyarwanda, Mongolian, Khmer, Lao,
Burmese, Sinhala.

Each sample was compared against fastText's own `f.predict(text + "\n", 1,
0.0, "strict")` (the pybind entry point - `fasttext-wheel`'s Python
`.predict()` wrapper is broken under numpy>=2), on raw uncleaned text (see
the note above about the upstream normalizer).

| Model | Top-1 agreement |
|---|---|
| `openlid-v2.onnx` (fp32) | **100.00 %** (59/59) |
| `openlid-v2.int8.onnx` | **100.00 %** (59/59) |

## Citation

```
@inproceedings{burchell-etal-2023-open,
    title = "An Open Dataset and Model for Language Identification",
    author = "Burchell, Laurie and Birch, Alexandra and Bogoychev, Nikolay and Heafield, Kenneth",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    year = "2023",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-short.75",
}
```