Upload folder using huggingface_hub

README.md

@@ -0,0 +1,152 @@
+---
+license: gpl-3.0
+tags:
+  - onnx
+  - language-identification
+  - fasttext
+library_name: onnx
+pipeline_tag: text-classification
+---
+
+# OpenLID-v2 - ONNX
+
+ONNX export of [`laurievb/OpenLID-v2`](https://huggingface.co/laurievb/OpenLID-v2)
+`model.bin`, an improved fastText supervised language identifier covering
+200 language varieties in `iso639-3_Script` label form (for example
+`eng_Latn`, `por_Latn`, `glg_Latn`), built on the
+[OpenLID-v2 dataset](https://huggingface.co/datasets/laurievb/OpenLID-v2), an
+updated version of Burchell et al., *An Open Dataset and Model for Language
+Identification* (ACL 2023).
+
+The licence is **GPL-3.0**, inherited unchanged from the original model.
+
+> [!NOTE]
+> The upstream model card recommends normalising/cleaning text with
+> `openlid_normer.clean_line` before classification for best results. This
+> export does not vendor that normalizer - it hashes raw text exactly like
+> `TigreGotico/openlid-onnx` and `TigreGotico/glotlid-onnx` do, and parity was
+> measured against fastText's own `predict()` on raw (uncleaned) text, so the
+> two stay comparable. Apply the same cleaning step yourself before calling
+> the featurizer if you want to match the upstream-recommended pipeline.
+
+## What the graph does and does not do
+
+fastText inference has five steps. The ONNX graph holds only the last two:
+
+| Step | Where |
+|---|---|
+| 1. Tokenize the text | Python (`openlid_v2_hash.py`) |
+| 2. Character n-grams per word (`minn`..`maxn`) | Python |
+| 3. Hash the n-grams into buckets | Python |
+| 4. Average the embedding rows of all feature ids | **ONNX** |
+| 5. Multiply by the output matrix, then softmax | **ONNX** |
+
+Steps 1 to 3 are string processing. ONNX has no portable operator for
+fastText's FNV-1a byte hash over UTF-8, so that work stays in Python. The
+split keeps the graph a pure numeric pipeline:
+
+```
+input_ids -> Gather(input_matrix) -> ReduceMean(axis=0) -> MatMul(output_matrix) -> Softmax -> probs
+```
+
+`openlid_v2_hash.py` is the reference implementation of steps 1 to 3, ported
+from the same recipe used for [`TigreGotico/glotlid-onnx`](https://huggingface.co/TigreGotico/glotlid-onnx).
+Two details are easy to get wrong:
+
+* fastText casts each byte to a **signed** `int8_t` before the FNV-1a XOR, so
+  bytes >= 0x80 are sign-extended. Without this, every non-ASCII n-gram lands
+  in the wrong bucket.
+* Each line ends with the `</s>` end-of-sentence token, and that token
+  contributes its own vocabulary row.
+
+This model uses plain softmax loss (not hierarchical softmax), so it does
+not need the Huffman-tree combination step that `TigreGotico/lid176-onnx`
+(fastText's classic `lid.176.bin`, trained with `loss=hs`) requires.
+
+## Files
+
+| File | Size | Purpose |
+|---|---|---|
+| `openlid-v2.onnx` | 1.13 GB | fp32 graph |
+| `openlid-v2.int8.onnx` | 290 MB | dynamic int8 graph |
+| `labels.json` | ~4 KB | 200 labels, in output order |
+| `vocab.txt` | 1.8 MB | 185286 vocabulary words, in id order |
+| `config.json` | 122 B | `dim`, `minn`, `maxn`, `bucket`, `nwords`, `nlabels`, `loss` |
+| `openlid_v2_hash.py` | 5.2 KB | reference feature extractor |
+
+## Model arguments
+
+```
+dim=256  minn=2  maxn=5  bucket=1000000  wordNgrams=1  loss=softmax
+nwords=185286  nlabels=200
+input_matrix=(1185286, 256)   # nwords + bucket
+output_matrix=(200, 256)
+```
+
+## Tensors
+
+| Name | Direction | Type | Shape |
+|---|---|---|---|
+| `input_ids` | input | int64 | `[num_features]` |
+| `probs` | output | float32 | `[200]` |
+
+`probs` is indexed by the order of `labels.json`.
+
+## Usage
+
+```python
+import json
+import numpy as np
+import onnxruntime as ort
+from huggingface_hub import snapshot_download
+
+from openlid_v2_hash import FastTextFeaturizer
+
+d = snapshot_download("TigreGotico/openlid-v2-onnx")
+feat = FastTextFeaturizer.from_files(f"{d}/vocab.txt", f"{d}/config.json")
+labels = json.load(open(f"{d}/labels.json", encoding="utf-8"))
+sess = ort.InferenceSession(f"{d}/openlid-v2.onnx", providers=["CPUExecutionProvider"])
+
+def detect(text, k=5):
+    probs = sess.run(None, {"input_ids": feat(text)})[0]
+    top = np.argsort(-probs)[:k]
+    return [(labels[i], float(probs[i])) for i in top]
+
+print(detect("O tempo está moi bo hoxe en Santiago"))
+# [('__label__glg_Latn', 0.9...), ...]
+```
+
+## Parity with fastText
+
+59 short samples spanning 59 languages, chosen for script and resource
+diversity: Portuguese, Galician, Catalan, Basque, Spanish, English, French,
+German, Italian, Dutch, Arabic, Chinese, Japanese, Russian, Hindi, Greek,
+Ukrainian, Swahili, Turkish, Polish, Czech, Finnish, Hungarian, Romanian,
+Swedish, Hebrew, Persian, Thai, Vietnamese, Indonesian, Tagalog, Amharic,
+Hausa, Yoruba, Igbo, Zulu, Somali, Bengali, Tamil, Telugu, Malayalam, Nepali,
+Georgian, Armenian, Icelandic, Welsh, Irish, Maltese, Esperanto, Quechua,
+Guarani, Haitian Creole, Malagasy, Kinyarwanda, Mongolian, Khmer, Lao,
+Burmese, Sinhala.
+
+Each sample was compared against fastText's own `f.predict(text + "\n", 1,
+0.0, "strict")` (the pybind entry point - `fasttext-wheel`'s Python
+`.predict()` wrapper is broken under numpy>=2), on raw uncleaned text (see
+the note above about the upstream normalizer).
+
+| Model | Top-1 agreement |
+|---|---|
+| `openlid-v2.onnx` (fp32) | **100.00 %** (59/59) |
+| `openlid-v2.int8.onnx` | **100.00 %** (59/59) |
+
+## Citation
+
+```
+@inproceedings{burchell-etal-2023-open,
+    title = "An Open Dataset and Model for Language Identification",
+    author = "Burchell, Laurie and Birch, Alexandra and Bogoychev, Nikolay and Heafield, Kenneth",
+    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
+    year = "2023",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2023.acl-short.75",
+}
+```

config.json

@@ -0,0 +1,9 @@
+{
+  "dim": 256,
+  "minn": 2,
+  "maxn": 5,
+  "bucket": 1000000,
+  "nwords": 185286,
+  "nlabels": 200,
+  "loss": "softmax"
+}

labels.json

@@ -0,0 +1 @@
+["__label__eng_Latn", "__label__arb_Arab", "__label__rus_Cyrl", "__label__por_Latn", "__label__pol_Latn", "__label__ekk_Latn", "__label__ell_Grek", "__label__slk_Latn", "__label__pes_Arab", "__label__slv_Latn", "__label__nld_Latn", "__label__hun_Latn", "__label__lvs_Latn", "__label__dan_Latn", "__label__swe_Latn", "__label__lit_Latn", "__label__fin_Latn", "__label__cmn_Hant", "__label__mlt_Latn", "__label__nob_Latn", "__label__ind_Latn", "__label__uzn_Latn", "__label__fil_Latn", "__label__ukr_Cyrl", "__label__hin_Deva", "__label__cmn_Hans", "__label__afr_Latn", "__label__mar_Deva", "__label__ceb_Latn", "__label__heb_Hebr", "__label__ilo_Latn", "__label__zul_Latn", "__label__xho_Latn", "__label__jpn_Jpan", "__label__vie_Latn", "__label__guj_Gujr", "__label__amh_Ethi", "__label__hrv_Latn", "__label__nya_Latn", "__label__tsn_Latn", "__label__sna_Latn", "__label__tso_Latn", "__label__tha_Thai", "__label__spa_Latn", "__label__deu_Latn", "__label__eus_Latn", "__label__tur_Latn", "__label__bul_Cyrl", "__label__fra_Latn", "__label__ewe_Latn", "__label__nso_Latn", "__label__tam_Taml", "__label__mya_Mymr", "__label__twi_Latn", "__label__lin_Latn", "__label__yor_Latn", "__label__ben_Beng", "__label__urd_Arab", "__label__ibo_Latn", "__label__ita_Latn", "__label__tir_Ethi", "__label__azj_Latn", "__label__tpi_Latn", "__label__run_Latn", "__label__kin_Latn", "__label__ron_Latn", "__label__ces_Latn", "__label__sin_Sinh", "__label__kat_Geor", "__label__zsm_Latn", "__label__pap_Latn", "__label__mkd_Cyrl", "__label__bem_Latn", "__label__mal_Mlym", "__label__kir_Cyrl", "__label__smo_Latn", "__label__hye_Armn", "__label__kan_Knda", "__label__fij_Latn", "__label__pan_Guru", "__label__kor_Hang", "__label__als_Latn", "__label__hau_Latn", "__label__epo_Latn", "__label__gaz_Latn", "__label__srp_Cyrl", "__label__hat_Latn", "__label__lua_Latn", "__label__pag_Latn", "__label__war_Latn", "__label__pbt_Arab", "__label__tel_Telu", "__label__tat_Cyrl", "__label__sag_Latn", "__label__lug_Latn", "__label__oci_Latn", "__label__tum_Latn", "__label__npi_Deva", "__label__swh_Latn", "__label__umb_Latn", "__label__ktu_Latn", "__label__bos_Latn", "__label__gle_Latn", "__label__mos_Latn", "__label__lus_Latn", "__label__som_Latn", "__label__khk_Cyrl", "__label__tuk_Latn", "__label__quy_Latn", "__label__ayr_Latn", "__label__luo_Latn", "__label__tgk_Cyrl", "__label__asm_Beng", "__label__cat_Latn", "__label__ssw_Latn", "__label__nno_Latn", "__label__apc_Arab", "__label__cym_Latn", "__label__kik_Latn", "__label__ory_Orya", "__label__kmb_Latn", "__label__bel_Cyrl", "__label__uig_Arab", "__label__gug_Latn", "__label__khm_Khmr", "__label__arz_Arab", "__label__ast_Latn", "__label__jav_Latn", "__label__bak_Cyrl", "__label__yue_Hant", "__label__fur_Latn", "__label__bho_Deva", "__label__hne_Deva", "__label__kbp_Latn", "__label__kam_Latn", "__label__kab_Latn", "__label__kaz_Cyrl", "__label__gla_Latn", "__label__snd_Arab", "__label__mri_Latn", "__label__lim_Latn", "__label__mni_Beng", "__label__plt_Latn", "__label__sun_Latn", "__label__san_Deva", "__label__srd_Latn", "__label__isl_Latn", "__label__vec_Latn", "__label__glg_Latn", "__label__scn_Latn", "__label__fao_Latn", "__label__ltz_Latn", "__label__cjk_Latn", "__label__lmo_Latn", "__label__mai_Deva", "__label__szl_Latn", "__label__min_Latn", "__label__prs_Arab", "__label__fon_Latn", "__label__sat_Olck", "__label__lij_Latn", "__label__ary_Arab", "__label__wol_Latn", "__label__dik_Latn", "__label__ars_Arab", "__label__ckb_Arab", "__label__lao_Laoo", "__label__shn_Mymr", "__label__aeb_Arab", "__label__bjn_Latn", "__label__crh_Latn", "__label__dyu_Latn", "__label__ace_Latn", "__label__ban_Latn", "__label__kmr_Latn", "__label__ltg_Latn", "__label__fuv_Latn", "__label__kac_Latn", "__label__azb_Arab", "__label__taq_Latn", "__label__zgh_Tfng", "__label__bam_Latn", "__label__awa_Deva", "__label__bug_Latn", "__label__dzo_Tibt", "__label__kas_Deva", "__label__ace_Arab", "__label__knc_Arab", "__label__mag_Deva", "__label__nus_Latn", "__label__taq_Tfng", "__label__bjn_Arab", "__label__knc_Latn", "__label__kas_Arab", "__label__kea_Latn", "__label__acm_Arab", "__label__bod_Tibt", "__label__sot_Latn", "__label__acq_Arab", "__label__ydd_Hebr"]

openlid-v2.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f85944feb2f699a5c4fe2ae69501c0410317e88c7b56dc50a7e4b3dcb1fdece9
+size 303485738

openlid-v2.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b6c8380d951e83a5779f138b705ff278c4f2020baacdab63b2604b2bf4f42bc
+size 1213938021

openlid_v2_hash.py

@@ -0,0 +1,144 @@
+"""fastText feature hashing, vendored from TigreGotico/glotlid-onnx's
+``glotlid_hash.py`` (https://huggingface.co/TigreGotico/glotlid-onnx).
+
+Kept logically identical on purpose, including the sign-extension of bytes
+>= 0x80 before the FNV-1a XOR: fastText's C++ ``Dictionary::hash`` treats
+each byte as a signed ``int8_t`` before XOR-ing it into the hash, so
+non-ASCII n-grams must be hashed the same way here or every non-Latin-script
+bucket lookup breaks.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Dict, List, Sequence
+
+import numpy as np
+
+EOS = "</s>"
+BOW = "<"
+EOW = ">"
+SEPARATORS = " \t\n\v\f\r"
+
+
+def fnv1a(s: str) -> int:
+    h = 2166136261
+    for b in s.encode("utf-8"):
+        if b >= 0x80:
+            b |= 0xFFFFFF00
+        h = ((h ^ b) * 16777619) & 0xFFFFFFFF
+    return h
+
+
+def tokenize(text: str) -> List[str]:
+    for sep in SEPARATORS[1:]:
+        text = text.replace(sep, " ")
+    return [t for t in text.split(" ") if t] + [EOS]
+
+
+def compute_subwords(word: str, minn: int, maxn: int, bucket: int,
+                      nwords: int) -> List[int]:
+    chars = list(word)
+    out: List[int] = []
+    n = len(chars)
+    for i in range(n):
+        for j in range(i + minn, min(n, i + maxn) + 1):
+            ngram = "".join(chars[i:j])
+            out.append(nwords + fnv1a(ngram) % bucket)
+    return out
+
+
+class FastTextFeaturizer:
+    """Maps raw text to the fastText feature ids the ONNX graph expects."""
+
+    def __init__(self, words: Sequence[str], nwords: int, minn: int = 2,
+                 maxn: int = 5, bucket: int = 1_000_000):
+        self.words: List[str] = list(words)
+        self.nwords = nwords
+        self.minn = minn
+        self.maxn = maxn
+        self.bucket = bucket
+        self.word2id: Dict[str, int] = {w: i for i, w in enumerate(self.words)}
+        self._cache: Dict[int, List[int]] = {}
+
+    @classmethod
+    def from_files(cls, vocab_path: str, meta_path: str) -> "FastTextFeaturizer":
+        with open(meta_path, encoding="utf-8") as fh:
+            meta = json.load(fh)
+        with open(vocab_path, encoding="utf-8") as fh:
+            words = fh.read().split("\n")
+        if words and words[-1] == "":
+            words.pop()
+        return cls(words, meta["nwords"], meta["minn"], meta["maxn"],
+                    meta["bucket"])
+
+    def subwords_of_known(self, wid: int) -> List[int]:
+        cached = self._cache.get(wid)
+        if cached is None:
+            word = self.words[wid]
+            if word == EOS:
+                cached = [wid]
+            else:
+                cached = [wid] + compute_subwords(
+                    BOW + word + EOW, self.minn, self.maxn, self.bucket,
+                    self.nwords)
+            self._cache[wid] = cached
+        return cached
+
+    def add_subwords(self, line: List[int], token: str) -> None:
+        wid = self.word2id.get(token, -1)
+        if wid < 0:
+            if token != EOS:
+                line.extend(compute_subwords(BOW + token + EOW, self.minn,
+                                              self.maxn, self.bucket,
+                                              self.nwords))
+        elif self.maxn <= 0:
+            line.append(wid)
+        else:
+            line.extend(self.subwords_of_known(wid))
+
+    def __call__(self, text: str) -> np.ndarray:
+        line: List[int] = []
+        for token in tokenize(text):
+            self.add_subwords(line, token)
+        return np.asarray(line, dtype=np.int64)
+
+
+class HSCombiner:
+    """Combines the ONNX graph's per-Huffman-node ``node_probs`` into
+    per-label probabilities for hierarchical-softmax fastText models
+    (e.g. fastText's own ``lid.176.bin``).
+
+    fastText's hierarchical softmax does not reduce to a flat softmax over
+    the output matrix: each row of the output matrix is a binary classifier
+    for one internal node of a Huffman tree built over the label
+    frequencies, and a label's probability is the product of the sigmoid
+    (or 1-sigmoid) values along the root-to-leaf path. Tree construction
+    (``hs_tree.build_tree``) is deterministic given the label counts, so it
+    is precomputed once at export time into ``hs_tree.json``; this class
+    just walks the paths, which is cheap pure-Python work analogous to the
+    string hashing above - there is no portable ONNX op for it.
+    """
+
+    def __init__(self, paths: List[List[int]], codes: List[List[bool]]):
+        self.paths = paths
+        self.codes = codes
+
+    @classmethod
+    def from_file(cls, path: str) -> "HSCombiner":
+        with open(path, encoding="utf-8") as fh:
+            data = json.load(fh)
+        return cls(data["paths"], data["codes"])
+
+    def __call__(self, node_probs: np.ndarray) -> np.ndarray:
+        """node_probs: sigmoid(dot(hidden, node)) for every Huffman node.
+        Returns: probability per label, same order as labels.json."""
+        log_f = np.log(np.clip(node_probs, 1e-12, 1.0))
+        log_1mf = np.log(np.clip(1.0 - node_probs, 1e-12, 1.0))
+        out = np.empty(len(self.paths), dtype=np.float64)
+        for i, (path, code) in enumerate(zip(self.paths, self.codes)):
+            s = 0.0
+            for node, bit in zip(path, code):
+                s += log_f[node] if bit else log_1mf[node]
+            out[i] = s
+        return np.exp(out)