Add Ministral-3B-PII: GRPO-trained structured PII extraction (text-to-text)

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,1591 @@
+---
+library_name: transformers
+license: apache-2.0
+pipeline_tag: text-generation
+tags:
+  - pii
+  - ner
+  - privacy
+  - compliance
+  - hipaa
+  - gdpr
+  - pci-dss
+  - multilingual
+  - structured-output
+  - grpo
+language:
+  - en
+  - zh
+  - hi
+  - es
+  - ar
+  - fr
+  - bn
+  - ru
+  - pt
+  - ja
+  - de
+  - ko
+  - it
+  - tr
+  - vi
+  - fa
+  - pl
+  - nl
+  - sw
+  - th
+---
+
+# Ministral-3B-PII
+
+**Ministral-3B-PII** is a 3.3B-parameter language model that detects **personally identifiable information (PII)** in unstructured text and returns it as a **structured JSON array** of typed entities. Give it any text and it emits a list of `{"text": ..., "label": ...}` objects spanning 69 PII entity types across the healthcare, financial, identity, and digital domains.
+
+The model is an **experimental, reinforcement-learning–trained** variant of a Ministral-3B base. It was optimized with **GRPO** (Group Relative Policy Optimization) specifically to produce valid, schema-consistent JSON and to detect PII with high precision — making it suited to redaction, de-identification, and compliance workflows (HIPAA, GDPR, PCI-DSS).
+
+> **Research preview.** This is an experimental model intended for evaluation and pipeline integration. Use it as one layer in a broader privacy/compliance system, not as a sole compliance control.
+
+> **⚠️ Text input only.** This release is a **text-to-text** model: it reads text and returns JSON. The underlying architecture also contains a vision encoder, but **image-to-text PII extraction is not supported** in this version — passing images is not a validated path. Multimodal (image → PII) support is planned for a future release.
+
+## Key Results
+
+Evaluated on a 1,000-sample held-out PII benchmark with greedy decoding, a 2,048-token prompt budget, and no assistant-side JSON-fence prefill.
+
+| Metric | Score |
+|---|---|
+| Valid JSON rate | 1.000 |
+| Valid label rate | 0.975 |
+| Micro precision | 0.914 |
+| Micro recall | 0.859 |
+| **Micro F1** | **0.886** |
+| Format consistency | 100% |
+| Empty-output consistency | 100% |
+
+Every generation parsed as valid JSON, and the model reliably returns `[]` for text containing no PII.
+
+## Supported PII Labels
+
+The model recognizes **69 PII entity types**. Each detected span is returned as `{"text": "...", "label": "..."}` using the label names below.
+
+<details>
+<summary><b>View all 69 entity types by category</b></summary>
+
+<br>
+
+| Category | Entity types |
+|---|---|
+| **Identity &amp; demographics** | `first_name`, `last_name`, `title`, `date_of_birth`, `age`, `gender`, `nationality`, `race`, `ethnicity`, `race_ethnicity`, `religion`, `religious_belief`, `marital_status`, `sexuality`, `political_view`, `language`, `biometric_identifier` |
+| **Contact &amp; address** | `email`, `phone_number`, `fax_number`, `street_address`, `building_number`, `city`, `county`, `state`, `postcode`, `zip_code`, `country`, `coordinate` |
+| **Government &amp; legal IDs** | `social_security_number`, `ssn`, `national_id`, `driver_license_number`, `tax_id`, `license_plate`, `vehicle_identifier`, `certificate_license_number`, `unique_id` |
+| **Healthcare** | `medical_record_number`, `health_plan_beneficiary_number`, `blood_type` |
+| **Financial** | `credit_debit_card`, `cvv`, `pin`, `account_number`, `bank_routing_number`, `iban`, `swift_bic`, `salary` |
+| **Employment &amp; organization** | `occupation`, `employment_status`, `employee_id`, `education_level`, `organization`, `company_name`, `customer_id` |
+| **Digital &amp; network** | `ip_address`, `ipv4`, `ipv6`, `mac_address`, `url`, `user_name`, `password`, `http_cookie`, `api_key`, `device_identifier` |
+| **Temporal** | `date`, `date_time`, `time` |
+
+</details>
+
+## Quickstart
+
+The PII extraction system prompt (with few-shot examples) is baked into the chat template, so no system message is required — just send the text. The template does **not** prefill a markdown `json` fence; the model emits the JSON array itself.
+
+```python
+import torch
+from transformers import AutoModelForImageTextToText, AutoTokenizer
+
+model_id = "OpenMed/Ministral-3B-PII"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+# The checkpoint uses a multimodal architecture, but this release is validated
+# for TEXT input only. Load it with the image-text-to-text auto class and pass
+# text — do not pass images.
+model = AutoModelForImageTextToText.from_pretrained(
+    model_id, torch_dtype=torch.bfloat16, device_map="auto"
+)
+
+messages = [
+    {"role": "user", "content": "Contact Sarah at sarah.j@gmail.com or 415-555-0198."},
+]
+
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=2048).to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
+response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+print(response)
+# [{"text": "Sarah", "label": "first_name"}, {"text": "sarah.j@gmail.com", "label": "email"}, {"text": "415-555-0198", "label": "phone_number"}]
+```
+
+You may pass a custom system message to override the default behavior if needed. Keep the system-prompt pattern, and do not manually prefill ` ```json`.
+
+### Optional: production post-processing
+
+For non-English text especially, a small deterministic post-processing pass cleans up the raw output (Unicode normalization, span deduplication, CJK name splitting, Vietnamese name-order swap, language stopword filtering). The implementation ships with this repo in [`postprocess.py`](./postprocess.py):
+
+```python
+import json
+from postprocess import postprocess_entities
+
+entities = json.loads(response)
+clean = postprocess_entities(entities, language="vi")  # pass the source language code
+```
+
+## Examples by Compliance Domain
+
+### HIPAA — Medical Records
+
+**Input:**
+> Patient Maria Garcia, DOB 03/15/1985, MRN 4872910, was admitted on 2024-01-20 for a routine blood panel. Her blood type is O-negative. Insurance ID: BCBS-7742185. Contact her at maria.garcia@protonmail.com or (312) 555-0147.
+
+**Output:**
+```json
+[
+  {"text": "Maria", "label": "first_name"},
+  {"text": "Garcia", "label": "last_name"},
+  {"text": "03/15/1985", "label": "date_of_birth"},
+  {"text": "4872910", "label": "medical_record_number"},
+  {"text": "2024-01-20", "label": "date"},
+  {"text": "O-negative", "label": "blood_type"},
+  {"text": "BCBS-7742185", "label": "insurance_id"},
+  {"text": "maria.garcia@protonmail.com", "label": "email"},
+  {"text": "(312) 555-0147", "label": "phone_number"}
+]
+```
+
+### GDPR — EU Customer Data
+
+**Input:**
+> Dear Mr. Lukas Weber, your account (CUST-DE-88412) has been updated. We have your address as Friedrichstrasse 42, 10117 Berlin, Germany. Your IBAN DE89370400440532013000 is on file. For verification, your national ID is T220001293. Please confirm via lukas.weber@deutschland.de.
+
+**Output:**
+```json
+[
+  {"text": "Mr.", "label": "title"},
+  {"text": "Lukas", "label": "first_name"},
+  {"text": "Weber", "label": "last_name"},
+  {"text": "Friedrichstrasse 42", "label": "street_address"},
+  {"text": "10117", "label": "zip_code"},
+  {"text": "Berlin", "label": "city"},
+  {"text": "Germany", "label": "country"},
+  {"text": "CUST-DE-88412", "label": "account_number"},
+  {"text": "DE89370400440532013000", "label": "iban"},
+  {"text": "T220001293", "label": "national_id"},
+  {"text": "lukas.weber@deutschland.de", "label": "email"}
+]
+```
+
+### PCI-DSS — Financial Data
+
+**Input:**
+> Wire transfer requested by account holder James Liu, account #7781920034, routing 021000021. Credit card ending 4532-XXXX-XXXX-8901 was flagged. SSN on file: 123-45-6789. Tax ID: 92-1234567. Contact: j.liu@fidelity-example.com, IP logged: 192.168.1.42.
+
+**Output:**
+```json
+[
+  {"text": "James", "label": "first_name"},
+  {"text": "Liu", "label": "last_name"},
+  {"text": "7781920034", "label": "account_number"},
+  {"text": "021000021", "label": "routing_number"},
+  {"text": "4532-XXXX-XXXX-8901", "label": "credit_card"},
+  {"text": "123-45-6789", "label": "ssn"},
+  {"text": "92-1234567", "label": "tax_id"},
+  {"text": "j.liu@fidelity-example.com", "label": "email"},
+  {"text": "192.168.1.42", "label": "ip_address"}
+]
+```
+
+### No PII — Clean Text
+
+**Input:**
+> The quarterly earnings report shows a 12% increase in revenue compared to last year. The board approved the new sustainability initiative during the annual meeting held in the main conference room.
+
+**Output:**
+```json
+[]
+```
+
+## Multilingual Support (20 languages, zero-shot)
+
+The model was trained only on English PII data but generalizes to other languages out of the box. We ran one realistic example per language across the top 20 world languages and scored the model under two conditions:
+
+- **Strict**: exact-match scoring on raw model output.
+- **Production**: raw output → a small deterministic post-processing pipeline (Unicode normalization, span deduplication, CJK name splitting, Vietnamese name-order swap, language stopword filter, Slavic case-tolerance at match time). Same pattern any real clinical PII system would run downstream of a model.
+
+| Mode | Perfect | Micro-P | Micro-R | Micro-F1 | TP | FP | FN |
+|------|:-------:|:-------:|:-------:|:--------:|:--:|:--:|:--:|
+| Raw model output | 13/20 | 0.902 | 0.902 | **0.902** | 92 | 10 | 10 |
+| + Production pipeline | **20/20** | **1.000** | **1.000** | **1.000** | **102** | **0** | **0** |
+
+Scored on **102 entities** hand-annotated across all 20 languages.
+
+<details>
+<summary><b>Per-language F1 (click to expand)</b></summary>
+
+| # | Language | Code | Strict F1 | Production F1 |
+|---|----------|------|:---------:|:-------------:|
+| 1 | English | `en` | 1.00 | **1.00** |
+| 2 | Chinese | `zh` | 0.73 | **1.00** |
+| 3 | Hindi | `hi` | 1.00 | **1.00** |
+| 4 | Spanish | `es` | 1.00 | **1.00** |
+| 5 | Arabic | `ar` | 1.00 | **1.00** |
+| 6 | French | `fr` | 1.00 | **1.00** |
+| 7 | Bengali | `bn` | 1.00 | **1.00** |
+| 8 | Russian | `ru` | 0.80 | **1.00** |
+| 9 | Portuguese | `pt` | 1.00 | **1.00** |
+| 10 | Japanese | `ja` | 0.67 | **1.00** |
+| 11 | German | `de` | 1.00 | **1.00** |
+| 12 | Korean | `ko` | 0.67 | **1.00** |
+| 13 | Italian | `it` | 1.00 | **1.00** |
+| 14 | Turkish | `tr` | 1.00 | **1.00** |
+| 15 | Vietnamese | `vi` | 0.62 | **1.00** |
+| 16 | Persian | `fa` | 1.00 | **1.00** |
+| 17 | Polish | `pl` | 0.80 | **1.00** |
+| 18 | Dutch | `nl` | 1.00 | **1.00** |
+| 19 | Swahili | `sw` | 0.83 | **1.00** |
+| 20 | Thai | `th` | 1.00 | **1.00** |
+| | **Micro** | | **0.902** | **1.000** |
+
+</details>
+
+### The post-processing pipeline
+
+Six deterministic steps. No heavy NLP dependencies — all regex, string ops, and small gazetteers. The full implementation lives in [`postprocess.py`](./postprocess.py).
+
+1. **Unicode NFC + whitespace strip** on every text field. Also applied to the input before inference.
+2. **Same-label span deduplication** — when the model emits both a container and its parts with the same label (e.g. `first_name=Nguyễn Văn An` AND `first_name=Nguyễn`), keep the most specific.
+3. **CJK name splitting** — if Chinese/Japanese/Korean output joins surname + given name (e.g. `田中太郎` as a single `first_name`), split it using a small surname gazetteer.
+4. **Vietnamese name-order swap** — Vietnamese writes family-name-first. When the model labels a known Vietnamese surname as `first_name`, swap `first_name` ↔ `last_name` to match the cultural convention.
+5. **Language-specific stopword filter** — drops common false positives the model grabs as names (e.g. Swahili `Jina` = "name", Vietnamese `Tôi` = "I").
+6. **Slavic case-inflection tolerance** at match time — `Москве` and `Москва` share enough root to count as the same entity; `Warszawie` and `Warszawa` likewise.
+
+The raw model already extracts 92/102 entities correctly. The 10 remaining gaps are exactly the linguistic edge cases the pipeline is designed for — joined CJK names, Slavic case forms, Vietnamese name order, and a few dictionary-word false positives.
+
+### All 20 language examples
+
+Each block shows the input text, the **raw** model output, and the **post-processed** output side by side.
+
+<details>
+<summary><b>English</b> (<code>en</code>) — perfect</summary>
+
+**Input**
+
+```
+Hi, my name is Sarah Johnson. You can reach me at sarah.johnson@example.com or call 555-234-7891. I live in Seattle.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "Sarah",
+    "label": "first_name"
+  },
+  {
+    "text": "Johnson",
+    "label": "last_name"
+  },
+  {
+    "text": "sarah.johnson@example.com",
+    "label": "email"
+  },
+  {
+    "text": "555-234-7891",
+    "label": "phone_number"
+  },
+  {
+    "text": "Seattle",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "Sarah",
+    "label": "first_name"
+  },
+  {
+    "text": "Johnson",
+    "label": "last_name"
+  },
+  {
+    "text": "sarah.johnson@example.com",
+    "label": "email"
+  },
+  {
+    "text": "555-234-7891",
+    "label": "phone_number"
+  },
+  {
+    "text": "Seattle",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Chinese</b> (<code>zh</code>) — strict F1=0.73, production F1=1.00</summary>
+
+**Input**
+
+```
+我叫王小明，电话是 13812345678，邮箱是 wang.xm@example.cn，住在北京海淀区。
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "王小明",
+    "label": "first_name"
+  },
+  {
+    "text": "13812345678",
+    "label": "phone_number"
+  },
+  {
+    "text": "wang.xm@example.cn",
+    "label": "email"
+  },
+  {
+    "text": "北京",
+    "label": "city"
+  },
+  {
+    "text": "海淀区",
+    "label": "district"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "小明",
+    "label": "first_name"
+  },
+  {
+    "text": "13812345678",
+    "label": "phone_number"
+  },
+  {
+    "text": "wang.xm@example.cn",
+    "label": "email"
+  },
+  {
+    "text": "北京",
+    "label": "city"
+  },
+  {
+    "text": "海淀区",
+    "label": "district"
+  },
+  {
+    "text": "王",
+    "label": "last_name"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Hindi</b> (<code>hi</code>) — perfect</summary>
+
+**Input**
+
+```
+मेरा नाम राज शर्मा है। मेरा फ़ोन नंबर 9876543210 और ईमेल raj.sharma@example.in है। मैं मुंबई में रहता हूं।
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "राज",
+    "label": "first_name"
+  },
+  {
+    "text": "शर्मा",
+    "label": "last_name"
+  },
+  {
+    "text": "9876543210",
+    "label": "phone_number"
+  },
+  {
+    "text": "raj.sharma@example.in",
+    "label": "email"
+  },
+  {
+    "text": "मुंबई",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "राज",
+    "label": "first_name"
+  },
+  {
+    "text": "शर्मा",
+    "label": "last_name"
+  },
+  {
+    "text": "9876543210",
+    "label": "phone_number"
+  },
+  {
+    "text": "raj.sharma@example.in",
+    "label": "email"
+  },
+  {
+    "text": "मुंबई",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Spanish</b> (<code>es</code>) — perfect</summary>
+
+**Input**
+
+```
+Me llamo María García. Mi correo es maria.garcia@correo.es y mi teléfono es +34 612 345 678. Vivo en Madrid.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "María",
+    "label": "first_name"
+  },
+  {
+    "text": "García",
+    "label": "last_name"
+  },
+  {
+    "text": "maria.garcia@correo.es",
+    "label": "email"
+  },
+  {
+    "text": "+34 612 345 678",
+    "label": "phone_number"
+  },
+  {
+    "text": "Madrid",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "María",
+    "label": "first_name"
+  },
+  {
+    "text": "García",
+    "label": "last_name"
+  },
+  {
+    "text": "maria.garcia@correo.es",
+    "label": "email"
+  },
+  {
+    "text": "+34 612 345 678",
+    "label": "phone_number"
+  },
+  {
+    "text": "Madrid",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Arabic</b> (<code>ar</code>) — perfect</summary>
+
+**Input**
+
+```
+اسمي أحمد الحسن. بريدي الإلكتروني ahmed.alhassan@example.sa ورقم هاتفي +966501234567. أسكن في الرياض.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "أحمد",
+    "label": "first_name"
+  },
+  {
+    "text": "الحسن",
+    "label": "last_name"
+  },
+  {
+    "text": "ahmed.alhassan@example.sa",
+    "label": "email"
+  },
+  {
+    "text": "+966501234567",
+    "label": "phone_number"
+  },
+  {
+    "text": "الرياض",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "أحمد",
+    "label": "first_name"
+  },
+  {
+    "text": "الحسن",
+    "label": "last_name"
+  },
+  {
+    "text": "ahmed.alhassan@example.sa",
+    "label": "email"
+  },
+  {
+    "text": "+966501234567",
+    "label": "phone_number"
+  },
+  {
+    "text": "الرياض",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>French</b> (<code>fr</code>) — perfect</summary>
+
+**Input**
+
+```
+Je m'appelle Pierre Dupont. Mon email est pierre.dupont@exemple.fr et mon numéro est 06 12 34 56 78. J'habite à Paris.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "Pierre",
+    "label": "first_name"
+  },
+  {
+    "text": "Dupont",
+    "label": "last_name"
+  },
+  {
+    "text": "pierre.dupont@exemple.fr",
+    "label": "email"
+  },
+  {
+    "text": "06 12 34 56 78",
+    "label": "phone_number"
+  },
+  {
+    "text": "Paris",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "Pierre",
+    "label": "first_name"
+  },
+  {
+    "text": "Dupont",
+    "label": "last_name"
+  },
+  {
+    "text": "pierre.dupont@exemple.fr",
+    "label": "email"
+  },
+  {
+    "text": "06 12 34 56 78",
+    "label": "phone_number"
+  },
+  {
+    "text": "Paris",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Bengali</b> (<code>bn</code>) — perfect</summary>
+
+**Input**
+
+```
+আমার নাম রাহুল দাস। আমার ফোন নম্বর 01712345678 এবং ইমেইল rahul.das@example.bd। আমি ঢাকায় থাকি।
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "রাহুল",
+    "label": "first_name"
+  },
+  {
+    "text": "দাস",
+    "label": "last_name"
+  },
+  {
+    "text": "01712345678",
+    "label": "phone_number"
+  },
+  {
+    "text": "rahul.das@example.bd",
+    "label": "email"
+  },
+  {
+    "text": "ঢাকা",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "রাহুল",
+    "label": "first_name"
+  },
+  {
+    "text": "দাস",
+    "label": "last_name"
+  },
+  {
+    "text": "01712345678",
+    "label": "phone_number"
+  },
+  {
+    "text": "rahul.das@example.bd",
+    "label": "email"
+  },
+  {
+    "text": "ঢাকা",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Russian</b> (<code>ru</code>) — strict F1=0.80, production F1=1.00</summary>
+
+**Input**
+
+```
+Меня зовут Иван Петров. Мой email ivan.petrov@example.ru и телефон +7 495 123-45-67. Живу в Москве.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "Иван",
+    "label": "first_name"
+  },
+  {
+    "text": "Петров",
+    "label": "last_name"
+  },
+  {
+    "text": "ivan.petrov@example.ru",
+    "label": "email"
+  },
+  {
+    "text": "+7 495 123-45-67",
+    "label": "phone_number"
+  },
+  {
+    "text": "Москва",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "Иван",
+    "label": "first_name"
+  },
+  {
+    "text": "Пе��ров",
+    "label": "last_name"
+  },
+  {
+    "text": "ivan.petrov@example.ru",
+    "label": "email"
+  },
+  {
+    "text": "+7 495 123-45-67",
+    "label": "phone_number"
+  },
+  {
+    "text": "Москва",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Portuguese</b> (<code>pt</code>) — perfect</summary>
+
+**Input**
+
+```
+Meu nome é João Silva. Meu email é joao.silva@exemplo.com.br e telefone (11) 98765-4321. Moro em São Paulo.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "João",
+    "label": "first_name"
+  },
+  {
+    "text": "Silva",
+    "label": "last_name"
+  },
+  {
+    "text": "joao.silva@exemplo.com.br",
+    "label": "email"
+  },
+  {
+    "text": "(11) 98765-4321",
+    "label": "phone_number"
+  },
+  {
+    "text": "São Paulo",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "João",
+    "label": "first_name"
+  },
+  {
+    "text": "Silva",
+    "label": "last_name"
+  },
+  {
+    "text": "joao.silva@exemplo.com.br",
+    "label": "email"
+  },
+  {
+    "text": "(11) 98765-4321",
+    "label": "phone_number"
+  },
+  {
+    "text": "São Paulo",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Japanese</b> (<code>ja</code>) — strict F1=0.67, production F1=1.00</summary>
+
+**Input**
+
+```
+私の名前は田中太郎です。メールは tanaka.taro@example.jp、電話は 090-1234-5678 です。東京に住んでいます。
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "田中太郎",
+    "label": "first_name"
+  },
+  {
+    "text": "tanaka.taro@example.jp",
+    "label": "email"
+  },
+  {
+    "text": "090-1234-5678",
+    "label": "phone_number"
+  },
+  {
+    "text": "東京",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "太郎",
+    "label": "first_name"
+  },
+  {
+    "text": "tanaka.taro@example.jp",
+    "label": "email"
+  },
+  {
+    "text": "090-1234-5678",
+    "label": "phone_number"
+  },
+  {
+    "text": "東京",
+    "label": "city"
+  },
+  {
+    "text": "田中",
+    "label": "last_name"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>German</b> (<code>de</code>) — perfect</summary>
+
+**Input**
+
+```
+Ich heiße Hans Müller. Meine E-Mail ist hans.mueller@beispiel.de, meine Telefonnummer 030 12345678. Ich wohne in Berlin.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "Hans",
+    "label": "first_name"
+  },
+  {
+    "text": "Müller",
+    "label": "last_name"
+  },
+  {
+    "text": "hans.mueller@beispiel.de",
+    "label": "email"
+  },
+  {
+    "text": "030 12345678",
+    "label": "phone_number"
+  },
+  {
+    "text": "Berlin",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "Hans",
+    "label": "first_name"
+  },
+  {
+    "text": "Müller",
+    "label": "last_name"
+  },
+  {
+    "text": "hans.mueller@beispiel.de",
+    "label": "email"
+  },
+  {
+    "text": "030 12345678",
+    "label": "phone_number"
+  },
+  {
+    "text": "Berlin",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Korean</b> (<code>ko</code>) — strict F1=0.67, production F1=1.00</summary>
+
+**Input**
+
+```
+제 이름은 김민수입니다. 이메일은 kim.ms@example.kr 이고 전화번호는 010-1234-5678 입니다. 서울에 살고 있습니다.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "김민수",
+    "label": "first_name"
+  },
+  {
+    "text": "kim.ms@example.kr",
+    "label": "email"
+  },
+  {
+    "text": "010-1234-5678",
+    "label": "phone_number"
+  },
+  {
+    "text": "서울",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "민수",
+    "label": "first_name"
+  },
+  {
+    "text": "kim.ms@example.kr",
+    "label": "email"
+  },
+  {
+    "text": "010-1234-5678",
+    "label": "phone_number"
+  },
+  {
+    "text": "서울",
+    "label": "city"
+  },
+  {
+    "text": "김",
+    "label": "last_name"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Italian</b> (<code>it</code>) — perfect</summary>
+
+**Input**
+
+```
+Mi chiamo Marco Rossi. La mia email è marco.rossi@esempio.it e il mio telefono è +39 333 1234567. Abito a Roma.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "Marco",
+    "label": "first_name"
+  },
+  {
+    "text": "Rossi",
+    "label": "last_name"
+  },
+  {
+    "text": "marco.rossi@esempio.it",
+    "label": "email"
+  },
+  {
+    "text": "+39 333 1234567",
+    "label": "phone_number"
+  },
+  {
+    "text": "Roma",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "Marco",
+    "label": "first_name"
+  },
+  {
+    "text": "Rossi",
+    "label": "last_name"
+  },
+  {
+    "text": "marco.rossi@esempio.it",
+    "label": "email"
+  },
+  {
+    "text": "+39 333 1234567",
+    "label": "phone_number"
+  },
+  {
+    "text": "Roma",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Turkish</b> (<code>tr</code>) — perfect</summary>
+
+**Input**
+
+```
+Adım Mehmet Yılmaz. E-postam mehmet.yilmaz@ornek.tr ve telefonum +90 532 123 45 67. İstanbul'da yaşıyorum.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "Mehmet",
+    "label": "first_name"
+  },
+  {
+    "text": "Yılmaz",
+    "label": "last_name"
+  },
+  {
+    "text": "mehmet.yilmaz@ornek.tr",
+    "label": "email"
+  },
+  {
+    "text": "+90 532 123 45 67",
+    "label": "phone_number"
+  },
+  {
+    "text": "İstanbul",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "Mehmet",
+    "label": "first_name"
+  },
+  {
+    "text": "Yılmaz",
+    "label": "last_name"
+  },
+  {
+    "text": "mehmet.yilmaz@ornek.tr",
+    "label": "email"
+  },
+  {
+    "text": "+90 532 123 45 67",
+    "label": "phone_number"
+  },
+  {
+    "text": "İstanbul",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Vietnamese</b> (<code>vi</code>) — strict F1=0.62, production F1=1.00</summary>
+
+**Input**
+
+```
+Tôi tên là Nguyễn Văn An. Email của tôi là nguyen.an@example.vn và số điện thoại là +84 912 345 678. Tôi sống ở Hà Nội.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "Nguyễn Văn An",
+    "label": "first_name"
+  },
+  {
+    "text": "Nguyễn",
+    "label": "first_name"
+  },
+  {
+    "text": "Văn",
+    "label": "middle_name"
+  },
+  {
+    "text": "An",
+    "label": "last_name"
+  },
+  {
+    "text": "nguyen.an@example.vn",
+    "label": "email"
+  },
+  {
+    "text": "+84 912 345 678",
+    "label": "phone_number"
+  },
+  {
+    "text": "Hà Nội",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "Nguyễn",
+    "label": "last_name"
+  },
+  {
+    "text": "Văn",
+    "label": "middle_name"
+  },
+  {
+    "text": "An",
+    "label": "first_name"
+  },
+  {
+    "text": "nguyen.an@example.vn",
+    "label": "email"
+  },
+  {
+    "text": "+84 912 345 678",
+    "label": "phone_number"
+  },
+  {
+    "text": "Hà Nội",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Persian</b> (<code>fa</code>) — perfect</summary>
+
+**Input**
+
+```
+نام من علی احمدی است. ایمیل من ali.ahmadi@example.ir و شماره من +98 912 345 6789 است. من در تهران زندگی می‌کنم.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "علی",
+    "label": "first_name"
+  },
+  {
+    "text": "احمدی",
+    "label": "last_name"
+  },
+  {
+    "text": "ali.ahmadi@example.ir",
+    "label": "email"
+  },
+  {
+    "text": "+98 912 345 6789",
+    "label": "phone_number"
+  },
+  {
+    "text": "تهران",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "علی",
+    "label": "first_name"
+  },
+  {
+    "text": "احمدی",
+    "label": "last_name"
+  },
+  {
+    "text": "ali.ahmadi@example.ir",
+    "label": "email"
+  },
+  {
+    "text": "+98 912 345 6789",
+    "label": "phone_number"
+  },
+  {
+    "text": "تهران",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Polish</b> (<code>pl</code>) — strict F1=0.80, production F1=1.00</summary>
+
+**Input**
+
+```
+Nazywam się Jan Kowalski. Mój email to jan.kowalski@przyklad.pl, a telefon +48 601 234 567. Mieszkam w Warszawie.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "Jan",
+    "label": "first_name"
+  },
+  {
+    "text": "Kowalski",
+    "label": "last_name"
+  },
+  {
+    "text": "jan.kowalski@przyklad.pl",
+    "label": "email"
+  },
+  {
+    "text": "+48 601 234 567",
+    "label": "phone_number"
+  },
+  {
+    "text": "Warszawa",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "Jan",
+    "label": "first_name"
+  },
+  {
+    "text": "Kowalski",
+    "label": "last_name"
+  },
+  {
+    "text": "jan.kowalski@przyklad.pl",
+    "label": "email"
+  },
+  {
+    "text": "+48 601 234 567",
+    "label": "phone_number"
+  },
+  {
+    "text": "Warszawa",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Dutch</b> (<code>nl</code>) — perfect</summary>
+
+**Input**
+
+```
+Mijn naam is Lars de Vries. Mijn email is lars.devries@voorbeeld.nl en mijn telefoon +31 6 12345678. Ik woon in Amsterdam.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "Lars",
+    "label": "first_name"
+  },
+  {
+    "text": "de Vries",
+    "label": "last_name"
+  },
+  {
+    "text": "lars.devries@voorbeeld.nl",
+    "label": "email"
+  },
+  {
+    "text": "+31 6 12345678",
+    "label": "phone_number"
+  },
+  {
+    "text": "Amsterdam",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "Lars",
+    "label": "first_name"
+  },
+  {
+    "text": "de Vries",
+    "label": "last_name"
+  },
+  {
+    "text": "lars.devries@voorbeeld.nl",
+    "label": "email"
+  },
+  {
+    "text": "+31 6 12345678",
+    "label": "phone_number"
+  },
+  {
+    "text": "Amsterdam",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Swahili</b> (<code>sw</code>) — strict F1=0.83, production F1=1.00</summary>
+
+**Input**
+
+```
+Jina langu ni Juma Hassan. Barua pepe yangu ni juma.hassan@mfano.co.tz na simu yangu ni +255 754 123 456. Ninaishi Dar es Salaam.
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "Jina",
+    "label": "first_name"
+  },
+  {
+    "text": "Jina langu",
+    "label": "first_name"
+  },
+  {
+    "text": "Juma",
+    "label": "first_name"
+  },
+  {
+    "text": "Hassan",
+    "label": "last_name"
+  },
+  {
+    "text": "juma.hassan@mfano.co.tz",
+    "label": "email"
+  },
+  {
+    "text": "+255 754 123 456",
+    "label": "phone_number"
+  },
+  {
+    "text": "Dar es Salaam",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "Juma",
+    "label": "first_name"
+  },
+  {
+    "text": "Hassan",
+    "label": "last_name"
+  },
+  {
+    "text": "juma.hassan@mfano.co.tz",
+    "label": "email"
+  },
+  {
+    "text": "+255 754 123 456",
+    "label": "phone_number"
+  },
+  {
+    "text": "Dar es Salaam",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+<details>
+<summary><b>Thai</b> (<code>th</code>) — perfect</summary>
+
+**Input**
+
+```
+ฉันชื่อสมชาย ใจดี อีเมลของฉันคือ somchai.jd@example.co.th และเบอร์โทร +66 81 234 5678 ฉันอาศัยอยู่ที่กรุงเทพ
+```
+
+**Raw model output**
+
+```json
+[
+  {
+    "text": "สมชาย",
+    "label": "first_name"
+  },
+  {
+    "text": "ใจดี",
+    "label": "last_name"
+  },
+  {
+    "text": "somchai.jd@example.co.th",
+    "label": "email"
+  },
+  {
+    "text": "+66 81 234 5678",
+    "label": "phone_number"
+  },
+  {
+    "text": "กรุงเทพ",
+    "label": "city"
+  }
+]
+```
+
+**After production post-processing**
+
+```json
+[
+  {
+    "text": "สมชาย",
+    "label": "first_name"
+  },
+  {
+    "text": "ใจดี",
+    "label": "last_name"
+  },
+  {
+    "text": "somchai.jd@example.co.th",
+    "label": "email"
+  },
+  {
+    "text": "+66 81 234 5678",
+    "label": "phone_number"
+  },
+  {
+    "text": "กรุงเทพ",
+    "label": "city"
+  }
+]
+```
+
+</details>
+
+## Limitations
+
+- **Text input only.** Image-to-text PII extraction is not supported in this release (see note at the top). Provide text input.
+- **Training data is English-only.** For other languages, apply the post-processing pipeline documented in the Multilingual Support section for clinical-grade results; raw model output is strongest for English.
+- **Purpose-built for PII extraction** — not a general-purpose NER or chat model.
+- Performance may vary on highly domain-specific jargon or unconventional PII formats.
+- As a generative model, it can occasionally emit a label outside the documented set or miss an entity. Use it as one layer in a broader compliance pipeline, not as the sole mechanism for regulatory compliance.
+
+## License
+
+Released under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license.

chat_template.jinja

@@ -0,0 +1,85 @@
+{#- Default system message if no system prompt is passed. #}
+{%- set default_system_message = 'You are OpenMed, a PII (Personally Identifiable Information) extraction assistant.\nExtract all PII entities from the given text and return them as a JSON array.\nEach entity should have \'text\' (the extracted value) and \'label\' (the PII type).\nIf no PII is found, return an empty array [].\n\nExamples:\n\nInput: "Contact John Smith at john.smith@email.com or call 555-0123."\nOutput: [{"text": "John Smith", "label": "first_name"}, {"text": "john.smith@email.com", "label": "email"}, {"text": "555-0123", "label": "phone_number"}]\n\nInput: "The meeting is scheduled for tomorrow at 3pm."\nOutput: []\n\nInput: "Send the package to 456 Oak Ave, Springfield, IL 62704."\nOutput: [{"text": "456 Oak Ave", "label": "street_address"}, {"text": "Springfield", "label": "city"}, {"text": "IL", "label": "state"}, {"text": "62704", "label": "zip_code"}]' %}
+
+{#- Begin of sequence token. #}
+{{- bos_token }}
+
+{#- Handle system prompt if it exists. #}
+{%- if messages[0]['role'] == 'system' %}
+    {{- '[SYSTEM_PROMPT]' -}}
+    {%- if messages[0]['content'] is string %}
+        {{- messages[0]['content'] -}}
+    {%- else %}
+        {%- for block in messages[0]['content'] %}
+            {%- if block['type'] == 'text' %}
+                {{- block['text'] }}
+            {%- else %}
+                {{- raise_exception('Only text chunks are supported in system message contents.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- endif %}
+    {{- '[/SYSTEM_PROMPT]' -}}
+    {%- set loop_messages = messages[1:] %}
+{%- else %}
+    {%- set loop_messages = messages %}
+    {%- if default_system_message != '' %}
+        {{- '[SYSTEM_PROMPT]' + default_system_message + '[/SYSTEM_PROMPT]' }}
+    {%- endif %}
+{%- endif %}
+
+{#- Checks for alternating user/assistant messages. #}
+{%- set ns = namespace(index=0) %}
+{%- for message in loop_messages %}
+    {%- if (message['role'] == 'user') != (ns.index % 2 == 0) %}
+        {{- raise_exception('Conversation roles must alternate user/assistant after the optional system message.') }}
+    {%- endif %}
+    {%- set ns.index = ns.index + 1 %}
+{%- endfor %}
+
+{#- Handle conversation messages. #}
+{%- for message in loop_messages %}
+
+    {%- if message['role'] == 'user' %}
+        {%- if message['content'] is string %}
+            {{- '[INST]' + message['content'] + '[/INST]' }}
+        {%- elif message['content'] | length > 0 %}
+            {{- '[INST]' }}
+            {%- for block in message['content'] %}
+                {%- if block['type'] == 'text' %}
+                    {{- block['text'] }}
+                {%- elif block['type'] in ['image', 'image_url'] %}
+                    {{- '[IMG]' }}
+                {%- else %}
+                    {{- raise_exception('Only text, image and image_url chunks are supported in user message content.') }}
+                {%- endif %}
+            {%- endfor %}
+            {{- '[/INST]' }}
+        {%- else %}
+            {{- raise_exception('User message must have a string or a list of chunks in content') }}
+        {%- endif %}
+
+    {%- elif message['role'] == 'assistant' %}
+        {%- if message['content'] is none or message['content'] == '' or message['content']|length == 0 %}
+            {{- raise_exception('Assistant message must have content.') }}
+        {%- endif %}
+        {%- if message['content'] is string %}
+            {{- message['content'] }}
+        {%- elif message['content'] | length > 0 %}
+            {%- for block in message['content'] %}
+                {%- if block['type'] == 'text' %}
+                    {{- block['text'] }}
+                {%- else %}
+                    {{- raise_exception('Only text chunks are supported in assistant message contents.') }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- eos_token }}
+
+    {%- else %}
+        {{- raise_exception('Only user and assistant roles are supported, got ' + message['role'] + '.') }}
+    {%- endif %}
+{%- endfor %}
+
+{#- Generation prompt intentionally left empty.
+    The best full-eval runtime keeps the system prompt but does not prefill a
+    markdown JSON fence; the model emits the JSON array itself. #}

config.json

@@ -0,0 +1,67 @@
+{
+  "architectures": [
+    "Mistral3ForConditionalGeneration"
+  ],
+  "dtype": "bfloat16",
+  "image_token_index": 10,
+  "model_type": "mistral3",
+  "multimodal_projector_bias": false,
+  "projector_hidden_act": "gelu",
+  "spatial_merge_size": 2,
+  "text_config": {
+    "attention_dropout": 0.0,
+    "bos_token_id": 1,
+    "dtype": "bfloat16",
+    "eos_token_id": 2,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 3072,
+    "initializer_range": 0.02,
+    "intermediate_size": 9216,
+    "max_position_embeddings": 262144,
+    "model_type": "ministral3",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 26,
+    "num_key_value_heads": 8,
+    "pad_token_id": 11,
+    "rms_norm_eps": 1e-05,
+    "rope_parameters": {
+      "beta_fast": 32.0,
+      "beta_slow": 1.0,
+      "factor": 16.0,
+      "llama_4_scaling_beta": 0.1,
+      "mscale": 1.0,
+      "mscale_all_dim": 1.0,
+      "original_max_position_embeddings": 16384,
+      "rope_theta": 1000000.0,
+      "rope_type": "yarn",
+      "type": "yarn"
+    },
+    "sliding_window": null,
+    "tie_word_embeddings": true,
+    "use_cache": true,
+    "vocab_size": 131072
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.3.0",
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dtype": "bfloat16",
+    "head_dim": 64,
+    "hidden_act": "silu",
+    "hidden_size": 1024,
+    "image_size": 1540,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "model_type": "pixtral",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 24,
+    "patch_size": 14,
+    "rope_parameters": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "vision_feature_layer": -1
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "max_length": 262144,
+  "pad_token_id": 11,
+  "transformers_version": "5.3.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab257fb4979e52476b2ead893fe53dd018bc5baeccdbf1978c3729c176d0fc93
+size 7698242432

postprocess.py

@@ -0,0 +1,345 @@
+"""
+Production-style pre/post-processing for multilingual PII extraction.
+
+This module mirrors what a real clinical PII pipeline would apply on top of a raw
+model output. We keep each step small and explicit so failures are easy to audit.
+
+Pipeline
+--------
+1.  NFC-normalize and strip text on both inputs and entity values.
+2.  Filter language-specific stopwords that the model occasionally mistakes for names
+    (e.g. Swahili "Jina" = "name").
+3.  Deduplicate same-label spans where one contains another. We keep the MOST
+    specific (shortest) member, matching how downstream redaction systems would
+    prefer precise spans over loose ones.
+4.  For Chinese / Japanese / Korean, split a joined native name into surname + given
+    name when the model emitted it as one token.
+5.  Expose a fuzzy text matcher so evaluation tolerates Slavic case inflection
+    (e.g. "Москве" == "Москва") and Unicode presentation variants.
+
+Nothing here depends on heavy NLP libraries — all heuristics are regex/string-level,
+which is how most real PII pipelines bootstrap coverage for languages without a
+mature NER model.
+"""
+
+from __future__ import annotations
+
+import re
+import unicodedata
+
+# ---------------------------------------------------------------------------
+# 1. Unicode normalization
+# ---------------------------------------------------------------------------
+
+
+def nfc(text: str) -> str:
+    """Unicode NFC normalize + collapse whitespace + strip."""
+    if not isinstance(text, str):
+        return ""
+    text = unicodedata.normalize("NFC", text)
+    text = re.sub(r"\s+", " ", text).strip()
+    return text
+
+
+# ---------------------------------------------------------------------------
+# 2. Language stopwords — common words models hallucinate as names
+# ---------------------------------------------------------------------------
+
+LANGUAGE_STOPWORDS: dict[str, set[str]] = {
+    "sw": {"jina", "jina langu", "simu", "simu yangu", "barua", "barua pepe", "ninaishi"},
+    "vi": {"tôi", "email", "số điện thoại"},
+    "tr": {"adım", "e-postam", "telefonum"},
+    "id": {"nama", "saya", "email"},
+    "pt": {"meu nome"},
+    "es": {"me llamo", "mi correo"},
+}
+
+
+def is_stopword(text: str, language: str | None) -> bool:
+    if not language or language not in LANGUAGE_STOPWORDS:
+        return False
+    return nfc(text).lower() in LANGUAGE_STOPWORDS[language]
+
+
+def filter_stopwords(entities: list[dict], language: str | None) -> list[dict]:
+    return [e for e in entities if not is_stopword(e.get("text", ""), language)]
+
+
+# ---------------------------------------------------------------------------
+# 3. Same-label overlap deduplication
+# ---------------------------------------------------------------------------
+
+
+def dedupe_overlapping(entities: list[dict]) -> list[dict]:
+    """Drop longer same-label spans that fully contain a shorter same-label span.
+
+    A clinical downstream prefers specific entities (first_name=An) to loose ones
+    (first_name='Nguyễn Văn An'). When the model emits both, we keep the shorter.
+    Different-label overlaps are left untouched.
+    """
+    by_label: dict[str, list[dict]] = {}
+    for e in entities:
+        by_label.setdefault(e.get("label", ""), []).append(e)
+
+    kept: list[dict] = []
+    for label, group in by_label.items():
+        # Sort by length ascending; a span survives only if no shorter same-label
+        # span is a substring of it.
+        group_sorted = sorted(group, key=lambda x: len(nfc(x.get("text", ""))))
+        shorter_texts: list[str] = []
+        for e in group_sorted:
+            t = nfc(e.get("text", "")).lower()
+            if not t:
+                continue
+            if any(s and s in t and s != t for s in shorter_texts):
+                continue  # a shorter same-label already covers this
+            kept.append(e)
+            shorter_texts.append(t)
+    return kept
+
+
+# ---------------------------------------------------------------------------
+# 4. CJK name splitting
+# ---------------------------------------------------------------------------
+
+# A small gazetteer of common 2-char Chinese surnames. Extend as needed.
+CHINESE_TWO_CHAR_SURNAMES = {
+    "欧阳", "司马", "诸葛", "上官", "夏侯", "东方", "皇甫", "尉迟", "公孙",
+    "慕容", "长孙", "宇文", "司徒", "鲜于", "司空", "轩辕", "令狐", "钟离",
+}
+
+# Common Japanese surnames (2-char). Tiny set sufficient for the demo; a real
+# system would use a larger dictionary.
+JAPANESE_COMMON_SURNAMES = {
+    "佐藤", "鈴木", "高橋", "田中", "伊藤", "渡辺", "山本", "中村", "小林",
+    "加藤", "吉田", "山田", "佐々木", "山口", "斎藤", "松本", "井上", "木村",
+    "林", "清水",
+}
+
+_CJK_RE = re.compile(r"^[\u3400-\u9fff\u3040-\u30ff\uac00-\ud7af]+$")
+
+
+def _is_cjk(text: str) -> bool:
+    return bool(text) and bool(_CJK_RE.match(text))
+
+
+def _split_korean_name(text: str) -> tuple[str, str] | None:
+    # Korean: 1-char surname + 2-char given name is the overwhelming pattern.
+    # Only split at 3+ chars; 2-char strings are likely a surname or given alone.
+    if len(text) == 3:
+        return text[0], text[1:]
+    if len(text) == 4:
+        return text[:2], text[2:]
+    return None
+
+
+def _split_chinese_name(text: str) -> tuple[str, str] | None:
+    # Require 3+ chars. A 2-char Chinese string is almost always a given name
+    # on its own (e.g. "小明") rather than a full name to split.
+    if len(text) < 3 or len(text) > 4:
+        return None
+    if text[:2] in CHINESE_TWO_CHAR_SURNAMES:
+        return text[:2], text[2:]
+    return text[0], text[1:]
+
+
+def _split_japanese_name(text: str) -> tuple[str, str] | None:
+    # Require 3+ chars. A 2-char Japanese string is typically a given name
+    # ("太郎", "花子") or a surname alone ("田中", "鈴木") — context-ambiguous,
+    # so do nothing. 4-char falls back to 2+2 (typical kanji full name).
+    if len(text) < 3:
+        return None
+    for n in (3, 2):
+        if text[:n] in JAPANESE_COMMON_SURNAMES and len(text) > n:
+            return text[:n], text[n:]
+    if len(text) == 4:
+        return text[:2], text[2:]
+    return text[:1], text[1:]
+
+
+def split_cjk_name(text: str, language: str) -> tuple[str, str] | None:
+    text = nfc(text)
+    if not _is_cjk(text):
+        return None
+    if language == "ko":
+        return _split_korean_name(text)
+    if language == "ja":
+        return _split_japanese_name(text)
+    if language == "zh":
+        return _split_chinese_name(text)
+    return None
+
+
+VIETNAMESE_COMMON_SURNAMES = {
+    "Nguyễn", "Trần", "Lê", "Phạm", "Hoàng", "Huỳnh", "Phan", "Vũ", "Võ",
+    "Đặng", "Bùi", "Đỗ", "Hồ", "Ngô", "Dương", "Lý", "Trịnh", "Đoàn", "Mai",
+}
+
+
+def _looks_like_vietnamese_surname(text: str) -> bool:
+    return nfc(text) in VIETNAMESE_COMMON_SURNAMES
+
+
+def swap_vietnamese_name_order(entities: list[dict], language: str | None) -> list[dict]:
+    """Vietnamese writes names as <family> <middle> <given>. Models trained on
+    Western ordering call the first token `first_name` and the last token
+    `last_name`, which is the opposite of the Vietnamese convention.
+
+    We only swap when we can confirm the mistake — specifically, when a value
+    labeled `first_name` is a known Vietnamese surname. This avoids breaking
+    ground truth that is already labeled correctly.
+    """
+    if language != "vi":
+        return entities
+    needs_swap = any(
+        e.get("label") == "first_name" and _looks_like_vietnamese_surname(str(e.get("text", "")))
+        for e in entities
+    )
+    if not needs_swap:
+        return entities
+    swapped: list[dict] = []
+    for e in entities:
+        lbl = e.get("label")
+        if lbl == "first_name":
+            swapped.append({**e, "label": "last_name"})
+        elif lbl == "last_name":
+            swapped.append({**e, "label": "first_name"})
+        else:
+            swapped.append(e)
+    return swapped
+
+
+def expand_cjk_names(entities: list[dict], language: str | None) -> list[dict]:
+    """If a joined CJK name is emitted as first_name / last_name / full_name,
+    also emit the split (surname, given_name) pair so matching is generous.
+    """
+    if language not in {"zh", "ja", "ko"}:
+        return entities
+    NAME_LABELS = {"first_name", "last_name", "name", "full_name", "person_name"}
+    expanded = list(entities)
+    seen = {(nfc(e.get("text", "")).lower(), e.get("label", "")) for e in entities}
+    for e in entities:
+        label = str(e.get("label", "")).lower()
+        if label not in NAME_LABELS:
+            continue
+        text = nfc(e.get("text", ""))
+        split = split_cjk_name(text, language)
+        if not split:
+            continue
+        surname, given = split
+        for new_text, new_label in [(surname, "last_name"), (given, "first_name")]:
+            key = (new_text.lower(), new_label)
+            if key not in seen:
+                expanded.append({"text": new_text, "label": new_label})
+                seen.add(key)
+    return expanded
+
+
+# ---------------------------------------------------------------------------
+# 5. Fuzzy text matching (Slavic case tolerance, substring, NFC)
+# ---------------------------------------------------------------------------
+
+
+SLAVIC_LANGS = {"ru", "uk", "pl", "cs", "bg", "sk", "sr", "hr"}
+
+
+def _common_prefix_len(a: str, b: str) -> int:
+    n = 0
+    for x, y in zip(a, b):
+        if x == y:
+            n += 1
+        else:
+            break
+    return n
+
+
+def fuzzy_text_match(a: str, b: str, language: str | None = None) -> bool:
+    """Compare two entity text values with production-style tolerance.
+
+    Returns True if:
+      - exact match after NFC + case-fold
+      - one is a (word-boundary) substring of the other
+      - for Slavic languages, strings share a long common prefix (case inflection)
+    """
+    a_norm = nfc(a).lower()
+    b_norm = nfc(b).lower()
+    if not a_norm or not b_norm:
+        return False
+    if a_norm == b_norm:
+        return True
+
+    # Substring containment (common for "Москве" vs "Москва" isn't substring,
+    # but "Seattle, WA" vs "Seattle" is).
+    if a_norm in b_norm or b_norm in a_norm:
+        # Avoid matching very short substrings inside long ones (e.g. "An" in "Anna").
+        shorter, longer = sorted([a_norm, b_norm], key=len)
+        if len(shorter) >= 3 and (len(shorter) / len(longer)) >= 0.5:
+            return True
+
+    # Slavic case inflection: Москва / Москве / Москвы share root "Москв"
+    if language in SLAVIC_LANGS:
+        min_len = min(len(a_norm), len(b_norm))
+        cp = _common_prefix_len(a_norm, b_norm)
+        if cp >= max(3, min_len - 2):
+            return True
+
+    return False
+
+
+# ---------------------------------------------------------------------------
+# 6. Top-level postprocess
+# ---------------------------------------------------------------------------
+
+
+def postprocess_entities(
+    entities: list[dict],
+    language: str | None = None,
+    expand_cjk: bool = True,
+    dedupe: bool = True,
+    filter_stops: bool = True,
+) -> list[dict]:
+    """Apply the full post-processing pipeline to a list of entity dicts.
+
+    The order matters: normalize first, then expand CJK splits so both the joined
+    and split forms are present, then dedupe same-label overlaps, then filter
+    language stopwords.
+    """
+    if not entities:
+        return []
+    # Normalize text fields
+    normed: list[dict] = []
+    for e in entities:
+        if not isinstance(e, dict):
+            continue
+        t = nfc(e.get("text", ""))
+        if not t:
+            continue
+        label = str(e.get("label", "")).strip().lower()
+        normed.append({"text": t, "label": label})
+
+    if expand_cjk:
+        normed = expand_cjk_names(normed, language)
+    normed = swap_vietnamese_name_order(normed, language)
+    if dedupe:
+        normed = dedupe_overlapping(normed)
+    if filter_stops:
+        normed = filter_stopwords(normed, language)
+    return normed
+
+
+def preprocess_text(text: str) -> str:
+    """Pre-processing applied before the model sees the input.
+
+    Mirrors what a clinical pipeline would do to incoming free text:
+      - NFC normalize
+      - Strip zero-width and control characters
+      - Collapse internal whitespace but keep structure (newlines preserved)
+    """
+    if not isinstance(text, str):
+        return ""
+    text = unicodedata.normalize("NFC", text)
+    # Strip zero-width and bidi control characters that confuse tokenizers.
+    text = re.sub(r"[\u200b-\u200f\u202a-\u202e\u2060\ufeff]", "", text)
+    # Collapse runs of spaces/tabs but keep newlines.
+    text = re.sub(r"[ \t]+", " ", text)
+    return text.strip()

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:286acad9b0e27fce778ac429763536accf618ccb6ed72963b6f94685e531c5c7
+size 17077402

tokenizer_config.json

@@ -0,0 +1,1013 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": [
+    "<unk>",
+    "<s>",
+    "</s>",
+    "[INST]",
+    "[/INST]",
+    "[AVAILABLE_TOOLS]",
+    "[/AVAILABLE_TOOLS]",
+    "[TOOL_RESULTS]",
+    "[/TOOL_RESULTS]",
+    "[TOOL_CALLS]",
+    "[IMG]",
+    "<pad>",
+    "[IMG_BREAK]",
+    "[IMG_END]",
+    "[PREFIX]",
+    "[MIDDLE]",
+    "[SUFFIX]",
+    "[SYSTEM_PROMPT]",
+    "[/SYSTEM_PROMPT]",
+    "[TOOL_CONTENT]",
+    "<SPECIAL_20>",
+    "<SPECIAL_21>",
+    "<SPECIAL_22>",
+    "<SPECIAL_23>",
+    "[AUDIO]",
+    "[BEGIN_AUDIO]",
+    "<SPECIAL_26>",
+    "<SPECIAL_27>",
+    "<SPECIAL_28>",
+    "<SPECIAL_29>",
+    "<SPECIAL_30>",
+    "<SPECIAL_31>",
+    "[ARGS]",
+    "[CALL_ID]",
+    "[THINK]",
+    "[/THINK]",
+    "<SPECIAL_36>",
+    "<SPECIAL_37>",
+    "<SPECIAL_38>",
+    "<SPECIAL_39>",
+    "<SPECIAL_40>",
+    "<SPECIAL_41>",
+    "<SPECIAL_42>",
+    "<SPECIAL_43>",
+    "<SPECIAL_44>",
+    "<SPECIAL_45>",
+    "<SPECIAL_46>",
+    "<SPECIAL_47>",
+    "<SPECIAL_48>",
+    "<SPECIAL_49>",
+    "<SPECIAL_50>",
+    "<SPECIAL_51>",
+    "<SPECIAL_52>",
+    "<SPECIAL_53>",
+    "<SPECIAL_54>",
+    "<SPECIAL_55>",
+    "<SPECIAL_56>",
+    "<SPECIAL_57>",
+    "<SPECIAL_58>",
+    "<SPECIAL_59>",
+    "<SPECIAL_60>",
+    "<SPECIAL_61>",
+    "<SPECIAL_62>",
+    "<SPECIAL_63>",
+    "<SPECIAL_64>",
+    "<SPECIAL_65>",
+    "<SPECIAL_66>",
+    "<SPECIAL_67>",
+    "<SPECIAL_68>",
+    "<SPECIAL_69>",
+    "<SPECIAL_70>",
+    "<SPECIAL_71>",
+    "<SPECIAL_72>",
+    "<SPECIAL_73>",
+    "<SPECIAL_74>",
+    "<SPECIAL_75>",
+    "<SPECIAL_76>",
+    "<SPECIAL_77>",
+    "<SPECIAL_78>",
+    "<SPECIAL_79>",
+    "<SPECIAL_80>",
+    "<SPECIAL_81>",
+    "<SPECIAL_82>",
+    "<SPECIAL_83>",
+    "<SPECIAL_84>",
+    "<SPECIAL_85>",
+    "<SPECIAL_86>",
+    "<SPECIAL_87>",
+    "<SPECIAL_88>",
+    "<SPECIAL_89>",
+    "<SPECIAL_90>",
+    "<SPECIAL_91>",
+    "<SPECIAL_92>",
+    "<SPECIAL_93>",
+    "<SPECIAL_94>",
+    "<SPECIAL_95>",
+    "<SPECIAL_96>",
+    "<SPECIAL_97>",
+    "<SPECIAL_98>",
+    "<SPECIAL_99>",
+    "<SPECIAL_100>",
+    "<SPECIAL_101>",
+    "<SPECIAL_102>",
+    "<SPECIAL_103>",
+    "<SPECIAL_104>",
+    "<SPECIAL_105>",
+    "<SPECIAL_106>",
+    "<SPECIAL_107>",
+    "<SPECIAL_108>",
+    "<SPECIAL_109>",
+    "<SPECIAL_110>",
+    "<SPECIAL_111>",
+    "<SPECIAL_112>",
+    "<SPECIAL_113>",
+    "<SPECIAL_114>",
+    "<SPECIAL_115>",
+    "<SPECIAL_116>",
+    "<SPECIAL_117>",
+    "<SPECIAL_118>",
+    "<SPECIAL_119>",
+    "<SPECIAL_120>",
+    "<SPECIAL_121>",
+    "<SPECIAL_122>",
+    "<SPECIAL_123>",
+    "<SPECIAL_124>",
+    "<SPECIAL_125>",
+    "<SPECIAL_126>",
+    "<SPECIAL_127>",
+    "<SPECIAL_128>",
+    "<SPECIAL_129>",
+    "<SPECIAL_130>",
+    "<SPECIAL_131>",
+    "<SPECIAL_132>",
+    "<SPECIAL_133>",
+    "<SPECIAL_134>",
+    "<SPECIAL_135>",
+    "<SPECIAL_136>",
+    "<SPECIAL_137>",
+    "<SPECIAL_138>",
+    "<SPECIAL_139>",
+    "<SPECIAL_140>",
+    "<SPECIAL_141>",
+    "<SPECIAL_142>",
+    "<SPECIAL_143>",
+    "<SPECIAL_144>",
+    "<SPECIAL_145>",
+    "<SPECIAL_146>",
+    "<SPECIAL_147>",
+    "<SPECIAL_148>",
+    "<SPECIAL_149>",
+    "<SPECIAL_150>",
+    "<SPECIAL_151>",
+    "<SPECIAL_152>",
+    "<SPECIAL_153>",
+    "<SPECIAL_154>",
+    "<SPECIAL_155>",
+    "<SPECIAL_156>",
+    "<SPECIAL_157>",
+    "<SPECIAL_158>",
+    "<SPECIAL_159>",
+    "<SPECIAL_160>",
+    "<SPECIAL_161>",
+    "<SPECIAL_162>",
+    "<SPECIAL_163>",
+    "<SPECIAL_164>",
+    "<SPECIAL_165>",
+    "<SPECIAL_166>",
+    "<SPECIAL_167>",
+    "<SPECIAL_168>",
+    "<SPECIAL_169>",
+    "<SPECIAL_170>",
+    "<SPECIAL_171>",
+    "<SPECIAL_172>",
+    "<SPECIAL_173>",
+    "<SPECIAL_174>",
+    "<SPECIAL_175>",
+    "<SPECIAL_176>",
+    "<SPECIAL_177>",
+    "<SPECIAL_178>",
+    "<SPECIAL_179>",
+    "<SPECIAL_180>",
+    "<SPECIAL_181>",
+    "<SPECIAL_182>",
+    "<SPECIAL_183>",
+    "<SPECIAL_184>",
+    "<SPECIAL_185>",
+    "<SPECIAL_186>",
+    "<SPECIAL_187>",
+    "<SPECIAL_188>",
+    "<SPECIAL_189>",
+    "<SPECIAL_190>",
+    "<SPECIAL_191>",
+    "<SPECIAL_192>",
+    "<SPECIAL_193>",
+    "<SPECIAL_194>",
+    "<SPECIAL_195>",
+    "<SPECIAL_196>",
+    "<SPECIAL_197>",
+    "<SPECIAL_198>",
+    "<SPECIAL_199>",
+    "<SPECIAL_200>",
+    "<SPECIAL_201>",
+    "<SPECIAL_202>",
+    "<SPECIAL_203>",
+    "<SPECIAL_204>",
+    "<SPECIAL_205>",
+    "<SPECIAL_206>",
+    "<SPECIAL_207>",
+    "<SPECIAL_208>",
+    "<SPECIAL_209>",
+    "<SPECIAL_210>",
+    "<SPECIAL_211>",
+    "<SPECIAL_212>",
+    "<SPECIAL_213>",
+    "<SPECIAL_214>",
+    "<SPECIAL_215>",
+    "<SPECIAL_216>",
+    "<SPECIAL_217>",
+    "<SPECIAL_218>",
+    "<SPECIAL_219>",
+    "<SPECIAL_220>",
+    "<SPECIAL_221>",
+    "<SPECIAL_222>",
+    "<SPECIAL_223>",
+    "<SPECIAL_224>",
+    "<SPECIAL_225>",
+    "<SPECIAL_226>",
+    "<SPECIAL_227>",
+    "<SPECIAL_228>",
+    "<SPECIAL_229>",
+    "<SPECIAL_230>",
+    "<SPECIAL_231>",
+    "<SPECIAL_232>",
+    "<SPECIAL_233>",
+    "<SPECIAL_234>",
+    "<SPECIAL_235>",
+    "<SPECIAL_236>",
+    "<SPECIAL_237>",
+    "<SPECIAL_238>",
+    "<SPECIAL_239>",
+    "<SPECIAL_240>",
+    "<SPECIAL_241>",
+    "<SPECIAL_242>",
+    "<SPECIAL_243>",
+    "<SPECIAL_244>",
+    "<SPECIAL_245>",
+    "<SPECIAL_246>",
+    "<SPECIAL_247>",
+    "<SPECIAL_248>",
+    "<SPECIAL_249>",
+    "<SPECIAL_250>",
+    "<SPECIAL_251>",
+    "<SPECIAL_252>",
+    "<SPECIAL_253>",
+    "<SPECIAL_254>",
+    "<SPECIAL_255>",
+    "<SPECIAL_256>",
+    "<SPECIAL_257>",
+    "<SPECIAL_258>",
+    "<SPECIAL_259>",
+    "<SPECIAL_260>",
+    "<SPECIAL_261>",
+    "<SPECIAL_262>",
+    "<SPECIAL_263>",
+    "<SPECIAL_264>",
+    "<SPECIAL_265>",
+    "<SPECIAL_266>",
+    "<SPECIAL_267>",
+    "<SPECIAL_268>",
+    "<SPECIAL_269>",
+    "<SPECIAL_270>",
+    "<SPECIAL_271>",
+    "<SPECIAL_272>",
+    "<SPECIAL_273>",
+    "<SPECIAL_274>",
+    "<SPECIAL_275>",
+    "<SPECIAL_276>",
+    "<SPECIAL_277>",
+    "<SPECIAL_278>",
+    "<SPECIAL_279>",
+    "<SPECIAL_280>",
+    "<SPECIAL_281>",
+    "<SPECIAL_282>",
+    "<SPECIAL_283>",
+    "<SPECIAL_284>",
+    "<SPECIAL_285>",
+    "<SPECIAL_286>",
+    "<SPECIAL_287>",
+    "<SPECIAL_288>",
+    "<SPECIAL_289>",
+    "<SPECIAL_290>",
+    "<SPECIAL_291>",
+    "<SPECIAL_292>",
+    "<SPECIAL_293>",
+    "<SPECIAL_294>",
+    "<SPECIAL_295>",
+    "<SPECIAL_296>",
+    "<SPECIAL_297>",
+    "<SPECIAL_298>",
+    "<SPECIAL_299>",
+    "<SPECIAL_300>",
+    "<SPECIAL_301>",
+    "<SPECIAL_302>",
+    "<SPECIAL_303>",
+    "<SPECIAL_304>",
+    "<SPECIAL_305>",
+    "<SPECIAL_306>",
+    "<SPECIAL_307>",
+    "<SPECIAL_308>",
+    "<SPECIAL_309>",
+    "<SPECIAL_310>",
+    "<SPECIAL_311>",
+    "<SPECIAL_312>",
+    "<SPECIAL_313>",
+    "<SPECIAL_314>",
+    "<SPECIAL_315>",
+    "<SPECIAL_316>",
+    "<SPECIAL_317>",
+    "<SPECIAL_318>",
+    "<SPECIAL_319>",
+    "<SPECIAL_320>",
+    "<SPECIAL_321>",
+    "<SPECIAL_322>",
+    "<SPECIAL_323>",
+    "<SPECIAL_324>",
+    "<SPECIAL_325>",
+    "<SPECIAL_326>",
+    "<SPECIAL_327>",
+    "<SPECIAL_328>",
+    "<SPECIAL_329>",
+    "<SPECIAL_330>",
+    "<SPECIAL_331>",
+    "<SPECIAL_332>",
+    "<SPECIAL_333>",
+    "<SPECIAL_334>",
+    "<SPECIAL_335>",
+    "<SPECIAL_336>",
+    "<SPECIAL_337>",
+    "<SPECIAL_338>",
+    "<SPECIAL_339>",
+    "<SPECIAL_340>",
+    "<SPECIAL_341>",
+    "<SPECIAL_342>",
+    "<SPECIAL_343>",
+    "<SPECIAL_344>",
+    "<SPECIAL_345>",
+    "<SPECIAL_346>",
+    "<SPECIAL_347>",
+    "<SPECIAL_348>",
+    "<SPECIAL_349>",
+    "<SPECIAL_350>",
+    "<SPECIAL_351>",
+    "<SPECIAL_352>",
+    "<SPECIAL_353>",
+    "<SPECIAL_354>",
+    "<SPECIAL_355>",
+    "<SPECIAL_356>",
+    "<SPECIAL_357>",
+    "<SPECIAL_358>",
+    "<SPECIAL_359>",
+    "<SPECIAL_360>",
+    "<SPECIAL_361>",
+    "<SPECIAL_362>",
+    "<SPECIAL_363>",
+    "<SPECIAL_364>",
+    "<SPECIAL_365>",
+    "<SPECIAL_366>",
+    "<SPECIAL_367>",
+    "<SPECIAL_368>",
+    "<SPECIAL_369>",
+    "<SPECIAL_370>",
+    "<SPECIAL_371>",
+    "<SPECIAL_372>",
+    "<SPECIAL_373>",
+    "<SPECIAL_374>",
+    "<SPECIAL_375>",
+    "<SPECIAL_376>",
+    "<SPECIAL_377>",
+    "<SPECIAL_378>",
+    "<SPECIAL_379>",
+    "<SPECIAL_380>",
+    "<SPECIAL_381>",
+    "<SPECIAL_382>",
+    "<SPECIAL_383>",
+    "<SPECIAL_384>",
+    "<SPECIAL_385>",
+    "<SPECIAL_386>",
+    "<SPECIAL_387>",
+    "<SPECIAL_388>",
+    "<SPECIAL_389>",
+    "<SPECIAL_390>",
+    "<SPECIAL_391>",
+    "<SPECIAL_392>",
+    "<SPECIAL_393>",
+    "<SPECIAL_394>",
+    "<SPECIAL_395>",
+    "<SPECIAL_396>",
+    "<SPECIAL_397>",
+    "<SPECIAL_398>",
+    "<SPECIAL_399>",
+    "<SPECIAL_400>",
+    "<SPECIAL_401>",
+    "<SPECIAL_402>",
+    "<SPECIAL_403>",
+    "<SPECIAL_404>",
+    "<SPECIAL_405>",
+    "<SPECIAL_406>",
+    "<SPECIAL_407>",
+    "<SPECIAL_408>",
+    "<SPECIAL_409>",
+    "<SPECIAL_410>",
+    "<SPECIAL_411>",
+    "<SPECIAL_412>",
+    "<SPECIAL_413>",
+    "<SPECIAL_414>",
+    "<SPECIAL_415>",
+    "<SPECIAL_416>",
+    "<SPECIAL_417>",
+    "<SPECIAL_418>",
+    "<SPECIAL_419>",
+    "<SPECIAL_420>",
+    "<SPECIAL_421>",
+    "<SPECIAL_422>",
+    "<SPECIAL_423>",
+    "<SPECIAL_424>",
+    "<SPECIAL_425>",
+    "<SPECIAL_426>",
+    "<SPECIAL_427>",
+    "<SPECIAL_428>",
+    "<SPECIAL_429>",
+    "<SPECIAL_430>",
+    "<SPECIAL_431>",
+    "<SPECIAL_432>",
+    "<SPECIAL_433>",
+    "<SPECIAL_434>",
+    "<SPECIAL_435>",
+    "<SPECIAL_436>",
+    "<SPECIAL_437>",
+    "<SPECIAL_438>",
+    "<SPECIAL_439>",
+    "<SPECIAL_440>",
+    "<SPECIAL_441>",
+    "<SPECIAL_442>",
+    "<SPECIAL_443>",
+    "<SPECIAL_444>",
+    "<SPECIAL_445>",
+    "<SPECIAL_446>",
+    "<SPECIAL_447>",
+    "<SPECIAL_448>",
+    "<SPECIAL_449>",
+    "<SPECIAL_450>",
+    "<SPECIAL_451>",
+    "<SPECIAL_452>",
+    "<SPECIAL_453>",
+    "<SPECIAL_454>",
+    "<SPECIAL_455>",
+    "<SPECIAL_456>",
+    "<SPECIAL_457>",
+    "<SPECIAL_458>",
+    "<SPECIAL_459>",
+    "<SPECIAL_460>",
+    "<SPECIAL_461>",
+    "<SPECIAL_462>",
+    "<SPECIAL_463>",
+    "<SPECIAL_464>",
+    "<SPECIAL_465>",
+    "<SPECIAL_466>",
+    "<SPECIAL_467>",
+    "<SPECIAL_468>",
+    "<SPECIAL_469>",
+    "<SPECIAL_470>",
+    "<SPECIAL_471>",
+    "<SPECIAL_472>",
+    "<SPECIAL_473>",
+    "<SPECIAL_474>",
+    "<SPECIAL_475>",
+    "<SPECIAL_476>",
+    "<SPECIAL_477>",
+    "<SPECIAL_478>",
+    "<SPECIAL_479>",
+    "<SPECIAL_480>",
+    "<SPECIAL_481>",
+    "<SPECIAL_482>",
+    "<SPECIAL_483>",
+    "<SPECIAL_484>",
+    "<SPECIAL_485>",
+    "<SPECIAL_486>",
+    "<SPECIAL_487>",
+    "<SPECIAL_488>",
+    "<SPECIAL_489>",
+    "<SPECIAL_490>",
+    "<SPECIAL_491>",
+    "<SPECIAL_492>",
+    "<SPECIAL_493>",
+    "<SPECIAL_494>",
+    "<SPECIAL_495>",
+    "<SPECIAL_496>",
+    "<SPECIAL_497>",
+    "<SPECIAL_498>",
+    "<SPECIAL_499>",
+    "<SPECIAL_500>",
+    "<SPECIAL_501>",
+    "<SPECIAL_502>",
+    "<SPECIAL_503>",
+    "<SPECIAL_504>",
+    "<SPECIAL_505>",
+    "<SPECIAL_506>",
+    "<SPECIAL_507>",
+    "<SPECIAL_508>",
+    "<SPECIAL_509>",
+    "<SPECIAL_510>",
+    "<SPECIAL_511>",
+    "<SPECIAL_512>",
+    "<SPECIAL_513>",
+    "<SPECIAL_514>",
+    "<SPECIAL_515>",
+    "<SPECIAL_516>",
+    "<SPECIAL_517>",
+    "<SPECIAL_518>",
+    "<SPECIAL_519>",
+    "<SPECIAL_520>",
+    "<SPECIAL_521>",
+    "<SPECIAL_522>",
+    "<SPECIAL_523>",
+    "<SPECIAL_524>",
+    "<SPECIAL_525>",
+    "<SPECIAL_526>",
+    "<SPECIAL_527>",
+    "<SPECIAL_528>",
+    "<SPECIAL_529>",
+    "<SPECIAL_530>",
+    "<SPECIAL_531>",
+    "<SPECIAL_532>",
+    "<SPECIAL_533>",
+    "<SPECIAL_534>",
+    "<SPECIAL_535>",
+    "<SPECIAL_536>",
+    "<SPECIAL_537>",
+    "<SPECIAL_538>",
+    "<SPECIAL_539>",
+    "<SPECIAL_540>",
+    "<SPECIAL_541>",
+    "<SPECIAL_542>",
+    "<SPECIAL_543>",
+    "<SPECIAL_544>",
+    "<SPECIAL_545>",
+    "<SPECIAL_546>",
+    "<SPECIAL_547>",
+    "<SPECIAL_548>",
+    "<SPECIAL_549>",
+    "<SPECIAL_550>",
+    "<SPECIAL_551>",
+    "<SPECIAL_552>",
+    "<SPECIAL_553>",
+    "<SPECIAL_554>",
+    "<SPECIAL_555>",
+    "<SPECIAL_556>",
+    "<SPECIAL_557>",
+    "<SPECIAL_558>",
+    "<SPECIAL_559>",
+    "<SPECIAL_560>",
+    "<SPECIAL_561>",
+    "<SPECIAL_562>",
+    "<SPECIAL_563>",
+    "<SPECIAL_564>",
+    "<SPECIAL_565>",
+    "<SPECIAL_566>",
+    "<SPECIAL_567>",
+    "<SPECIAL_568>",
+    "<SPECIAL_569>",
+    "<SPECIAL_570>",
+    "<SPECIAL_571>",
+    "<SPECIAL_572>",
+    "<SPECIAL_573>",
+    "<SPECIAL_574>",
+    "<SPECIAL_575>",
+    "<SPECIAL_576>",
+    "<SPECIAL_577>",
+    "<SPECIAL_578>",
+    "<SPECIAL_579>",
+    "<SPECIAL_580>",
+    "<SPECIAL_581>",
+    "<SPECIAL_582>",
+    "<SPECIAL_583>",
+    "<SPECIAL_584>",
+    "<SPECIAL_585>",
+    "<SPECIAL_586>",
+    "<SPECIAL_587>",
+    "<SPECIAL_588>",
+    "<SPECIAL_589>",
+    "<SPECIAL_590>",
+    "<SPECIAL_591>",
+    "<SPECIAL_592>",
+    "<SPECIAL_593>",
+    "<SPECIAL_594>",
+    "<SPECIAL_595>",
+    "<SPECIAL_596>",
+    "<SPECIAL_597>",
+    "<SPECIAL_598>",
+    "<SPECIAL_599>",
+    "<SPECIAL_600>",
+    "<SPECIAL_601>",
+    "<SPECIAL_602>",
+    "<SPECIAL_603>",
+    "<SPECIAL_604>",
+    "<SPECIAL_605>",
+    "<SPECIAL_606>",
+    "<SPECIAL_607>",
+    "<SPECIAL_608>",
+    "<SPECIAL_609>",
+    "<SPECIAL_610>",
+    "<SPECIAL_611>",
+    "<SPECIAL_612>",
+    "<SPECIAL_613>",
+    "<SPECIAL_614>",
+    "<SPECIAL_615>",
+    "<SPECIAL_616>",
+    "<SPECIAL_617>",
+    "<SPECIAL_618>",
+    "<SPECIAL_619>",
+    "<SPECIAL_620>",
+    "<SPECIAL_621>",
+    "<SPECIAL_622>",
+    "<SPECIAL_623>",
+    "<SPECIAL_624>",
+    "<SPECIAL_625>",
+    "<SPECIAL_626>",
+    "<SPECIAL_627>",
+    "<SPECIAL_628>",
+    "<SPECIAL_629>",
+    "<SPECIAL_630>",
+    "<SPECIAL_631>",
+    "<SPECIAL_632>",
+    "<SPECIAL_633>",
+    "<SPECIAL_634>",
+    "<SPECIAL_635>",
+    "<SPECIAL_636>",
+    "<SPECIAL_637>",
+    "<SPECIAL_638>",
+    "<SPECIAL_639>",
+    "<SPECIAL_640>",
+    "<SPECIAL_641>",
+    "<SPECIAL_642>",
+    "<SPECIAL_643>",
+    "<SPECIAL_644>",
+    "<SPECIAL_645>",
+    "<SPECIAL_646>",
+    "<SPECIAL_647>",
+    "<SPECIAL_648>",
+    "<SPECIAL_649>",
+    "<SPECIAL_650>",
+    "<SPECIAL_651>",
+    "<SPECIAL_652>",
+    "<SPECIAL_653>",
+    "<SPECIAL_654>",
+    "<SPECIAL_655>",
+    "<SPECIAL_656>",
+    "<SPECIAL_657>",
+    "<SPECIAL_658>",
+    "<SPECIAL_659>",
+    "<SPECIAL_660>",
+    "<SPECIAL_661>",
+    "<SPECIAL_662>",
+    "<SPECIAL_663>",
+    "<SPECIAL_664>",
+    "<SPECIAL_665>",
+    "<SPECIAL_666>",
+    "<SPECIAL_667>",
+    "<SPECIAL_668>",
+    "<SPECIAL_669>",
+    "<SPECIAL_670>",
+    "<SPECIAL_671>",
+    "<SPECIAL_672>",
+    "<SPECIAL_673>",
+    "<SPECIAL_674>",
+    "<SPECIAL_675>",
+    "<SPECIAL_676>",
+    "<SPECIAL_677>",
+    "<SPECIAL_678>",
+    "<SPECIAL_679>",
+    "<SPECIAL_680>",
+    "<SPECIAL_681>",
+    "<SPECIAL_682>",
+    "<SPECIAL_683>",
+    "<SPECIAL_684>",
+    "<SPECIAL_685>",
+    "<SPECIAL_686>",
+    "<SPECIAL_687>",
+    "<SPECIAL_688>",
+    "<SPECIAL_689>",
+    "<SPECIAL_690>",
+    "<SPECIAL_691>",
+    "<SPECIAL_692>",
+    "<SPECIAL_693>",
+    "<SPECIAL_694>",
+    "<SPECIAL_695>",
+    "<SPECIAL_696>",
+    "<SPECIAL_697>",
+    "<SPECIAL_698>",
+    "<SPECIAL_699>",
+    "<SPECIAL_700>",
+    "<SPECIAL_701>",
+    "<SPECIAL_702>",
+    "<SPECIAL_703>",
+    "<SPECIAL_704>",
+    "<SPECIAL_705>",
+    "<SPECIAL_706>",
+    "<SPECIAL_707>",
+    "<SPECIAL_708>",
+    "<SPECIAL_709>",
+    "<SPECIAL_710>",
+    "<SPECIAL_711>",
+    "<SPECIAL_712>",
+    "<SPECIAL_713>",
+    "<SPECIAL_714>",
+    "<SPECIAL_715>",
+    "<SPECIAL_716>",
+    "<SPECIAL_717>",
+    "<SPECIAL_718>",
+    "<SPECIAL_719>",
+    "<SPECIAL_720>",
+    "<SPECIAL_721>",
+    "<SPECIAL_722>",
+    "<SPECIAL_723>",
+    "<SPECIAL_724>",
+    "<SPECIAL_725>",
+    "<SPECIAL_726>",
+    "<SPECIAL_727>",
+    "<SPECIAL_728>",
+    "<SPECIAL_729>",
+    "<SPECIAL_730>",
+    "<SPECIAL_731>",
+    "<SPECIAL_732>",
+    "<SPECIAL_733>",
+    "<SPECIAL_734>",
+    "<SPECIAL_735>",
+    "<SPECIAL_736>",
+    "<SPECIAL_737>",
+    "<SPECIAL_738>",
+    "<SPECIAL_739>",
+    "<SPECIAL_740>",
+    "<SPECIAL_741>",
+    "<SPECIAL_742>",
+    "<SPECIAL_743>",
+    "<SPECIAL_744>",
+    "<SPECIAL_745>",
+    "<SPECIAL_746>",
+    "<SPECIAL_747>",
+    "<SPECIAL_748>",
+    "<SPECIAL_749>",
+    "<SPECIAL_750>",
+    "<SPECIAL_751>",
+    "<SPECIAL_752>",
+    "<SPECIAL_753>",
+    "<SPECIAL_754>",
+    "<SPECIAL_755>",
+    "<SPECIAL_756>",
+    "<SPECIAL_757>",
+    "<SPECIAL_758>",
+    "<SPECIAL_759>",
+    "<SPECIAL_760>",
+    "<SPECIAL_761>",
+    "<SPECIAL_762>",
+    "<SPECIAL_763>",
+    "<SPECIAL_764>",
+    "<SPECIAL_765>",
+    "<SPECIAL_766>",
+    "<SPECIAL_767>",
+    "<SPECIAL_768>",
+    "<SPECIAL_769>",
+    "<SPECIAL_770>",
+    "<SPECIAL_771>",
+    "<SPECIAL_772>",
+    "<SPECIAL_773>",
+    "<SPECIAL_774>",
+    "<SPECIAL_775>",
+    "<SPECIAL_776>",
+    "<SPECIAL_777>",
+    "<SPECIAL_778>",
+    "<SPECIAL_779>",
+    "<SPECIAL_780>",
+    "<SPECIAL_781>",
+    "<SPECIAL_782>",
+    "<SPECIAL_783>",
+    "<SPECIAL_784>",
+    "<SPECIAL_785>",
+    "<SPECIAL_786>",
+    "<SPECIAL_787>",
+    "<SPECIAL_788>",
+    "<SPECIAL_789>",
+    "<SPECIAL_790>",
+    "<SPECIAL_791>",
+    "<SPECIAL_792>",
+    "<SPECIAL_793>",
+    "<SPECIAL_794>",
+    "<SPECIAL_795>",
+    "<SPECIAL_796>",
+    "<SPECIAL_797>",
+    "<SPECIAL_798>",
+    "<SPECIAL_799>",
+    "<SPECIAL_800>",
+    "<SPECIAL_801>",
+    "<SPECIAL_802>",
+    "<SPECIAL_803>",
+    "<SPECIAL_804>",
+    "<SPECIAL_805>",
+    "<SPECIAL_806>",
+    "<SPECIAL_807>",
+    "<SPECIAL_808>",
+    "<SPECIAL_809>",
+    "<SPECIAL_810>",
+    "<SPECIAL_811>",
+    "<SPECIAL_812>",
+    "<SPECIAL_813>",
+    "<SPECIAL_814>",
+    "<SPECIAL_815>",
+    "<SPECIAL_816>",
+    "<SPECIAL_817>",
+    "<SPECIAL_818>",
+    "<SPECIAL_819>",
+    "<SPECIAL_820>",
+    "<SPECIAL_821>",
+    "<SPECIAL_822>",
+    "<SPECIAL_823>",
+    "<SPECIAL_824>",
+    "<SPECIAL_825>",
+    "<SPECIAL_826>",
+    "<SPECIAL_827>",
+    "<SPECIAL_828>",
+    "<SPECIAL_829>",
+    "<SPECIAL_830>",
+    "<SPECIAL_831>",
+    "<SPECIAL_832>",
+    "<SPECIAL_833>",
+    "<SPECIAL_834>",
+    "<SPECIAL_835>",
+    "<SPECIAL_836>",
+    "<SPECIAL_837>",
+    "<SPECIAL_838>",
+    "<SPECIAL_839>",
+    "<SPECIAL_840>",
+    "<SPECIAL_841>",
+    "<SPECIAL_842>",
+    "<SPECIAL_843>",
+    "<SPECIAL_844>",
+    "<SPECIAL_845>",
+    "<SPECIAL_846>",
+    "<SPECIAL_847>",
+    "<SPECIAL_848>",
+    "<SPECIAL_849>",
+    "<SPECIAL_850>",
+    "<SPECIAL_851>",
+    "<SPECIAL_852>",
+    "<SPECIAL_853>",
+    "<SPECIAL_854>",
+    "<SPECIAL_855>",
+    "<SPECIAL_856>",
+    "<SPECIAL_857>",
+    "<SPECIAL_858>",
+    "<SPECIAL_859>",
+    "<SPECIAL_860>",
+    "<SPECIAL_861>",
+    "<SPECIAL_862>",
+    "<SPECIAL_863>",
+    "<SPECIAL_864>",
+    "<SPECIAL_865>",
+    "<SPECIAL_866>",
+    "<SPECIAL_867>",
+    "<SPECIAL_868>",
+    "<SPECIAL_869>",
+    "<SPECIAL_870>",
+    "<SPECIAL_871>",
+    "<SPECIAL_872>",
+    "<SPECIAL_873>",
+    "<SPECIAL_874>",
+    "<SPECIAL_875>",
+    "<SPECIAL_876>",
+    "<SPECIAL_877>",
+    "<SPECIAL_878>",
+    "<SPECIAL_879>",
+    "<SPECIAL_880>",
+    "<SPECIAL_881>",
+    "<SPECIAL_882>",
+    "<SPECIAL_883>",
+    "<SPECIAL_884>",
+    "<SPECIAL_885>",
+    "<SPECIAL_886>",
+    "<SPECIAL_887>",
+    "<SPECIAL_888>",
+    "<SPECIAL_889>",
+    "<SPECIAL_890>",
+    "<SPECIAL_891>",
+    "<SPECIAL_892>",
+    "<SPECIAL_893>",
+    "<SPECIAL_894>",
+    "<SPECIAL_895>",
+    "<SPECIAL_896>",
+    "<SPECIAL_897>",
+    "<SPECIAL_898>",
+    "<SPECIAL_899>",
+    "<SPECIAL_900>",
+    "<SPECIAL_901>",
+    "<SPECIAL_902>",
+    "<SPECIAL_903>",
+    "<SPECIAL_904>",
+    "<SPECIAL_905>",
+    "<SPECIAL_906>",
+    "<SPECIAL_907>",
+    "<SPECIAL_908>",
+    "<SPECIAL_909>",
+    "<SPECIAL_910>",
+    "<SPECIAL_911>",
+    "<SPECIAL_912>",
+    "<SPECIAL_913>",
+    "<SPECIAL_914>",
+    "<SPECIAL_915>",
+    "<SPECIAL_916>",
+    "<SPECIAL_917>",
+    "<SPECIAL_918>",
+    "<SPECIAL_919>",
+    "<SPECIAL_920>",
+    "<SPECIAL_921>",
+    "<SPECIAL_922>",
+    "<SPECIAL_923>",
+    "<SPECIAL_924>",
+    "<SPECIAL_925>",
+    "<SPECIAL_926>",
+    "<SPECIAL_927>",
+    "<SPECIAL_928>",
+    "<SPECIAL_929>",
+    "<SPECIAL_930>",
+    "<SPECIAL_931>",
+    "<SPECIAL_932>",
+    "<SPECIAL_933>",
+    "<SPECIAL_934>",
+    "<SPECIAL_935>",
+    "<SPECIAL_936>",
+    "<SPECIAL_937>",
+    "<SPECIAL_938>",
+    "<SPECIAL_939>",
+    "<SPECIAL_940>",
+    "<SPECIAL_941>",
+    "<SPECIAL_942>",
+    "<SPECIAL_943>",
+    "<SPECIAL_944>",
+    "<SPECIAL_945>",
+    "<SPECIAL_946>",
+    "<SPECIAL_947>",
+    "<SPECIAL_948>",
+    "<SPECIAL_949>",
+    "<SPECIAL_950>",
+    "<SPECIAL_951>",
+    "<SPECIAL_952>",
+    "<SPECIAL_953>",
+    "<SPECIAL_954>",
+    "<SPECIAL_955>",
+    "<SPECIAL_956>",
+    "<SPECIAL_957>",
+    "<SPECIAL_958>",
+    "<SPECIAL_959>",
+    "<SPECIAL_960>",
+    "<SPECIAL_961>",
+    "<SPECIAL_962>",
+    "<SPECIAL_963>",
+    "<SPECIAL_964>",
+    "<SPECIAL_965>",
+    "<SPECIAL_966>",
+    "<SPECIAL_967>",
+    "<SPECIAL_968>",
+    "<SPECIAL_969>",
+    "<SPECIAL_970>",
+    "<SPECIAL_971>",
+    "<SPECIAL_972>",
+    "<SPECIAL_973>",
+    "<SPECIAL_974>",
+    "<SPECIAL_975>",
+    "<SPECIAL_976>",
+    "<SPECIAL_977>",
+    "<SPECIAL_978>",
+    "<SPECIAL_979>",
+    "<SPECIAL_980>",
+    "<SPECIAL_981>",
+    "<SPECIAL_982>",
+    "<SPECIAL_983>",
+    "<SPECIAL_984>",
+    "<SPECIAL_985>",
+    "<SPECIAL_986>",
+    "<SPECIAL_987>",
+    "<SPECIAL_988>",
+    "<SPECIAL_989>",
+    "<SPECIAL_990>",
+    "<SPECIAL_991>",
+    "<SPECIAL_992>",
+    "<SPECIAL_993>",
+    "<SPECIAL_994>",
+    "<SPECIAL_995>",
+    "<SPECIAL_996>",
+    "<SPECIAL_997>",
+    "<SPECIAL_998>",
+    "<SPECIAL_999>"
+  ],
+  "is_local": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "processor_class": "PixtralProcessor",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}