---
license: apache-2.0
base_model: OpenMed/privacy-filter-multilingual
base_model_relation: quantized
pipeline_tag: token-classification
library_name: gguf
tags:
  - gguf
  - llama-cpp
  - localai
  - token-classification
  - pii
  - ner
  - privacy
  - redaction
  - multilingual
  - openai-privacy-filter
language:
  - ar
  - bn
  - de
  - en
  - es
  - fr
  - hi
  - it
  - ja
  - ko
  - nl
  - pt
  - te
  - tr
  - vi
  - zh
---

# privacy-filter-multilingual — GGUF (F16)

GGUF conversion of [`OpenMed/privacy-filter-multilingual`](https://huggingface.co/OpenMed/privacy-filter-multilingual),
a multilingual PII **token-classification** model (a fine-tune of
[`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter)). It labels every
token with a BIOES tag over **54 PII categories (217 classes)** across **16 languages**, so
it can be served locally with **no Python** as the encoder/NER tier of a PII redactor.

For the full model description, label space, evaluation, limitations, and citations, see the
**[source model card](https://huggingface.co/OpenMed/privacy-filter-multilingual)** — this
card only covers the GGUF packaging and how to run it.

---

## ⚠️ Requires a patched llama.cpp — does NOT run on stock builds

> **Read this before downloading.** This GGUF uses a **custom architecture,
> `openai-privacy-filter`, that is not (yet) part of upstream llama.cpp.** It will **fail to
> load** in stock `llama.cpp`, `llama-cpp-python`, Ollama, LM Studio, or any other off-the-shelf
> GGUF runtime. You will see an error like `unknown model architecture: 'openai-privacy-filter'`.
>
> It runs **only** on a build that carries the patches **[LocalAI PR#10160](https://github.com/mudler/LocalAI/pull/10160)**
> ships in `backend/cpp/llama-cpp/patches/` (applied automatically by the backend's
> `prepare.sh`). Those five patches add:
> 1. **TOKEN_CLS pooling** (`LLAMA_POOLING_TYPE_TOKEN_CLS`) — a per-token classification head
>    (a reduced subset of upstream **[PR #19725](https://github.com/ggml-org/llama.cpp/pull/19725)**, still open);
> 2. the **`openai-privacy-filter` architecture** registration;
> 3. the **HF→GGUF converter** for it;
> 4. the **bidirectional banded-attention graph** + loader (non-causal symmetric sliding
>    window, attention sinks, interleaved RoPE, exact YaRN `truncate=false` frequencies);
> 5. an **all-SWA no-cache mask fix** required for an encoder that windows every layer.
>
> These are **carry-patches against a pinned llama.cpp commit**, not upstream features. Until
> the architecture is upstreamed (it depends on PR #19725 landing first), running this model
> means using LocalAI's vendored backend. At time of writing the support lives on a LocalAI
> feature branch — use a LocalAI build that includes the `openai-privacy-filter` patches.

---

## Use with LocalAI

Install from the LocalAI model gallery (the entry sets `backend: llama-cpp`,
`embeddings: true`, and `known_usecases: [token_classify]`):

```bash
local-ai models install privacy-filter-multilingual
```

The model is **not** a chat/completion model — it exposes the gRPC `TokenClassify` RPC. It is a
**PII detector**: it carries its own detection policy in a `pii_detection:` block, and other
models opt in by listing it in `pii.detectors`. The gallery entry ships a sensible default
policy (mask everything detected; block credentials/financial-secrets/crypto):

```yaml
# the detector model (this GGUF) — policy lives here
name: privacy-filter-multilingual
backend: llama-cpp
embeddings: true
known_usecases: [token_classify]
pii_detection:
  min_score: 0.5
  default_action: mask          # mask | block | allow
  entity_actions:               # which categories to block vs mask
    PASSWORD: block
    CREDITCARD: block
    CVV: block
```

```yaml
# any chat or cloud-proxy model — opt in and reference the detector(s)
name: my-assistant
backend: llama-cpp
pii:
  enabled: true
  detectors:
    - privacy-filter-multilingual
```

LocalAI runs the model's **constrained BIOES Viterbi** decode in the backend and returns
entity spans with **UTF-8 byte offsets**; the redactor masks/blocks per the detector's
`pii_detection` policy. Multiple detectors union their hits (strongest action wins).

> Load note: the model must be loaded with **TOKEN_CLS pooling** (it is the GGUF's default —
> the LocalAI gallery config and `embeddings: true` handle this). If you drive `llama-embedding`
> directly for testing, do **not** pass `--pooling none`; that overrides the model default and
> you get raw hidden states instead of label logits.

## Files

| File | Precision | Size | Notes |
|---|---|---|---|
| `privacy-filter-multilingual-f16.gguf` | F16 | ~2.7 GB | 156 tensors; 217 `classifier.output_labels`; `pooling_type = TOKEN_CLS`. Validated artifact. |

F16 is the validated, shipped precision. Quantized variants are deferred until they can be
evaluated with a **task metric (span-F1 per language) + KL-vs-F16** — perplexity is meaningless
for a classifier, so a naively-quantized GGUF is not published here yet.

## Architecture & conversion

gpt-oss-style sparse **MoE** (8 layers, `d_model=640`, 128 experts, top-4 routing, ~50M active
per token), **bidirectional banded attention** (symmetric sliding window 128, attention sinks
retained), **interleaved (GPT-J) RoPE** with YaRN (θ=150000, factor 32), o200k (`o200k_base`)
tokenizer, and a 217-way token-classification head (`score` → `cls.output`).

The conversion reproduces the HF reference **exactly at F16**: token-for-token argmax match
(12/12 on the parity prompt set), **full-logit cosine = 1.0**, every layer's residual-stream
cosine = 1.0 (relerr ≈ 2e-4, i.e. F16 rounding). The two load-bearing conversion choices — the
expert `gate_up` `chunk(2)` split and the `n_swa = 2·sliding_window` window mapping — are both
confirmed by that parity. See LocalAI's `backend/cpp/llama-cpp/patches/README.md` for the full
provenance.

## Label space

`O` plus `B-`/`I-`/`E-`/`S-` for each of 54 categories (1 + 54×4 = 217), spanning identity,
contact, address, dates/time, government IDs, financial, crypto, vehicle, digital, and auth
entities. The ordered `id2label` table is embedded in the GGUF (`classifier.output_labels`).
See the [source card](https://huggingface.co/OpenMed/privacy-filter-multilingual#label-space-54-categories)
for the full list.

## Limitations & intended use

Identical to the [source model](https://huggingface.co/OpenMed/privacy-filter-multilingual#limitations--intended-use):
multilingual but uneven (strongest on de/es/fr/it/hi/te/en; weaker on CJK), trained on
synthetic AI4Privacy data, **not** a substitute for legal/compliance review, and **not** a
clinical PHI model. Use it as one tier behind deterministic regex pre-filters and human review.

## License

**Apache-2.0**, inherited from `openai/privacy-filter` and `OpenMed/privacy-filter-multilingual`.

## Credits & citation

Conversion and runtime support by the **LocalAI** project. The model itself is by **OpenMed**,
fine-tuned from **OpenAI**'s `privacy-filter`, on **AI4Privacy** datasets — please cite all of
them (BibTeX in the [source card](https://huggingface.co/OpenMed/privacy-filter-multilingual#citation)).