LocalAI-io/privacy-filter-multilingual-GGUF

privacy-filter-multilingual — GGUF (F16)

GGUF conversion of OpenMed/privacy-filter-multilingual, a multilingual PII token-classification model (a fine-tune of openai/privacy-filter). It labels every token with a BIOES tag over 54 PII categories (217 classes) across 16 languages, so it can be served locally with no Python as the encoder/NER tier of a PII redactor.

For the full model description, label space, evaluation, limitations, and citations, see the source model card — this card only covers the GGUF packaging and how to run it.

---

⚠️ Requires a patched llama.cpp — does NOT run on stock builds

Read this before downloading. This GGUF uses a custom architecture, openai-privacy-filter, that is not (yet) part of upstream llama.cpp. It will fail to load in stock llama.cpp, llama-cpp-python, Ollama, LM Studio, or any other off-the-shelf GGUF runtime. You will see an error like unknown model architecture: 'openai-privacy-filter'.

It runs only on a build that carries the patches LocalAI PR#10160 ships in backend/cpp/llama-cpp/patches/ (applied automatically by the backend's prepare.sh). Those five patches add:

1. TOKEN_CLS pooling (LLAMA_POOLING_TYPE_TOKEN_CLS) — a per-token classification head (a reduced subset of upstream PR #19725, still open);
2. the openai-privacy-filter architecture registration;
3. the HF→GGUF converter for it;
4. the bidirectional banded-attention graph + loader (non-causal symmetric sliding window, attention sinks, interleaved RoPE, exact YaRN truncate=false frequencies);
5. an all-SWA no-cache mask fix required for an encoder that windows every layer.

These are carry-patches against a pinned llama.cpp commit, not upstream features. Until the architecture is upstreamed (it depends on PR #19725 landing first), running this model means using LocalAI's vendored backend. At time of writing the support lives on a LocalAI feature branch — use a LocalAI build that includes the openai-privacy-filter patches.

---

Use with LocalAI

Install from the LocalAI model gallery (the entry sets backend: llama-cpp, embeddings: true, and known_usecases: [token_classify]):

local-ai models install privacy-filter-multilingual

The model is not a chat/completion model — it exposes the gRPC TokenClassify RPC. It is a PII detector: it carries its own detection policy in a pii_detection: block, and other models opt in by listing it in pii.detectors. The gallery entry ships a sensible default policy (mask everything detected; block credentials/financial-secrets/crypto):

# the detector model (this GGUF) — policy lives here
name: privacy-filter-multilingual
backend: llama-cpp
embeddings: true
known_usecases: [token_classify]
pii_detection:
  min_score: 0.5
  default_action: mask          # mask | block | allow
  entity_actions:               # which categories to block vs mask
    PASSWORD: block
    CREDITCARD: block
    CVV: block

# any chat or cloud-proxy model — opt in and reference the detector(s)
name: my-assistant
backend: llama-cpp
pii:
  enabled: true
  detectors:
    - privacy-filter-multilingual

LocalAI runs the model's constrained BIOES Viterbi decode in the backend and returns entity spans with UTF-8 byte offsets; the redactor masks/blocks per the detector's pii_detection policy. Multiple detectors union their hits (strongest action wins).

Load note: the model must be loaded with TOKEN_CLS pooling (it is the GGUF's default — the LocalAI gallery config and embeddings: true handle this). If you drive llama-embedding directly for testing, do not pass --pooling none; that overrides the model default and you get raw hidden states instead of label logits.

Files

File	Precision	Size	Notes
privacy-filter-multilingual-f16.gguf	F16	~2.7 GB	156 tensors; 217 classifier.output_labels; pooling_type = TOKEN_CLS. Validated artifact.

F16 is the validated, shipped precision. Quantized variants are deferred until they can be evaluated with a task metric (span-F1 per language) + KL-vs-F16 — perplexity is meaningless for a classifier, so a naively-quantized GGUF is not published here yet.

Architecture & conversion

gpt-oss-style sparse MoE (8 layers, d_model=640, 128 experts, top-4 routing, ~50M active per token), bidirectional banded attention (symmetric sliding window 128, attention sinks retained), interleaved (GPT-J) RoPE with YaRN (θ=150000, factor 32), o200k (o200k_base) tokenizer, and a 217-way token-classification head (score → cls.output).

The conversion reproduces the HF reference exactly at F16: token-for-token argmax match (12/12 on the parity prompt set), full-logit cosine = 1.0, every layer's residual-stream cosine = 1.0 (relerr ≈ 2e-4, i.e. F16 rounding). The two load-bearing conversion choices — the expert gate_up chunk(2) split and the n_swa = 2·sliding_window window mapping — are both confirmed by that parity. See LocalAI's backend/cpp/llama-cpp/patches/README.md for the full provenance.

Label space

O plus B-/I-/E-/S- for each of 54 categories (1 + 54×4 = 217), spanning identity, contact, address, dates/time, government IDs, financial, crypto, vehicle, digital, and auth entities. The ordered id2label table is embedded in the GGUF (classifier.output_labels). See the source card for the full list.

Limitations & intended use

Identical to the source model: multilingual but uneven (strongest on de/es/fr/it/hi/te/en; weaker on CJK), trained on synthetic AI4Privacy data, not a substitute for legal/compliance review, and not a clinical PHI model. Use it as one tier behind deterministic regex pre-filters and human review.

License

Apache-2.0, inherited from openai/privacy-filter and OpenMed/privacy-filter-multilingual.

Credits & citation

Conversion and runtime support by the LocalAI project. The model itself is by OpenMed, fine-tuned from OpenAI's privacy-filter, on AI4Privacy datasets — please cite all of them (BibTeX in the source card).