privacy-filter-nemotron-v2-GGUF

GGUF conversion of OpenMed/privacy-filter-nemotron-v2 for the privacy-filter.cpp runtime, with the Mixture-of-Experts weights requantized to Q8_0.

This is the Nemotron label schema checkpoint: 221 BIOES classes over 55 fine-grained PII categories. Its sibling, Owlchemy/privacy-filter-multilingual-v2-GGUF, uses the multilingual schema with 217 classes.

Files

file	bytes	sha256
pf-nemotron-v2-q8.gguf	1,637,817,216	1ffcd80d23cf99d2e21a2a790843d4f9f42266315505461de7a6135ef7797ac5

How it was produced

Converted from OpenMed/privacy-filter-nemotron-v2 at revision 968247329d18998cc7d5338b941b52f1b2a9abd9, using the conversion scripts from privacy-filter.cpp at commit 735a6c28607ee82afc3a670383f41b55266a3b9a.

Step 1 - Hugging Face checkpoint to f16 GGUF, with scripts/convert.py:

hf download OpenMed/privacy-filter-nemotron-v2 --local-dir pf-src

python scripts/convert.py \
    --model pf-src \
    --outfile pf-nemotron-v2-f16.gguf \
    --outtype f16

convert.py is self-contained: it reads config.json, model.safetensors and tokenizer.json directly and does not depend on llama.cpp or its conversion script. The f16 output is 2,817,465,120 bytes.

Step 2 - requantize the expert weights to Q8_0, with scripts/requant_q8.py:

python scripts/requant_q8.py \
    --in pf-nemotron-v2-f16.gguf \
    --out pf-nemotron-v2-q8.gguf

Environment. torch (CPU build is sufficient), safetensors, and gguf>=0.10, as pinned by scripts/requirements.txt. Roughly 30 seconds for the conversion and 60 seconds for the requant on a CPU-only machine.

The upstream project's own ci.yml converts checkpoints to f16 and f32 on its nightly tier and gates them against reference fixtures; it does not run the Q8_0 requant, so step 2 above is the part that is not covered by upstream CI.

Architecture

The checkpoint is the openai/privacy-filter body (a gpt-oss style MoE) fine-tuned on the Nemotron PII mix and repurposed as a bidirectional token classifier. As recorded in the GGUF metadata:

key	value
general.architecture	openai-privacy-filter
block_count	8
embedding_length	640
attention.head_count / head_count_kv	14 / 2
expert_count / expert_used_count	128 / 4
context_length	131072
tokenizer.ggml.model / pre	gpt2 / gpt-4o
classifier.output_labels	221 BIOES classes

Notes

• The requant is partial. Only tensors matching ffn_gate_exps.weight, ffn_up_exps.weight and ffn_down_exps.weight (the MoE expert weights) become Q8_0. Every other tensor is copied verbatim and stays f16. This is what makes the file 1.64 GB rather than the 2.8 GB f16 intermediate, and it is the configuration the int8 mul_mat_id kernel is fastest on.
• general.file_type still reports f16 (value 1). requant_q8.py copies metadata fields verbatim, so the declared file type is not updated to reflect the Q8_0 expert tensors.
• gguf-py cannot reopen these files. The reader raises KeyError: Duplicate GGUF.version already in list. The cause is not a duplicate in the file: requant_q8.py copies the reader's synthetic fields into the output as real metadata, so the file carries literal GGUF.version, GGUF.tensor_count and GGUF.kv_count entries that collide with the ones gguf-py injects when reading. llama.cpp and privacy-filter.cpp both load the files normally. The published LocalAI-io q8 files share this quirk.
• The label schema differs from the multilingual checkpoint. Class names are snake_case here (B-account_number) where the multilingual checkpoint uses uppercase concatenated names (B-ACCOUNTNAME), and the two have different class counts. Do not swap one for the other without re-checking the label mapping.