Token Classification
Transformers
ONNX
Safetensors
English
distilbert
pii
privacy
experimental
datafog
Instructions to use DataFog/pii-en-65m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use DataFog/pii-en-65m with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="DataFog/pii-en-65m")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("DataFog/pii-en-65m") model = AutoModelForTokenClassification.from_pretrained("DataFog/pii-en-65m", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Use canonical PII EN 65M name and current installation instructions
Browse files- DOWNLOAD.md +57 -16
- README.md +6 -4
- manifest.json +4 -4
DOWNLOAD.md
CHANGED
|
@@ -1,30 +1,71 @@
|
|
| 1 |
-
#
|
| 2 |
|
| 3 |
-
|
| 4 |
|
| 5 |
```sh
|
| 6 |
-
|
| 7 |
-
|
| 8 |
```
|
| 9 |
|
| 10 |
-
|
| 11 |
|
| 12 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
```toml
|
| 15 |
-
version = 1
|
| 16 |
[model]
|
| 17 |
-
bundle_directory = "/absolute/
|
| 18 |
```
|
| 19 |
|
| 20 |
-
|
| 21 |
|
| 22 |
-
```
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
cd datafog-mcp
|
| 26 |
-
uv sync --frozen --group dev --python 3.12
|
| 27 |
-
DATAFOG_TEST_MODEL_BUNDLE=/absolute/path/to/datafog-pii-bundle uv run --frozen pytest tests/test_model_runtime.py tests/test_csv_processing.py --no-cov
|
| 28 |
```
|
| 29 |
|
| 30 |
-
|
|
|
|
| 1 |
+
# Download and install the qualified runtime
|
| 2 |
|
| 3 |
+
Use Python 3.12 or newer for installation. Inference runs through the native executable and needs neither Python nor network access. The Hugging Face repository is private; authenticate locally with an account authorized for DataFog.
|
| 4 |
|
| 5 |
```sh
|
| 6 |
+
python -m pip install 'huggingface_hub==1.27.0'
|
| 7 |
+
hf auth login
|
| 8 |
```
|
| 9 |
|
| 10 |
+
Select exactly one tested target: `linux-x86_64`, `linux-aarch64`, `macos-arm64`, `macos-x86_64`, or `windows-x86_64`. The release index records the tested OS, exact archive hash, runtime source commit, model revision and MCP compatibility revision. Linux ARM64 qualification on Ubuntu is not a Raspberry Pi OS qualification. Windows qualification on Server 2022 does not establish every desktop Windows version.
|
| 11 |
|
| 12 |
+
Save this script as `download_install.py`. The script pins immutable runtime commit `7cadd41e21f59b2a149e31378d073cf6ca057531` from the staging report. It downloads only your selected archive and the versioned installer, forces a fresh download, and verifies the archive and all installed file hashes.
|
| 13 |
+
|
| 14 |
+
```python
|
| 15 |
+
import argparse
|
| 16 |
+
import json
|
| 17 |
+
import subprocess
|
| 18 |
+
import sys
|
| 19 |
+
from pathlib import Path
|
| 20 |
+
from huggingface_hub import hf_hub_download
|
| 21 |
+
|
| 22 |
+
parser = argparse.ArgumentParser()
|
| 23 |
+
parser.add_argument('--platform', required=True)
|
| 24 |
+
parser.add_argument('--destination', type=Path, required=True)
|
| 25 |
+
args = parser.parse_args()
|
| 26 |
+
revision = '7cadd41e21f59b2a149e31378d073cf6ca057531'
|
| 27 |
+
repo = 'DataFog/pii-en-65m'
|
| 28 |
+
prefix = 'runtimes/v0.2.0'
|
| 29 |
+
downloads = Path('datafog-runtime-download').resolve()
|
| 30 |
+
|
| 31 |
+
def download(name):
|
| 32 |
+
return Path(hf_hub_download(
|
| 33 |
+
repo_id=repo, revision=revision, filename=f'{prefix}/{name}',
|
| 34 |
+
local_dir=downloads, force_download=True,
|
| 35 |
+
))
|
| 36 |
+
|
| 37 |
+
index = json.loads(download('release-index.json').read_text(encoding='utf-8'))
|
| 38 |
+
record = next(item for item in index['platforms'] if item['platform'] == args.platform)
|
| 39 |
+
archive = download(record['archive']['filename'])
|
| 40 |
+
download('tools/common.py')
|
| 41 |
+
installer = download('tools/install_bundle.py')
|
| 42 |
+
subprocess.run([
|
| 43 |
+
sys.executable, str(installer), '--archive', str(archive),
|
| 44 |
+
'--sha256', record['archive']['sha256'],
|
| 45 |
+
'--destination', str(args.destination.resolve()),
|
| 46 |
+
], check=True)
|
| 47 |
+
```
|
| 48 |
+
|
| 49 |
+
Run it with your target and a new absolute destination; the installer refuses an existing directory and a mismatched host architecture.
|
| 50 |
+
|
| 51 |
+
```sh
|
| 52 |
+
python download_install.py --platform macos-arm64 --destination /absolute/new/datafog-bundle
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
On Windows, use `--platform windows-x86_64` and a path such as `C:\DataFog\bundle`. Install Microsoft's x64 Visual C++ Redistributable (2019-compatible v14). The installer probes startup and reports a missing runtime dependency. Intel macOS requires 15 or newer; ARM macOS requires 14 or newer.
|
| 56 |
+
|
| 57 |
+
Use datafog-mcp PR #33's compatibility revision `01d7f74e693a7892ab1ed504d0d8f8c1e7ce51d3` (stacked on PR #32). Configure the installed path in MCP policy:
|
| 58 |
|
| 59 |
```toml
|
|
|
|
| 60 |
[model]
|
| 61 |
+
bundle_directory = "/absolute/new/datafog-bundle"
|
| 62 |
```
|
| 63 |
|
| 64 |
+
On Windows, TOML literal strings avoid escaping backslashes:
|
| 65 |
|
| 66 |
+
```toml
|
| 67 |
+
[model]
|
| 68 |
+
bundle_directory = 'C:\DataFog\bundle'
|
|
|
|
|
|
|
|
|
|
| 69 |
```
|
| 70 |
|
| 71 |
+
Keep weights, tokenizer, configuration and calibration together. GLiNER is not part of this download.
|
README.md
CHANGED
|
@@ -13,9 +13,9 @@ tags:
|
|
| 13 |
- experimental
|
| 14 |
- datafog
|
| 15 |
---
|
| 16 |
-
# DataFog PII
|
| 17 |
|
| 18 |
-
DataFog PII is a **65.2M-parameter English token classifier** for locating and redacting seven types of personally identifiable information. It returns typed spans, calibrated span confidence, UTF-8 byte offsets and redacted text through a local JSONL runtime. Inference uses the CPU and requires no network access after installation.
|
| 19 |
|
| 20 |
The frozen **model v0.1.0** achieves **98.32% exact typed-span F1** on 90,018 held-out synthetic documents. **Runtime v0.2.0** has qualified native bundles for five platform environments. The checkpoint and acceptance threshold remain **experimental**; performance on reviewed customer tickets has not been established.
|
| 21 |
|
|
@@ -53,6 +53,8 @@ The root safetensors are a standard Transformers token-classification checkpoint
|
|
| 53 |
| `reproduction/`, `provenance/` | Training/evaluation code, locks, partition/source provenance and original audit |
|
| 54 |
| `LICENSE`, `NOTICE`, `licenses/`, `LICENSING.md` | Model/source licensing, dataset attribution and upstream dependency notices |
|
| 55 |
|
|
|
|
|
|
|
| 56 |
Pin model revision `02cc6ca86a11dcb2b328861687770a22a7ea1b76` and runtime revision `7cadd41e21f59b2a149e31378d073cf6ca057531` for reproducible downloads. Original tags `v0.1.0` and `runtime-v0.2.0` remain preserved. The original `bundle/` executable is macOS ARM64-only; use the versioned runtime archives for other platforms. The root manifest covers the original model files and current card evidence; the versioned runtime has its own manifest.
|
| 57 |
|
| 58 |
## Usage
|
|
@@ -74,7 +76,7 @@ import sys
|
|
| 74 |
from pathlib import Path
|
| 75 |
from huggingface_hub import hf_hub_download
|
| 76 |
|
| 77 |
-
REPO = "DataFog/
|
| 78 |
REVISION = "7cadd41e21f59b2a149e31378d073cf6ca057531"
|
| 79 |
PREFIX = "runtimes/v0.2.0"
|
| 80 |
TARGET = "macos-arm64" # Select your tested platform below.
|
|
@@ -122,7 +124,7 @@ print([(f["label"], f["start"], f["end"]) for f in result["findings"]])
|
|
| 122 |
# [('first_name', 8, 12), ('last_name', 13, 18), ('email', 22, 38)]
|
| 123 |
```
|
| 124 |
|
| 125 |
-
The example output was checked against the released runtime. See the [
|
| 126 |
|
| 127 |
### MCP integration
|
| 128 |
|
|
|
|
| 13 |
- experimental
|
| 14 |
- datafog
|
| 15 |
---
|
| 16 |
+
# DataFog PII EN 65M
|
| 17 |
|
| 18 |
+
DataFog PII EN 65M is a **65.2M-parameter English token classifier** for locating and redacting seven types of personally identifiable information. It returns typed spans, calibrated span confidence, UTF-8 byte offsets and redacted text through a local JSONL runtime. Inference uses the CPU and requires no network access after installation.
|
| 19 |
|
| 20 |
The frozen **model v0.1.0** achieves **98.32% exact typed-span F1** on 90,018 held-out synthetic documents. **Runtime v0.2.0** has qualified native bundles for five platform environments. The checkpoint and acceptance threshold remain **experimental**; performance on reviewed customer tickets has not been established.
|
| 21 |
|
|
|
|
| 53 |
| `reproduction/`, `provenance/` | Training/evaluation code, locks, partition/source provenance and original audit |
|
| 54 |
| `LICENSE`, `NOTICE`, `licenses/`, `LICENSING.md` | Model/source licensing, dataset attribution and upstream dependency notices |
|
| 55 |
|
| 56 |
+
Canonical Hub ID: `DataFog/pii-en-65m` (formerly `DataFog/datafog-pii`, which redirects here). Frozen archive/executable filenames retain their original names and hashes. The older [DataFog PII EN 71M](https://huggingface.co/DataFog/pii-en-71m) is a separate 41-entity custom model and is not the MCP runtime described here.
|
| 57 |
+
|
| 58 |
Pin model revision `02cc6ca86a11dcb2b328861687770a22a7ea1b76` and runtime revision `7cadd41e21f59b2a149e31378d073cf6ca057531` for reproducible downloads. Original tags `v0.1.0` and `runtime-v0.2.0` remain preserved. The original `bundle/` executable is macOS ARM64-only; use the versioned runtime archives for other platforms. The root manifest covers the original model files and current card evidence; the versioned runtime has its own manifest.
|
| 59 |
|
| 60 |
## Usage
|
|
|
|
| 76 |
from pathlib import Path
|
| 77 |
from huggingface_hub import hf_hub_download
|
| 78 |
|
| 79 |
+
REPO = "DataFog/pii-en-65m"
|
| 80 |
REVISION = "7cadd41e21f59b2a149e31378d073cf6ca057531"
|
| 81 |
PREFIX = "runtimes/v0.2.0"
|
| 82 |
TARGET = "macos-arm64" # Select your tested platform below.
|
|
|
|
| 124 |
# [('first_name', 8, 12), ('last_name', 13, 18), ('email', 22, 38)]
|
| 125 |
```
|
| 126 |
|
| 127 |
+
The example output was checked against the released runtime. See the [download guide](DOWNLOAD.md) for installation details.
|
| 128 |
|
| 129 |
### MCP integration
|
| 130 |
|
manifest.json
CHANGED
|
@@ -3,8 +3,8 @@
|
|
| 3 |
"version": "0.1.0",
|
| 4 |
"files": {
|
| 5 |
"DOWNLOAD.md": {
|
| 6 |
-
"sha256": "
|
| 7 |
-
"bytes":
|
| 8 |
},
|
| 9 |
"LICENSE": {
|
| 10 |
"sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
|
|
@@ -23,8 +23,8 @@
|
|
| 23 |
"bytes": 2526
|
| 24 |
},
|
| 25 |
"README.md": {
|
| 26 |
-
"sha256": "
|
| 27 |
-
"bytes":
|
| 28 |
},
|
| 29 |
"archives/datafog-pii-v0.1.0-macos-arm64.tar.gz": {
|
| 30 |
"sha256": "2dd5251c1b984011852a46a3f4f30b176decf82f543a691c5a288e0a9f93b7a1",
|
|
|
|
| 3 |
"version": "0.1.0",
|
| 4 |
"files": {
|
| 5 |
"DOWNLOAD.md": {
|
| 6 |
+
"sha256": "f31de18000dfd29c2cc448c6383de6d7a05e9624a0f79c21b65bfde0adde19a4",
|
| 7 |
+
"bytes": 3209
|
| 8 |
},
|
| 9 |
"LICENSE": {
|
| 10 |
"sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
|
|
|
|
| 23 |
"bytes": 2526
|
| 24 |
},
|
| 25 |
"README.md": {
|
| 26 |
+
"sha256": "ca69541ef1b5e5bc21186e14886e7bdf92d6dd55d7446a68f19fca0e3e62ba9a",
|
| 27 |
+
"bytes": 19624
|
| 28 |
},
|
| 29 |
"archives/datafog-pii-v0.1.0-macos-arm64.tar.gz": {
|
| 30 |
"sha256": "2dd5251c1b984011852a46a3f4f30b176decf82f543a691c5a288e0a9f93b7a1",
|