smoh commited on
Commit
555c45f
·
verified ·
1 Parent(s): 07ab3c2

Use canonical PII EN 65M name and current installation instructions

Browse files
Files changed (3) hide show
  1. DOWNLOAD.md +57 -16
  2. README.md +6 -4
  3. manifest.json +4 -4
DOWNLOAD.md CHANGED
@@ -1,30 +1,71 @@
1
- # Reproducible install
2
 
3
- After publication under DataFog/datafog-pii, record the immutable 40-character Hub commit SHA. Do not deploy an unpinned main revision. The release tag will be v0.1.0.
4
 
5
  ```sh
6
- hf download DataFog/datafog-pii --revision <40-character-commit-sha> --local-dir ./datafog-pii-release
7
- python3 ./datafog-pii-release/tools/install_bundle.py --snapshot ./datafog-pii-release --destination ./datafog-pii-bundle
8
  ```
9
 
10
- The installer verifies the complete snapshot manifest, checks host OS/architecture, refuses an existing destination, safely extracts the versioned archive, and restores executable mode. Run `datafog-pii-bundle/datafog-pii datafog-pii-bundle` and send one JSON object with a text string per stdin line. Successful inference never requires Python or network access.
11
 
12
- Policy for PR #32:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
 
14
  ```toml
15
- version = 1
16
  [model]
17
- bundle_directory = "/absolute/path/to/datafog-pii-bundle"
18
  ```
19
 
20
- Reproduce actual MCP verification with a clean checkout:
21
 
22
- ```sh
23
- git clone https://github.com/DataFog/datafog-mcp.git
24
- git -C datafog-mcp checkout c2c45ec62728d85b7472131a3df9ccb67afd4a02
25
- cd datafog-mcp
26
- uv sync --frozen --group dev --python 3.12
27
- DATAFOG_TEST_MODEL_BUNDLE=/absolute/path/to/datafog-pii-bundle uv run --frozen pytest tests/test_model_runtime.py tests/test_csv_processing.py --no-cov
28
  ```
29
 
30
- Use the frozen lock, not the GLiNER comparison requirements. Fresh archive transport tests can be run locally before publication; only a real Hub download establishes Hub delivery. The release report distinguishes these statuses.
 
1
+ # Download and install the qualified runtime
2
 
3
+ Use Python 3.12 or newer for installation. Inference runs through the native executable and needs neither Python nor network access. The Hugging Face repository is private; authenticate locally with an account authorized for DataFog.
4
 
5
  ```sh
6
+ python -m pip install 'huggingface_hub==1.27.0'
7
+ hf auth login
8
  ```
9
 
10
+ Select exactly one tested target: `linux-x86_64`, `linux-aarch64`, `macos-arm64`, `macos-x86_64`, or `windows-x86_64`. The release index records the tested OS, exact archive hash, runtime source commit, model revision and MCP compatibility revision. Linux ARM64 qualification on Ubuntu is not a Raspberry Pi OS qualification. Windows qualification on Server 2022 does not establish every desktop Windows version.
11
 
12
+ Save this script as `download_install.py`. The script pins immutable runtime commit `7cadd41e21f59b2a149e31378d073cf6ca057531` from the staging report. It downloads only your selected archive and the versioned installer, forces a fresh download, and verifies the archive and all installed file hashes.
13
+
14
+ ```python
15
+ import argparse
16
+ import json
17
+ import subprocess
18
+ import sys
19
+ from pathlib import Path
20
+ from huggingface_hub import hf_hub_download
21
+
22
+ parser = argparse.ArgumentParser()
23
+ parser.add_argument('--platform', required=True)
24
+ parser.add_argument('--destination', type=Path, required=True)
25
+ args = parser.parse_args()
26
+ revision = '7cadd41e21f59b2a149e31378d073cf6ca057531'
27
+ repo = 'DataFog/pii-en-65m'
28
+ prefix = 'runtimes/v0.2.0'
29
+ downloads = Path('datafog-runtime-download').resolve()
30
+
31
+ def download(name):
32
+ return Path(hf_hub_download(
33
+ repo_id=repo, revision=revision, filename=f'{prefix}/{name}',
34
+ local_dir=downloads, force_download=True,
35
+ ))
36
+
37
+ index = json.loads(download('release-index.json').read_text(encoding='utf-8'))
38
+ record = next(item for item in index['platforms'] if item['platform'] == args.platform)
39
+ archive = download(record['archive']['filename'])
40
+ download('tools/common.py')
41
+ installer = download('tools/install_bundle.py')
42
+ subprocess.run([
43
+ sys.executable, str(installer), '--archive', str(archive),
44
+ '--sha256', record['archive']['sha256'],
45
+ '--destination', str(args.destination.resolve()),
46
+ ], check=True)
47
+ ```
48
+
49
+ Run it with your target and a new absolute destination; the installer refuses an existing directory and a mismatched host architecture.
50
+
51
+ ```sh
52
+ python download_install.py --platform macos-arm64 --destination /absolute/new/datafog-bundle
53
+ ```
54
+
55
+ On Windows, use `--platform windows-x86_64` and a path such as `C:\DataFog\bundle`. Install Microsoft's x64 Visual C++ Redistributable (2019-compatible v14). The installer probes startup and reports a missing runtime dependency. Intel macOS requires 15 or newer; ARM macOS requires 14 or newer.
56
+
57
+ Use datafog-mcp PR #33's compatibility revision `01d7f74e693a7892ab1ed504d0d8f8c1e7ce51d3` (stacked on PR #32). Configure the installed path in MCP policy:
58
 
59
  ```toml
 
60
  [model]
61
+ bundle_directory = "/absolute/new/datafog-bundle"
62
  ```
63
 
64
+ On Windows, TOML literal strings avoid escaping backslashes:
65
 
66
+ ```toml
67
+ [model]
68
+ bundle_directory = 'C:\DataFog\bundle'
 
 
 
69
  ```
70
 
71
+ Keep weights, tokenizer, configuration and calibration together. GLiNER is not part of this download.
README.md CHANGED
@@ -13,9 +13,9 @@ tags:
13
  - experimental
14
  - datafog
15
  ---
16
- # DataFog PII
17
 
18
- DataFog PII is a **65.2M-parameter English token classifier** for locating and redacting seven types of personally identifiable information. It returns typed spans, calibrated span confidence, UTF-8 byte offsets and redacted text through a local JSONL runtime. Inference uses the CPU and requires no network access after installation.
19
 
20
  The frozen **model v0.1.0** achieves **98.32% exact typed-span F1** on 90,018 held-out synthetic documents. **Runtime v0.2.0** has qualified native bundles for five platform environments. The checkpoint and acceptance threshold remain **experimental**; performance on reviewed customer tickets has not been established.
21
 
@@ -53,6 +53,8 @@ The root safetensors are a standard Transformers token-classification checkpoint
53
  | `reproduction/`, `provenance/` | Training/evaluation code, locks, partition/source provenance and original audit |
54
  | `LICENSE`, `NOTICE`, `licenses/`, `LICENSING.md` | Model/source licensing, dataset attribution and upstream dependency notices |
55
 
 
 
56
  Pin model revision `02cc6ca86a11dcb2b328861687770a22a7ea1b76` and runtime revision `7cadd41e21f59b2a149e31378d073cf6ca057531` for reproducible downloads. Original tags `v0.1.0` and `runtime-v0.2.0` remain preserved. The original `bundle/` executable is macOS ARM64-only; use the versioned runtime archives for other platforms. The root manifest covers the original model files and current card evidence; the versioned runtime has its own manifest.
57
 
58
  ## Usage
@@ -74,7 +76,7 @@ import sys
74
  from pathlib import Path
75
  from huggingface_hub import hf_hub_download
76
 
77
- REPO = "DataFog/datafog-pii"
78
  REVISION = "7cadd41e21f59b2a149e31378d073cf6ca057531"
79
  PREFIX = "runtimes/v0.2.0"
80
  TARGET = "macos-arm64" # Select your tested platform below.
@@ -122,7 +124,7 @@ print([(f["label"], f["start"], f["end"]) for f in result["findings"]])
122
  # [('first_name', 8, 12), ('last_name', 13, 18), ('email', 22, 38)]
123
  ```
124
 
125
- The example output was checked against the released runtime. See the [versioned download guide](runtimes/v0.2.0/DOWNLOAD.md) for installation details.
126
 
127
  ### MCP integration
128
 
 
13
  - experimental
14
  - datafog
15
  ---
16
+ # DataFog PII EN 65M
17
 
18
+ DataFog PII EN 65M is a **65.2M-parameter English token classifier** for locating and redacting seven types of personally identifiable information. It returns typed spans, calibrated span confidence, UTF-8 byte offsets and redacted text through a local JSONL runtime. Inference uses the CPU and requires no network access after installation.
19
 
20
  The frozen **model v0.1.0** achieves **98.32% exact typed-span F1** on 90,018 held-out synthetic documents. **Runtime v0.2.0** has qualified native bundles for five platform environments. The checkpoint and acceptance threshold remain **experimental**; performance on reviewed customer tickets has not been established.
21
 
 
53
  | `reproduction/`, `provenance/` | Training/evaluation code, locks, partition/source provenance and original audit |
54
  | `LICENSE`, `NOTICE`, `licenses/`, `LICENSING.md` | Model/source licensing, dataset attribution and upstream dependency notices |
55
 
56
+ Canonical Hub ID: `DataFog/pii-en-65m` (formerly `DataFog/datafog-pii`, which redirects here). Frozen archive/executable filenames retain their original names and hashes. The older [DataFog PII EN 71M](https://huggingface.co/DataFog/pii-en-71m) is a separate 41-entity custom model and is not the MCP runtime described here.
57
+
58
  Pin model revision `02cc6ca86a11dcb2b328861687770a22a7ea1b76` and runtime revision `7cadd41e21f59b2a149e31378d073cf6ca057531` for reproducible downloads. Original tags `v0.1.0` and `runtime-v0.2.0` remain preserved. The original `bundle/` executable is macOS ARM64-only; use the versioned runtime archives for other platforms. The root manifest covers the original model files and current card evidence; the versioned runtime has its own manifest.
59
 
60
  ## Usage
 
76
  from pathlib import Path
77
  from huggingface_hub import hf_hub_download
78
 
79
+ REPO = "DataFog/pii-en-65m"
80
  REVISION = "7cadd41e21f59b2a149e31378d073cf6ca057531"
81
  PREFIX = "runtimes/v0.2.0"
82
  TARGET = "macos-arm64" # Select your tested platform below.
 
124
  # [('first_name', 8, 12), ('last_name', 13, 18), ('email', 22, 38)]
125
  ```
126
 
127
+ The example output was checked against the released runtime. See the [download guide](DOWNLOAD.md) for installation details.
128
 
129
  ### MCP integration
130
 
manifest.json CHANGED
@@ -3,8 +3,8 @@
3
  "version": "0.1.0",
4
  "files": {
5
  "DOWNLOAD.md": {
6
- "sha256": "d9bec8da2b7a4efcb32e7c748fcaad44074fa7b7031303d6f34d56df38deba81",
7
- "bytes": 1536
8
  },
9
  "LICENSE": {
10
  "sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
@@ -23,8 +23,8 @@
23
  "bytes": 2526
24
  },
25
  "README.md": {
26
- "sha256": "80205bcdc3939661cb9151fcdf71c5599635083b09ef62426adf78542e7da46c",
27
- "bytes": 19311
28
  },
29
  "archives/datafog-pii-v0.1.0-macos-arm64.tar.gz": {
30
  "sha256": "2dd5251c1b984011852a46a3f4f30b176decf82f543a691c5a288e0a9f93b7a1",
 
3
  "version": "0.1.0",
4
  "files": {
5
  "DOWNLOAD.md": {
6
+ "sha256": "f31de18000dfd29c2cc448c6383de6d7a05e9624a0f79c21b65bfde0adde19a4",
7
+ "bytes": 3209
8
  },
9
  "LICENSE": {
10
  "sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
 
23
  "bytes": 2526
24
  },
25
  "README.md": {
26
+ "sha256": "ca69541ef1b5e5bc21186e14886e7bdf92d6dd55d7446a68f19fca0e3e62ba9a",
27
+ "bytes": 19624
28
  },
29
  "archives/datafog-pii-v0.1.0-macos-arm64.tar.gz": {
30
  "sha256": "2dd5251c1b984011852a46a3f4f30b176decf82f543a691c5a288e0a9f93b7a1",