Expand model card with measured performance, usage and native qualification

README.md

@@ -13,40 +13,264 @@ tags:
 - experimental
 - datafog
 ---
-# DataFog PII v0.1.0
+# DataFog PII
 
-Experimental English PII token classifier trained by DataFog. This is the September 25, 2026 DistilBERT model used by datafog-mcp PR #32. GLiNER is a separate CI comparison and is neither the backbone nor an artifact in this release.
+DataFog PII is a **65.2M-parameter English token classifier** for locating and redacting seven types of personally identifiable information. It returns typed spans, calibrated span confidence, UTF-8 byte offsets and redacted text through a local JSONL runtime. Inference uses the CPU and requires no network access after installation.
 
-## Checkpoint and training
+The frozen **model v0.1.0** achieves **98.32% exact typed-span F1** on 90,018 held-out synthetic documents. **Runtime v0.2.0** has qualified native bundles for five platform environments. The checkpoint and acceptance threshold remain **experimental**; performance on reviewed customer tickets has not been established.
 
-Release checkpoint: **epoch 2**, selected by lowest validation token cross entropy under the predeclared two-epoch recipe. Epoch 1: 0.003570595682300194; epoch 2: 0.0018816228932983772. The safetensors bytes equal `training/epoch-2/model/model.safetensors`; the frozen ONNX, tokenizer, calibration and native macOS bundle are copied unchanged. No final-test tuning or retraining occurred for release.
+[Model](#model) · [Files](#files) · [Usage](#usage) · [Input-and-output contract](#input-and-output-contract) · [Results](#results) · [Training](#training-and-reproduction) · [Limitations](#limitations) · [License](#license)
 
-Backbone: `distilbert/distilbert-base-cased` at `6ea81172465e8b0ad3fddeed32b986cdcdcffcf0`. NVIDIA Nemotron-PII at `b70ffaf5ff39e079776134c5bf4381f00a9fd1ed`. Seed 42, float32 MPS, batch 16, learning rate 2e-5, weight decay 0.01, two epochs (8,102 seconds). Prepared partitions: train 89,307; validation 514; calibration-fit 501; calibration-select 532; test 90,018 documents. Grouping by UID, exact text and normalized annotated templates prevents evidenced family overlap; semantic annotation noise remains. See `provenance/docs/data.md` and `provenance/artifacts/data-manifest.json` for filtering, immutable source hashes and split provenance.
+## Model
 
-## Labels and inference contract
+| Property | Value |
+|---|---|
+| Backbone | [distilbert/distilbert-base-cased](https://huggingface.co/distilbert/distilbert-base-cased), pinned revision `6ea81172465e8b0ad3fddeed32b986cdcdcffcf0` |
+| Parameters | 65,202,447, counted from the released safetensors tensor shapes |
+| Precision | Float32 weights and ONNX inference |
+| Task | BIO token classification: 15 classes, including `O` |
+| Supported labels | `first_name`, `last_name`, `email`, `phone_number`, `street_address`, `customer_id`, `account_number` |
+| Long-input processing | 512-token windows: 510 content tokens with 42-token overlap |
+| Confidence | Pooled isotonic calibration of exact typed-span correctness |
+| Experimental threshold | `0.6486486486486487`; authoritative value in `bundle/calibration.json` |
+| Native execution | Rust + ONNX Runtime 1.30.0/API27, CPU |
+| Model / runtime versions | Model `0.1.0`; native runtime `0.2.0`; versioned separately |
 
-Seven labels in this exact order: first_name, last_name, email, phone_number, street_address, customer_id, account_number. Class 0 is O; each label has B then I, for 15 classes. Both mappings are preserved in root `config.json` and `label-mappings.json`. Root configuration is the Transformers checkpoint configuration; `bundle/config.json` is the frozen native inference configuration. Do not interchange them.
+The root safetensors are a standard Transformers token-classification checkpoint. The native runtime additionally supplies complete-input windowing, boundary-fragment suppression, overlap resolution, calibration, thresholding and byte-based redaction. A generic Transformers pipeline does not reproduce that full contract. GLiNER is a separate CI comparison and is not this model's backbone or a release artifact.
 
-The released JSONL runtime tokenizes the entire input, constructs 512-token windows (510 content tokens), and overlaps 42 content tokens. Minimum predicted-class token probability forms the raw span score. Boundary-fragment suppression precedes greedy overlap resolution. Half-open offsets count UTF-8 **bytes**, not Python characters. Pooled isotonic calibration uses clipped linear interpolation; the experimental threshold is 0.6486486486486487 (the exact value in `bundle/calibration.json` is authoritative). Calibration fit and threshold selection use separate partitions. Preserve tokenizer, config and calibration together with weights. The runtime returns `{findings, contains_pii, redacted}`; invalid requests and unsupported full-window entities return explicit errors.
+## Files
 
-Transformers can load the root safetensors using `AutoModelForTokenClassification.from_pretrained(path, local_files_only=True)`. A generic token-classification pipeline does **not** reproduce native windowing, calibrated confidence, thresholding, offsets or redaction. Use the bundle for MCP qualification; `reproduction/pii/core.py` preserves the Python reference.
+| File or directory | Purpose |
+|---|---|
+| `model.safetensors`, `config.json` | Frozen selected checkpoint and Transformers label mappings |
+| `tokenizer.json`, `tokenizer_config.json`, `label-mappings.json` | Tokenizer and explicit class/label contract |
+| `bundle/model.onnx`, `bundle/config.json`, `bundle/calibration.json` | Original native-model assets; native configuration differs from root Transformers configuration |
+| `runtimes/v0.2.0/*.tar.gz` | Five platform-specific native bundles, including model assets, executable, matching ORT libraries and notices |
+| `runtimes/v0.2.0/release-index.json` | Exact archive hashes, source/runtime provenance and native qualification reports |
+| `runtimes/v0.2.0/tools/` | Hash-verifying bundle installer and archive helpers |
+| `provenance/artifacts/evaluation/`, `provenance/REPORT.md` | Frozen detection, redaction, calibration and support-fixture results |
+| `card-evidence/` | Original regex comparison summaries and recorded execution environment |
+| `reproduction/`, `provenance/` | Training/evaluation code, locks, partition/source provenance and original audit |
+| `LICENSE`, `NOTICE`, `licenses/`, `LICENSING.md` | Model/source licensing, dataset attribution and upstream dependency notices |
 
-## Evaluation
+Pin model revision `02cc6ca86a11dcb2b328861687770a22a7ea1b76` and runtime revision `7cadd41e21f59b2a149e31378d073cf6ca057531` for reproducible downloads. Original tags `v0.1.0` and `runtime-v0.2.0` remain preserved. The original `bundle/` executable is macOS ARM64-only; use the versioned runtime archives for other platforms. The root manifest covers the original model files and current card evidence; the versioned runtime has its own manifest.
 
-Frozen held-out synthetic evaluation: exact typed-span precision 0.9901, recall 0.9763, F1 0.9832; TP 242,853, FP 2,424, FN 5,886. UTF-8 byte redaction F1 0.9876. Hand-authored support fixtures recovered 14/18 entities (F1 0.8485). These are agreement with synthetic annotations, not verified accuracy on private customer tickets. Full metrics, calibration reliability, limitations and original audit are in `provenance/REPORT.md` and `provenance/artifacts/evaluation/`.
+## Usage
 
-## MCP integration and platform distribution
+### Install a native bundle
 
-PR #32 at `c2c45ec62728d85b7472131a3df9ccb67afd4a02` launches `<bundle>/datafog-pii <bundle>` locally. It maps first_name/last_name to PERSON and street_address to STREET_ADDRESS; it ignores other model categories in favor of Core detectors. No downloads occur during MCP inference. Configure `[model].bundle_directory` after installing the bundle. See `DOWNLOAD.md`.
+Use Python 3.12+ for the download/installer. Native inference itself requires neither Python nor a GPU. While this repository is private, authenticate using `hf auth login` with an account authorized for DataFog.
 
-Only **macOS ARM64** has a qualified executable in this release. The preserved binary uses the bundled ONNX Runtime 1.30.0 dylib, API 27, CPU execution. Require macOS 14.0+ for the packaged dylib (Mach-O LC_BUILD_VERSION minos 14.0). The model and tokenizer are platform-neutral; the native executable/library are not. `PLATFORMS.md` documents candidate builds for macOS Intel, Linux x86_64/AArch64 and Windows x86_64, including Windows adapter requirements. None is advertised as qualified before a native build and actual MCP fixture test. Do not copy the ARM dylib to another platform.
+```sh
+python -m pip install huggingface_hub==1.27.0
+```
 
-The adapter previously timed out on a 100 MB input at a 120-second model deadline. This release does not establish the 100 MB complete-model-scan target, production throughput or real-world privacy guarantees. Finite window overlap cannot cover arbitrary entity lengths; 16 training spans had boundaries the tokenizer cannot express exactly. A clean result only concerns seven supported labels and accepted predictions.
+Select `TARGET` from the platform table below. This example downloads only that archive and its installer, verifies the expected archive and file hashes, and installs to a new directory.
 
-## License and attribution
+```python
+import json
+import subprocess
+import sys
+from pathlib import Path
+from huggingface_hub import hf_hub_download
 
-DataFog additions, trained weights and runtime source: Apache License 2.0; `LICENSE`. DistilBERT backbone and tokenizer: Apache-2.0, retained backbone card. Training dataset: CC-BY-4.0, attribution in `NOTICE` and preserved dataset card. ONNX Runtime: MIT with complete upstream ThirdPartyNotices. Rust dependencies retain their own licenses in `licenses/cargo/`; `licenses/cargo-inventory.json` records all 98 locked packages across targets, a conservative superset rather than a claim that all are linked. See `LICENSING.md` for redistribution details.
+REPO = "DataFog/datafog-pii"
+REVISION = "7cadd41e21f59b2a149e31378d073cf6ca057531"
+PREFIX = "runtimes/v0.2.0"
+TARGET = "macos-arm64"  # Select your tested platform below.
+downloads = Path("datafog-download").resolve()
+bundle = Path("datafog-bundle").resolve()  # Must not already exist.
 
-## Reproduction and versioning
+def download(name):
+    return Path(hf_hub_download(
+        REPO, filename=f"{PREFIX}/{name}", revision=REVISION,
+        local_dir=downloads, force_download=True,
+    ))
 
-`release.json` identifies the version, selection, revisions and weights. `manifest.json` contains SHA-256 and size for every distributed file except itself. `archives/datafog-pii-v0.1.0-macos-arm64.tar.gz` preserves executable mode and includes original bundle checksums. Pin a Hub commit SHA rather than `main` for downloads; tag `v0.1.0` is a convenient release alias after publication. `reproduction/` includes the original scripts and uv.lock; run those from a restored workspace layout following `provenance/README.md`. Original training optimizer states, raw datasets, obsolete caches and diagnostic probe weights are intentionally omitted. Exact inference artifacts are reproducible by checksum; bitwise repeatability of MPS training is not promised.
+index = json.loads(download("release-index.json").read_text(encoding="utf-8"))
+record = next(item for item in index["platforms"] if item["platform"] == TARGET)
+archive = download(record["archive"]["filename"])
+download("tools/common.py")
+installer = download("tools/install_bundle.py")
+subprocess.run([
+    sys.executable, str(installer), "--archive", str(archive),
+    "--sha256", record["archive"]["sha256"], "--destination", str(bundle),
+], check=True)
+```
+
+### Run inference
+
+Send one JSON object per line to `<bundle>/datafog-pii <bundle>` (`datafog-pii.exe` on Windows). The process can serve multiple requests without reloading the model.
+
+```python
+import json
+import os
+import subprocess
+from pathlib import Path
+
+bundle = Path("datafog-bundle").resolve()
+executable = bundle / ("datafog-pii.exe" if os.name == "nt" else "datafog-pii")
+response = subprocess.run(
+    [str(executable), str(bundle)],
+    input=json.dumps({"text": "Contact John Smith at john@example.com."}) + "\n",
+    text=True, encoding="utf-8", capture_output=True, check=True,
+)
+result = json.loads(response.stdout)
+print(result["redacted"])
+# Contact [REDACTED] [REDACTED] at [REDACTED].
+print([(f["label"], f["start"], f["end"]) for f in result["findings"]])
+# [('first_name', 8, 12), ('last_name', 13, 18), ('email', 22, 38)]
+```
+
+The example output was checked against the released runtime. See the [versioned download guide](runtimes/v0.2.0/DOWNLOAD.md) for installation details.
+
+### MCP integration
+
+The model supplies `PERSON` (first/last-name spans) and `STREET_ADDRESS` to the proposed DataFog MCP integration. Other model categories are ignored by that adapter in favor of Core detectors. Configure the installed absolute path in the MCP policy:
+
+```toml
+[model]
+bundle_directory = "/absolute/path/to/datafog-bundle"
+```
+
+Windows TOML paths can use literal strings, for example `bundle_directory = 'C:\DataFog\bundle'`. [MCP PR #33](https://github.com/DataFog/datafog-mcp/pull/33), stacked on #32, supplies Windows `.exe` selection. Qualified production/test compatibility revision: `01d7f74e693a7892ab1ed504d0d8f8c1e7ce51d3`. Qualification below covers that fixture contract; the final packaged-MCP/model release gate remains part of the coordinated release work.
+
+## Input-and-output contract
+
+| Field | Meaning |
+|---|---|
+| Input `text` | Entire input string, processed across overlapping windows |
+| Output `findings` | Accepted spans with `label`, `start`, `end`, `raw_score`, `confidence` |
+| `start`, `end` | Half-open UTF-8 **byte** offsets into the original input; not character indices |
+| `raw_score` | Minimum predicted-class token probability within a span |
+| `confidence` | Pooled calibrated exact-span score, using clipped linear interpolation |
+| `contains_pii` | At least one accepted finding among the seven supported labels |
+| `redacted` | Original text with accepted spans replaced by `[REDACTED]` |
+
+Class 0 is `O`; each supported label has `B` then `I` in the label order above. The complete mappings are in `label-mappings.json`. Keep weights, tokenizer, native configuration and calibration together. Malformed requests and unsupported full-window entities return explicit errors. Finite overlap does not cover arbitrarily long entities.
+
+## Results
+
+### Held-out detection
+
+Evaluation uses **90,018 synthetic Nemotron-PII documents** and **248,739 gold entities**, with the checkpoint and threshold frozen before final-test evaluation. A true positive requires both the **exact label and exact span boundaries**. Precision, recall and F1 are percentages; overall scores are micro-averaged across the seven labels.
+
+| Label | Gold entities | Precision | Recall | F1 |
+|---|---:|---:|---:|---:|
+| Overall | 248,739 | 99.01 | 97.63 | **98.32** |
+| first_name | 74,498 | 99.40 | 98.30 | **98.85** |
+| last_name | 52,864 | 99.31 | 97.96 | **98.63** |
+| email | 49,246 | 98.57 | 97.74 | **98.15** |
+| phone_number | 22,033 | 98.29 | 95.28 | **96.76** |
+| street_address | 15,382 | 98.47 | 96.66 | **97.56** |
+| customer_id | 19,085 | 99.27 | 98.00 | **98.63** |
+| account_number | 15,631 | 98.76 | 96.83 | **97.78** |
+
+Overall: **242,853 TP / 2,424 FP / 5,886 FN**. UTF-8 byte-union redaction precision is **99.66%**, recall **97.89%**, F1 **98.76%**; that coverage metric ignores labels and weights multibyte characters by byte length. There were 63,160 missed gold bytes and 10,061 unnecessary redaction bytes.
+
+Among 16,233 target-clean documents, 81 had an accepted finding. Among 73,785 target-positive documents, 469 had no accepted finding; **3,947** had at least one missed entity, including partial misses. A document-level detection therefore does not imply complete PII coverage. These scores measure agreement with synthetic annotations, not reviewed private-ticket accuracy. [Full test metrics](provenance/artifacts/evaluation/test.json).
+
+### Regex comparison
+
+Both systems were evaluated on the **same 90,018 documents**, restricted to **email and phone**, the two labels supported by the recorded DataFog 4.8.1 regex baseline within this model's label contract. Combined support is 71,279 gold entities. Other labels are excluded from both sides. Scores are percentages; the better F1 in each row is bold.
+
+| Scope | Model precision | Model recall | Model F1 | Regex precision | Regex recall | Regex F1 |
+|---|---:|---:|---:|---:|---:|---:|
+| Email + phone | 98.48 | 96.98 | **97.73** | 71.71 | 93.54 | 81.18 |
+| email | 98.57 | 97.74 | 98.15 | 99.38 | 98.99 | **99.19** |
+| phone_number | 98.29 | 95.28 | **96.76** | 40.81 | 81.36 | 54.35 |
+
+The regex baseline performs better on email F1; the model performs better on phone F1 and the combined scoped score. This is a comparison with the recorded `datafog==4.8.1` release, not a benchmark of today's Core detector or a GLiNER comparison. [Regex metrics](card-evidence/regex-metrics.json), [model scoped metrics](card-evidence/model-email-phone-metrics.json), [baseline version/API provenance](card-evidence/regex-provenance.json).
+
+### Support fixtures
+
+Seven hand-authored synthetic documents contain 18 gold entities and cover Unicode signatures, pasted fields, multiline addresses, repeated entities, clean numeric text and long inputs.
+
+| Evaluation | Precision | Recall | F1 | TP / FP / FN |
+|---|---:|---:|---:|---|
+| Exact typed spans | 93.33% | 77.78% | 84.85% | 14 / 1 / 4 |
+| UTF-8 redaction byte coverage | 100.00% | 64.32% | 78.29% | 128 / 0 / 71 bytes |
+
+These cases expose failures despite the high aggregate held-out score: email recovered **2/4** entities, phone **0/1**, and street address **0/1**. Their small support does not estimate general label accuracy. [Full fixture metrics](provenance/artifacts/evaluation/support-fixtures.json).
+
+### Calibration
+
+Calibration fit used 501 separate documents with 1,220 predicted spans (1,172 exact matches); threshold selection used another 532 documents. The threshold maximizes exact typed-span F1 on calibration-select, with ties favoring recall and then the higher threshold. Test reliability uses all 250,771 predicted spans **before thresholding**, including rejected predictions. Lower Brier and expected calibration error (ECE) are better.
+
+| Score | Brier | ECE |
+|---|---:|---:|
+| Raw token-derived span score | 0.014821 | 0.011747 |
+| Calibrated exact-span score | 0.013717 | 0.007427 |
+
+Reliability correctness means exact label and boundaries. Missed entities are absent from the prediction population, so recall must be read separately. ECE uses ten equal-width bins and depends on binning. Only 48 incorrect spans supported calibration fitting; sparse confidence regions remain uncertain. This is a population-specific experimental score, not a universal probability or a production threshold.
+
+### Runtime performance
+
+These are the **original v0.1.0 macOS ARM64 measurements**, using native Rust + ONNX Runtime CPU with **one inference thread**. The recorded environment is macOS 26.6.2 ARM64; the chip model was not recorded. They are single observations, not p50/p95 benchmarks, and do not measure runtime v0.2.0 on all five platforms.
+
+| Input UTF-8 bytes | Warm end-to-end latency (ms) |
+|---|---:|
+| 115 | 6.005 |
+| 752 | 20.579 |
+| 2,010 | 81.282 |
+| 4,502 | 172.720 |
+
+| Resource measurement | Observed value |
+|---|---:|
+| Cold process launch through first empty-input response | 170.013 ms |
+| Maximum RSS sampled after responses | 473.4 MiB |
+| Original unpacked bundle size | 286.2 MiB |
+
+The four request lengths were selected at observed min/median/p90/max validation byte lengths, **not latency percentiles**. Warm timings include end-to-end request processing. The RSS sample is not an allocator high-water mark. Cross-platform percentile latency, sustained throughput and large-file performance remain unmeasured. [Benchmark artifact](provenance/artifacts/native-benchmark.json), [recorded environment](card-evidence/environment-check.json).
+
+### Native platform qualification
+
+Runtime v0.2.0 preserves the model/configuration/calibration bytes while adding platform library selection and explicit telemetry disabling. Each target passed native unit/boundary tests, installation on a separate fresh runner, seven parity/golden cases and **33 actual MCP runtime/CSV tests with zero failures, errors or skips**.
+
+| Archive target | Actual tested environment | MCP tests | Max native/Python logit difference |
+|---|---|---:|---:|
+| `linux-x86_64` | Linux-6.8.0-1064-azure-x86_64-with-glibc2.35 | 33 passed | 0.0 |
+| `linux-aarch64` | Linux-6.17.0-1022-azure-aarch64-with-glibc2.39 | 33 passed | 0.0 |
+| `macos-arm64` | macOS-14.8.9-arm64-arm-64bit | 33 passed | 0.0 |
+| `macos-x86_64` | macOS-15.7.9-x86_64-i386-64bit | 33 passed | 0.0 |
+| `windows-x86_64` | Windows-2022Server-10.0.20348-SP0 | 33 passed | 0.0 |
+
+All five matched the Python reference logits bit-for-bit on the seven fixtures and matched the original Mac golden boundaries/labels/redaction. This establishes fixture consistency, not general detection quality or speed. [Release index and reports](runtimes/v0.2.0/release-index.json), [successful native qualification](https://github.com/DataFog/DataFog-models/actions/runs/37249503334).
+
+macOS packaging minimums are ARM64 14 and Intel 15; only the named environments were tested. Intel macOS uses a source build of ONNX Runtime commit `f2c39fe2f838cf35ce7da92824f5a5e3ee6e88a7`, since upstream has no 1.30.0 Intel binary. Windows requires Microsoft's x64 Visual C++ Redistributable (2019-compatible v14). Ubuntu ARM64 evidence does not qualify Raspberry Pi OS, and Server 2022 evidence does not qualify every desktop Windows edition. Other OS versions/distributions, Windows ARM64 and GPU execution remain unqualified.
+
+## Training and reproduction
+
+Trained on [NVIDIA Nemotron-PII](https://huggingface.co/datasets/nvidia/Nemotron-PII), pinned revision `b70ffaf5ff39e079776134c5bf4381f00a9fd1ed`. Documents are grouped by UID, exact text and normalized annotated templates to reduce family leakage. Structural filtering and split checks do not eliminate semantic annotation noise or every possible near-duplicate.
+
+| Partition | Documents |
+|---|---:|
+| Training | 89,307 |
+| Validation | 514 |
+| Calibration fit | 501 |
+| Calibration select | 532 |
+| Held-out test | 90,018 |
+
+Training used seed 42, float32 MPS, batch size 16, learning rate 2e-5, weight decay 0.01 and a predeclared two-epoch recipe. Total elapsed time was 8,102 seconds. **Epoch 2** was selected by minimum validation token cross entropy:
+
+| Epoch | Validation token loss | Release checkpoint |
+|---|---:|---|
+| 1 | 0.003570595682300194 | No |
+| 2 | 0.0018816228932983772 | Yes |
+
+No final-test tuning or release-time retraining occurred. [Training history](provenance/artifacts/training/complete.json), [data provenance](provenance/docs/data.md), [partition/source manifest](provenance/artifacts/data-manifest.json). Inference/evaluation code and dependency locks are in `reproduction/`; restore the documented workspace layout following [provenance/README.md](provenance/README.md). Exact inference assets are reproducible by checksum; bitwise MPS training repeatability is not promised. Raw datasets, optimizer states and obsolete diagnostic checkpoints are omitted from this release.
+
+## Limitations
+
+- Evaluation is synthetic; reviewed English customer-support and other real-world domains remain unvalidated. Labels can be semantically wrong or incomplete despite structural filtering.
+- Only seven labels are supported. `contains_pii: false` is not assurance that a document contains no sensitive information or missed supported entities.
+- The small support-fixture evaluation includes misses in email, phone and street address; aggregate test scores should not hide those failures.
+- Tokenization cannot express every character boundary: 16 unrepresentable training-source entity boundaries were recorded. Window overlap cannot cover arbitrary entity lengths.
+- The MCP adapter previously timed out on a 100 MB input at a 120-second model deadline. This release does not establish a 100 MB complete-model-scan target or a production throughput guarantee.
+- Native qualification covers the reported environments and finite fixture contract. Model inference performance and the final packaged-MCP/model release acceptance are separate checks.
+
+## License
+
+DataFog source and trained weights: **Apache-2.0**. DistilBERT backbone/tokenizer: Apache-2.0. Training data: **CC-BY-4.0**; attribution to NVIDIA and Amy Steier, Andre Manoel, Alexa Haushalter and Maarten Van Segbroeck, *Nemotron-PII* (2025), is preserved in `NOTICE` and the dataset card. ONNX Runtime: MIT with complete third-party notices. The Rust inventory preserves licenses for all 98 locked packages, a conservative target superset. The MCP snapshot retains MIT licensing.
+
+See [LICENSE](LICENSE), [NOTICE](NOTICE), [LICENSING.md](LICENSING.md) and `licenses/` for redistribution details. Source and release tooling: [DataFog/DataFog-models](https://github.com/DataFog/DataFog-models).

card-evidence/environment-check.json

@@ -0,0 +1,27 @@
+{
+  "platform": "macOS-26.6.2-arm64-arm-64bit",
+  "python": "3.12.13",
+  "versions": {
+    "torch": "2.14.0",
+    "transformers": "5.17.0",
+    "datasets": "5.0.1",
+    "accelerate": "1.15.0",
+    "safetensors": "0.8.0",
+    "scikit-learn": "1.9.1",
+    "onnx": "1.23.0",
+    "onnxscript": "0.7.2",
+    "onnxruntime": "1.30.0"
+  },
+  "mps_built": true,
+  "mps_available": true,
+  "onnxruntime_providers": [
+    "CoreMLExecutionProvider",
+    "AzureExecutionProvider",
+    "CPUExecutionProvider"
+  ],
+  "mps_smoke_check": {
+    "finite_loss": true,
+    "finite_gradients": true,
+    "weights_changed": true
+  }
+}

card-evidence/model-email-phone-metrics.json

@@ -0,0 +1,267 @@
+{
+  "threshold": 0.6486486486486487,
+  "overall": {
+    "tp": 69126,
+    "fp": 1064,
+    "fn": 2153,
+    "predicted": 70190,
+    "support": 71279,
+    "precision": 0.9848411454623166,
+    "recall": 0.9697947502069333,
+    "f1": 0.9772600357675533
+  },
+  "per_label": {
+    "first_name": {
+      "tp": 0,
+      "fp": 0,
+      "fn": 0,
+      "predicted": 0,
+      "support": 0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    },
+    "last_name": {
+      "tp": 0,
+      "fp": 0,
+      "fn": 0,
+      "predicted": 0,
+      "support": 0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    },
+    "email": {
+      "tp": 48134,
+      "fp": 698,
+      "fn": 1112,
+      "predicted": 48832,
+      "support": 49246,
+      "precision": 0.9857060943643512,
+      "recall": 0.9774194858465662,
+      "f1": 0.9815453006790513
+    },
+    "phone_number": {
+      "tp": 20992,
+      "fp": 366,
+      "fn": 1041,
+      "predicted": 21358,
+      "support": 22033,
+      "precision": 0.9828635640041202,
+      "recall": 0.9527526891480961,
+      "f1": 0.9675739208591644
+    },
+    "street_address": {
+      "tp": 0,
+      "fp": 0,
+      "fn": 0,
+      "predicted": 0,
+      "support": 0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    },
+    "customer_id": {
+      "tp": 0,
+      "fp": 0,
+      "fn": 0,
+      "predicted": 0,
+      "support": 0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    },
+    "account_number": {
+      "tp": 0,
+      "fp": 0,
+      "fn": 0,
+      "predicted": 0,
+      "support": 0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    }
+  },
+  "redaction_byte_coverage": {
+    "tp": 1533224,
+    "fp": 5298,
+    "fn": 32976,
+    "predicted": 1538522,
+    "support": 1566200,
+    "precision": 0.9965564353320915,
+    "recall": 0.9789452177244286,
+    "f1": 0.9876723262179351
+  },
+  "documents": {
+    "total": 90018,
+    "positive_support": 43658,
+    "clean_support": 46360,
+    "false_positive": 119,
+    "false_negative": 424,
+    "with_any_missed_entity": 1717,
+    "with_any_spurious_entity": 784
+  },
+  "raw_reliability": {
+    "support": 72438,
+    "brier": 0.021829755183972725,
+    "ece": 0.018571813107282845,
+    "binning": "10 equal-width bins; left-inclusive, right-exclusive except final bin includes 1",
+    "bins": [
+      {
+        "lower": 0.0,
+        "upper": 0.1,
+        "support": 0,
+        "mean_score": null,
+        "accuracy": null
+      },
+      {
+        "lower": 0.1,
+        "upper": 0.2,
+        "support": 0,
+        "mean_score": null,
+        "accuracy": null
+      },
+      {
+        "lower": 0.2,
+        "upper": 0.3,
+        "support": 0,
+        "mean_score": null,
+        "accuracy": null
+      },
+      {
+        "lower": 0.3,
+        "upper": 0.4,
+        "support": 3,
+        "mean_score": 0.3863314212266449,
+        "accuracy": 0.0
+      },
+      {
+        "lower": 0.4,
+        "upper": 0.5,
+        "support": 95,
+        "mean_score": 0.4782374044011286,
+        "accuracy": 0.14736842105263157
+      },
+      {
+        "lower": 0.5,
+        "upper": 0.6,
+        "support": 1049,
+        "mean_score": 0.5422019699381606,
+        "accuracy": 0.21067683508102955
+      },
+      {
+        "lower": 0.6,
+        "upper": 0.7,
+        "support": 618,
+        "mean_score": 0.6473821870437928,
+        "accuracy": 0.39158576051779936
+      },
+      {
+        "lower": 0.7,
+        "upper": 0.8,
+        "support": 641,
+        "mean_score": 0.751043175894807,
+        "accuracy": 0.5647425897035881
+      },
+      {
+        "lower": 0.8,
+        "upper": 0.9,
+        "support": 890,
+        "mean_score": 0.8564894739866963,
+        "accuracy": 0.7
+      },
+      {
+        "lower": 0.9,
+        "upper": 1.0,
+        "support": 69142,
+        "mean_score": 0.9973556895161934,
+        "accuracy": 0.989427554887044
+      }
+    ]
+  },
+  "confidence_reliability": {
+    "support": 72438,
+    "brier": 0.019584471057273317,
+    "ece": 0.007393019371353872,
+    "binning": "10 equal-width bins; left-inclusive, right-exclusive except final bin includes 1",
+    "bins": [
+      {
+        "lower": 0.0,
+        "upper": 0.1,
+        "support": 455,
+        "mean_score": 0.0007816997886464416,
+        "accuracy": 0.15164835164835164
+      },
+      {
+        "lower": 0.1,
+        "upper": 0.2,
+        "support": 9,
+        "mean_score": 0.1469554986102011,
+        "accuracy": 0.1111111111111111
+      },
+      {
+        "lower": 0.2,
+        "upper": 0.3,
+        "support": 317,
+        "mean_score": 0.22929724681409186,
+        "accuracy": 0.2302839116719243
+      },
+      {
+        "lower": 0.3,
+        "upper": 0.4,
+        "support": 111,
+        "mean_score": 0.32947968452441123,
+        "accuracy": 0.26126126126126126
+      },
+      {
+        "lower": 0.4,
+        "upper": 0.5,
+        "support": 31,
+        "mean_score": 0.45223669021161506,
+        "accuracy": 0.25806451612903225
+      },
+      {
+        "lower": 0.5,
+        "upper": 0.6,
+        "support": 1311,
+        "mean_score": 0.5399227828147602,
+        "accuracy": 0.4248665141113654
+      },
+      {
+        "lower": 0.6,
+        "upper": 0.7,
+        "support": 2174,
+        "mean_score": 0.6484848726369532,
+        "accuracy": 0.7465501379944802
+      },
+      {
+        "lower": 0.7,
+        "upper": 0.8,
+        "support": 3,
+        "mean_score": 0.7473658018005699,
+        "accuracy": 1.0
+      },
+      {
+        "lower": 0.8,
+        "upper": 0.9,
+        "support": 5,
+        "mean_score": 0.8363934597936151,
+        "accuracy": 0.8
+      },
+      {
+        "lower": 0.9,
+        "upper": 1.0,
+        "support": 68022,
+        "mean_score": 0.9937033861243243,
+        "accuracy": 0.9924142189291699
+      }
+    ]
+  },
+  "reliability_population": "All predicted spans before threshold, correctness = exact typed span. Missed gold entities excluded; see recall.",
+  "confidence_semantics": "Pooled calibrated exact-span score",
+  "coverage_semantics": "Union of UTF-8 bytes; ignores labels. Zero-denominator P/R/F1 reported as 0.",
+  "comparison_scope": [
+    "email",
+    "phone_number"
+  ]
+}

card-evidence/regex-metrics.json

@@ -0,0 +1,117 @@
+{
+  "threshold": null,
+  "overall": {
+    "tp": 66676,
+    "fp": 26307,
+    "fn": 4603,
+    "predicted": 92983,
+    "support": 71279,
+    "precision": 0.7170773152081563,
+    "recall": 0.9354227752914603,
+    "f1": 0.8118250112624953
+  },
+  "per_label": {
+    "first_name": {
+      "tp": 0,
+      "fp": 0,
+      "fn": 0,
+      "predicted": 0,
+      "support": 0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    },
+    "last_name": {
+      "tp": 0,
+      "fp": 0,
+      "fn": 0,
+      "predicted": 0,
+      "support": 0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    },
+    "email": {
+      "tp": 48751,
+      "fp": 306,
+      "fn": 495,
+      "predicted": 49057,
+      "support": 49246,
+      "precision": 0.9937623580732617,
+      "recall": 0.9899484222068797,
+      "f1": 0.9918517237520726
+    },
+    "phone_number": {
+      "tp": 17925,
+      "fp": 26001,
+      "fn": 4108,
+      "predicted": 43926,
+      "support": 22033,
+      "precision": 0.4080726676683513,
+      "recall": 0.8135523986747152,
+      "f1": 0.5435194590578996
+    },
+    "street_address": {
+      "tp": 0,
+      "fp": 0,
+      "fn": 0,
+      "predicted": 0,
+      "support": 0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    },
+    "customer_id": {
+      "tp": 0,
+      "fp": 0,
+      "fn": 0,
+      "predicted": 0,
+      "support": 0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    },
+    "account_number": {
+      "tp": 0,
+      "fp": 0,
+      "fn": 0,
+      "predicted": 0,
+      "support": 0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    }
+  },
+  "redaction_byte_coverage": {
+    "tp": 1513432,
+    "fp": 282773,
+    "fn": 52768,
+    "predicted": 1796205,
+    "support": 1566200,
+    "precision": 0.8425719781428066,
+    "recall": 0.9663082620354999,
+    "f1": 0.9002080356173632
+  },
+  "documents": {
+    "total": 90018,
+    "positive_support": 43658,
+    "clean_support": 46360,
+    "false_positive": 11261,
+    "false_negative": 344,
+    "with_any_missed_entity": 3806,
+    "with_any_spurious_entity": 20923
+  },
+  "coverage_semantics": "Union of UTF-8 bytes; ignores labels. Zero-denominator P/R/F1 reported as 0.",
+  "score_semantics": "Regex match indicator, no probability calibration evaluated",
+  "comparison_scope": [
+    "email",
+    "phone_number"
+  ],
+  "unsupported_labels": [
+    "first_name",
+    "last_name",
+    "street_address",
+    "customer_id",
+    "account_number"
+  ]
+}

card-evidence/regex-provenance.json

@@ -0,0 +1,21 @@
+{
+  "package": "datafog",
+  "version": "4.8.1",
+  "official_repository": "https://github.com/DataFog/datafog-python",
+  "release": "https://pypi.org/project/datafog/4.8.1/",
+  "input_sha256": "f7737b588c95c0910a58374a9bee655cb5d4f080da795ed0d74a676da6d9d128",
+  "documents": 90018,
+  "elapsed_seconds": 16.65799858304672,
+  "API": "datafog.scan(text, engine='regex', entity_types=['EMAIL','PHONE'])",
+  "offset_conversion": "Python character offsets converted to original UTF-8 byte offsets",
+  "dependencies": {
+    "pydantic_core": "2.46.5",
+    "pydantic": "2.13.5",
+    "pydantic-settings": "2.15.0",
+    "datafog": "4.8.1",
+    "annotated-types": "0.8.0",
+    "typing-inspection": "0.4.4",
+    "python-dotenv": "1.2.3",
+    "typing_extensions": "4.16.0"
+  }
+}

manifest.json

@@ -23,8 +23,8 @@
       "bytes": 2526
     },
     "README.md": {
-      "sha256": "4569cdcf29a50391127f5889a6b67296583bb7a9c4f149662af0ccee49bcd421",
-      "bytes": 6451
+      "sha256": "80205bcdc3939661cb9151fcdf71c5599635083b09ef62426adf78542e7da46c",
+      "bytes": 19311
     },
     "archives/datafog-pii-v0.1.0-macos-arm64.tar.gz": {
       "sha256": "2dd5251c1b984011852a46a3f4f30b176decf82f543a691c5a288e0a9f93b7a1",
@@ -1957,6 +1957,22 @@
     "validation.json": {
       "sha256": "98ef0cd2b71c95a57a5bb054abe41f6165cc8cdeab05145bc984b445001013da",
       "bytes": 464
+    },
+    "card-evidence/environment-check.json": {
+      "sha256": "7ce07dba86189bebbd023c42e407ba05a9872dbf92c36a28e5788cfcb3cb6177",
+      "bytes": 612
+    },
+    "card-evidence/regex-provenance.json": {
+      "sha256": "c0e0a3eabbfb8aaed91c4386d2eaca4d44c2fb553b4d18b79ac3f194063b27a3",
+      "bytes": 765
+    },
+    "card-evidence/regex-metrics.json": {
+      "sha256": "ff38e4cee40469881d484d217ce6a60ab6c56349819ec7fd28ab4c26715ce243",
+      "bytes": 2464
+    },
+    "card-evidence/model-email-phone-metrics.json": {
+      "sha256": "98d75034af1a0de4eee52f7b6d98c703a0de22901ced0e7b38c99454e4f4b3c2",
+      "bytes": 6122
     }
   }
 }