DataFog PII prototype evaluation
This is an evaluated prototype, not a production-readiness claim. All numbers below come from saved artifacts; absent evidence is explicitly marked unavailable.
Required evaluation artifacts are present.
| Contract criterion | Status | Evidence | Interpretation / limitation |
|---|---|---|---|
| Pinned data and disjoint document families | Evidence present | artifacts/data-manifest.json | Structural validation and split rules are saved; annotation semantics remain imperfect. |
| Trained checkpoint and resumability | Evidence present | artifacts/training/complete.json, artifacts/training-check.json | Completion plus separate optimizer/RNG resume diagnostic; inspect reported MPS parameter differences. |
| Native Rust and Python parity | Evidence present | artifacts/bundle/rust-parity.json | Checks tokenizer/global byte offsets, outputs/redaction, and explicit error recovery; parity does not prove quality. |
| Complete-input boundary behavior | Evidence present | artifacts/boundary-proof.json | Manual global-offset window construction; finite fixture evidence and representability limits apply. |
| Independent calibration and experimental threshold | Evidence present | artifacts/bundle/calibration.json, artifacts/evaluation/test.json | Fit/select partitions are separate; final reliability is reported on held-out predicted spans. |
| Held-out quality and support mechanics | Evidence present | artifacts/evaluation/test.json, artifacts/evaluation/support-fixtures.json | Synthetic population only; no reviewed private tickets. |
| Existing regex comparison | Evidence present | artifacts/regex-test/regex-metrics.json, artifacts/regex-test/model-email-phone-metrics.json | Official published DataFog 4.8.1; email and phone only. |
| Native resource measurements | Evidence present | artifacts/native-benchmark.json | Local observed load, warm request latency, memory, and bundle size; no performance target assumed. |
Data and execution
Dataset: nvidia/Nemotron-PII revision b70ffaf5ff39e079776134c5bf4381f00a9fd1ed. License: CC-BY-4.0. NVIDIA; Amy Steier, Andre Manoel, Alexa Haushalter, Maarten Van Segbroeck, Nemotron-PII (2025). https://huggingface.co/datasets/nvidia/Nemotron-PII
| Partition | Documents | Target-clean documents |
|---|---|---|
| train | 89307 | 23563 |
| validation | 514 | 104 |
| calibration_fit | 501 | 107 |
| calibration_select | 532 | 105 |
| test | 90018 | 16233 |
Structural exclusions: source_slice_mismatch: 19,101, duplicate_or_overlap: 20, test_family_overlap: 7.
| Epoch | Train mean batch loss | Validation token loss | Selected at epoch | Elapsed seconds |
|---|---|---|---|---|
| 1 | 0.0135 | 0.0036 | True | 4,166.3 |
| 2 | 0.0035 | 0.0019 | True | 8,102.2 |
Training elapsed: 8,102.2 seconds. Best validation token loss: 0.0019.
Corrective iterations: two of the three-round fuse used. Round 1 fixed tokenizer overflow by manually constructing complete-input windows; round 2 kept the ONNX export wrapper and Python reference in evaluation mode. Neither correction tuned against final test outcomes. Further corrective work must remain within the agreed fuse.
Held-out detection
| Label | Gold entities | Predicted | TP | FP | FN | Precision | Recall | F1 |
|---|---|---|---|---|---|---|---|---|
| overall | 248739 | 245277 | 242853 | 2424 | 5886 | 0.9901 | 0.9763 | 0.9832 |
| first_name | 74498 | 73673 | 73234 | 439 | 1264 | 0.9940 | 0.9830 | 0.9885 |
| last_name | 52864 | 52148 | 51786 | 362 | 1078 | 0.9931 | 0.9796 | 0.9863 |
| 49246 | 48832 | 48134 | 698 | 1112 | 0.9857 | 0.9774 | 0.9815 | |
| phone_number | 22033 | 21358 | 20992 | 366 | 1041 | 0.9829 | 0.9528 | 0.9676 |
| street_address | 15382 | 15099 | 14868 | 231 | 514 | 0.9847 | 0.9666 | 0.9756 |
| customer_id | 19085 | 18842 | 18704 | 138 | 381 | 0.9927 | 0.9800 | 0.9863 |
| account_number | 15631 | 15325 | 15135 | 190 | 496 | 0.9876 | 0.9683 | 0.9778 |
UTF-8 byte-union redaction precision 0.9966, recall 0.9789, F1 0.9876. Missed gold bytes: 63,160; unnecessary redaction bytes: 10,061. Coverage ignores labels and weights multibyte characters by byte length.
| Document measure | Count |
|---|---|
| total | 90,018 |
| positive_support | 73,785 |
| clean_support | 16,233 |
| false_positive | 81 |
| false_negative | 469 |
| with_any_missed_entity | 3,947 |
| with_any_spurious_entity | 1,730 |
Document false negatives mean no accepted span on a document containing target PII; partial misses are counted separately in with_any_missed_entity. False positives mean at least one accepted span on a target-clean document.
Confidence and operating point
Method: isotonic, pooled across labels. Experimental acceptance threshold: 0.64864865. Objective: Max exact typed-span F1 on calibration-select; ties favor recall, then higher threshold.
Calibration fit support: 501 documents, 1,220 predicted spans, 1,172 exact matches. Selection support: 532 documents.
| Score | Predicted-span support | Brier | ECE |
|---|---|---|---|
| Raw token-derived span score | 250771 | 0.014821 | 0.011747 |
| Calibrated exact-span score | 250771 | 0.013717 | 0.007427 |
| Confidence bin | Raw n | Raw mean | Raw accuracy | Calibrated n | Calibrated mean | Calibrated accuracy |
|---|---|---|---|---|---|---|
| [0.0, 0.1) | 0 | Unavailable | Unavailable | 1245 | 0.0007 | 0.1614 |
| [0.1, 0.2) | 0 | Unavailable | Unavailable | 14 | 0.1477 | 0.0714 |
| [0.2, 0.3) | 1 | 0.2717 | 0.0000 | 668 | 0.2290 | 0.2320 |
| [0.3, 0.4) | 57 | 0.3739 | 0.1228 | 256 | 0.3294 | 0.2539 |
| [0.4, 0.5) | 502 | 0.4679 | 0.1653 | 74 | 0.4540 | 0.2838 |
| [0.5, 0.6) | 2223 | 0.5447 | 0.2231 | 3207 | 0.5402 | 0.4185 |
| [0.6, 0.7) | 1540 | 0.6486 | 0.3922 | 5336 | 0.6485 | 0.7731 |
| [0.7, 0.8) | 1564 | 0.7521 | 0.5403 | 10 | 0.7604 | 1.0000 |
| [0.8, 0.9) | 2132 | 0.8565 | 0.7148 | 11 | 0.8455 | 0.7273 |
| [0.9, 1.0] | 242752 | 0.9975 | 0.9932 | 239950 | 0.9925 | 0.9949 |
Reliability uses all predicted spans before thresholding and exact label/boundary correctness. Missed entities are absent from this table: recall must be read separately. Ten equal-width bins make ECE bin-dependent. Calibration is conditional on this synthetic evaluation population and annotation quality; the threshold is not a production default.
Existing regex comparison
| System | Scope | Gold entities | Precision | Recall | F1 |
|---|---|---|---|---|---|
| DataFog 4.8.1 regex | overall | 71279 | 0.7171 | 0.9354 | 0.8118 |
| DataFog 4.8.1 regex | 49246 | 0.9938 | 0.9899 | 0.9919 | |
| DataFog 4.8.1 regex | phone_number | 22033 | 0.4081 | 0.8136 | 0.5435 |
| Trained model, experimental threshold | overall | 71279 | 0.9848 | 0.9698 | 0.9773 |
| Trained model, experimental threshold | 49246 | 0.9857 | 0.9774 | 0.9815 | |
| Trained model, experimental threshold | phone_number | 22033 | 0.9829 | 0.9528 | 0.9676 |
Only email and phone are supported by this existing regex baseline within the seven-label contract. First name, last name, street address, customer ID, and account number are excluded from both sides of this head-to-head comparison. All documents remain in the comparison. Constant regex match indicators are not calibrated probabilities. See baseline provenance.
Synthetic support fixtures
7 fixture documents; exact-span precision 0.9333, recall 0.7778, F1 0.8485. Gold entity support: 18.
These hand-authored cases include Unicode signatures, pasted fields, multiline addresses, repeated entities, clean numeric text, and long inputs. They exercise mechanics and expose case-specific failures; they do not establish real customer-support quality.
Native resource measurements
| Measurement | Observed value |
|---|---|
| bundle_bytes | 300,076,892 |
| cold_load_seconds | 0.170 |
| peak_rss_bytes | 496,418,816 |
| Input UTF-8 bytes | Warm end-to-end seconds |
|---|---|
| 115 | 0.006005 |
| 752 | 0.020579 |
| 2010 | 0.081282 |
| 4502 | 0.172720 |
Maximum absolute Python/Rust logit difference on parity cases: 0.00000000. Token IDs, masks and global byte offsets agree: True; findings/redaction agree: True.
Remaining limitations
- Nemotron is broad synthetic data with semantic annotation errors. Structural filtering cannot prove that labels are semantically correct or complete.
- Split allocation uses documented label-support heuristics and grouped families. It does not guarantee statistical precision or eliminate every possible near-duplicate.
- No private customer tickets were supplied or uploaded; real-world English customer-support generalization remains unproven.
- Token boundaries cannot represent every character boundary. Observed training-source unrepresentable boundaries: first_name=10, last_name=5, phone_number=1.
- Native inference requires the packaged runtime and pinned tokenizer/configuration. A successful fixture parity check is not a universal numerical guarantee.
contains_piimeans at least one accepted finding among the seven selected labels; it is not an independently calibrated probability or assurance that a document contains no other sensitive information.- No existing DataFog checkout was modified, no Git repository was created, and no model was published as part of this standalone prototype.
Reproduction commands and artifact locations are in README.md.
Final artifact review
The frozen inference/evaluation inputs remained unchanged throughout the final run. Independent recount confirmed all 90,018 test documents and TP242,853 / FP2,424 / FN5,886. All16 bundle checksums verified. The copied native executable successfully processed Unicode and clean inputs with no Python on PATH. Evidence: final-audit.json.
Runtime measurements are single observations at four observed validation input lengths, not percentile latency guarantees. Cold load includes process launch and the first empty-input inference. Memory is the maximum RSS sampled after responses, not an allocator high-water mark. The pooled calibration fit had only 48 incorrect predicted spans; sparse confidence regions remain unreliable even though aggregate held-out Brier improved. No production threshold is assigned.