# DataFog PII prototype evaluation

This is an evaluated prototype, not a production-readiness claim. All numbers below come from saved artifacts; absent evidence is explicitly marked unavailable.

Required evaluation artifacts are present.

| Contract criterion | Status | Evidence | Interpretation / limitation |
| --- | --- | --- | --- |
| Pinned data and disjoint document families | Evidence present | [artifacts/data-manifest.json](artifacts/data-manifest.json) | Structural validation and split rules are saved; annotation semantics remain imperfect. |
| Trained checkpoint and resumability | Evidence present | [artifacts/training/complete.json](artifacts/training/complete.json), [artifacts/training-check.json](artifacts/training-check.json) | Completion plus separate optimizer/RNG resume diagnostic; inspect reported MPS parameter differences. |
| Native Rust and Python parity | Evidence present | [artifacts/bundle/rust-parity.json](artifacts/bundle/rust-parity.json) | Checks tokenizer/global byte offsets, outputs/redaction, and explicit error recovery; parity does not prove quality. |
| Complete-input boundary behavior | Evidence present | [artifacts/boundary-proof.json](artifacts/boundary-proof.json) | Manual global-offset window construction; finite fixture evidence and representability limits apply. |
| Independent calibration and experimental threshold | Evidence present | [artifacts/bundle/calibration.json](artifacts/bundle/calibration.json), [artifacts/evaluation/test.json](artifacts/evaluation/test.json) | Fit/select partitions are separate; final reliability is reported on held-out predicted spans. |
| Held-out quality and support mechanics | Evidence present | [artifacts/evaluation/test.json](artifacts/evaluation/test.json), [artifacts/evaluation/support-fixtures.json](artifacts/evaluation/support-fixtures.json) | Synthetic population only; no reviewed private tickets. |
| Existing regex comparison | Evidence present | [artifacts/regex-test/regex-metrics.json](artifacts/regex-test/regex-metrics.json), [artifacts/regex-test/model-email-phone-metrics.json](artifacts/regex-test/model-email-phone-metrics.json) | Official published DataFog 4.8.1; email and phone only. |
| Native resource measurements | Evidence present | [artifacts/native-benchmark.json](artifacts/native-benchmark.json) | Local observed load, warm request latency, memory, and bundle size; no performance target assumed. |

## Data and execution

Dataset: `nvidia/Nemotron-PII` revision `b70ffaf5ff39e079776134c5bf4381f00a9fd1ed`. License: CC-BY-4.0. NVIDIA; Amy Steier, Andre Manoel, Alexa Haushalter, Maarten Van Segbroeck, Nemotron-PII (2025). https://huggingface.co/datasets/nvidia/Nemotron-PII

| Partition | Documents | Target-clean documents |
| --- | --- | --- |
| train | 89307 | 23563 |
| validation | 514 | 104 |
| calibration_fit | 501 | 107 |
| calibration_select | 532 | 105 |
| test | 90018 | 16233 |

Structural exclusions: source_slice_mismatch: 19,101, duplicate_or_overlap: 20, test_family_overlap: 7.

| Epoch | Train mean batch loss | Validation token loss | Selected at epoch | Elapsed seconds |
| --- | --- | --- | --- | --- |
| 1 | 0.0135 | 0.0036 | True | 4,166.3 |
| 2 | 0.0035 | 0.0019 | True | 8,102.2 |

Training elapsed: 8,102.2 seconds. Best validation token loss: 0.0019.

Corrective iterations: two of the three-round fuse used. Round 1 fixed tokenizer overflow by manually constructing complete-input windows; round 2 kept the ONNX export wrapper and Python reference in evaluation mode. Neither correction tuned against final test outcomes. Further corrective work must remain within the agreed fuse.

## Held-out detection

| Label | Gold entities | Predicted | TP | FP | FN | Precision | Recall | F1 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| overall | 248739 | 245277 | 242853 | 2424 | 5886 | 0.9901 | 0.9763 | 0.9832 |
| first_name | 74498 | 73673 | 73234 | 439 | 1264 | 0.9940 | 0.9830 | 0.9885 |
| last_name | 52864 | 52148 | 51786 | 362 | 1078 | 0.9931 | 0.9796 | 0.9863 |
| email | 49246 | 48832 | 48134 | 698 | 1112 | 0.9857 | 0.9774 | 0.9815 |
| phone_number | 22033 | 21358 | 20992 | 366 | 1041 | 0.9829 | 0.9528 | 0.9676 |
| street_address | 15382 | 15099 | 14868 | 231 | 514 | 0.9847 | 0.9666 | 0.9756 |
| customer_id | 19085 | 18842 | 18704 | 138 | 381 | 0.9927 | 0.9800 | 0.9863 |
| account_number | 15631 | 15325 | 15135 | 190 | 496 | 0.9876 | 0.9683 | 0.9778 |

UTF-8 byte-union redaction precision 0.9966, recall 0.9789, F1 0.9876. Missed gold bytes: 63,160; unnecessary redaction bytes: 10,061. Coverage ignores labels and weights multibyte characters by byte length.

| Document measure | Count |
| --- | --- |
| total | 90,018 |
| positive_support | 73,785 |
| clean_support | 16,233 |
| false_positive | 81 |
| false_negative | 469 |
| with_any_missed_entity | 3,947 |
| with_any_spurious_entity | 1,730 |

Document false negatives mean no accepted span on a document containing target PII; partial misses are counted separately in `with_any_missed_entity`. False positives mean at least one accepted span on a target-clean document.

## Confidence and operating point

Method: isotonic, pooled across labels. Experimental acceptance threshold: 0.64864865. Objective: Max exact typed-span F1 on calibration-select; ties favor recall, then higher threshold.

Calibration fit support: 501 documents, 1,220 predicted spans, 1,172 exact matches. Selection support: 532 documents.

| Score | Predicted-span support | Brier | ECE |
| --- | --- | --- | --- |
| Raw token-derived span score | 250771 | 0.014821 | 0.011747 |
| Calibrated exact-span score | 250771 | 0.013717 | 0.007427 |

| Confidence bin | Raw n | Raw mean | Raw accuracy | Calibrated n | Calibrated mean | Calibrated accuracy |
| --- | --- | --- | --- | --- | --- | --- |
| [0.0, 0.1) | 0 | Unavailable | Unavailable | 1245 | 0.0007 | 0.1614 |
| [0.1, 0.2) | 0 | Unavailable | Unavailable | 14 | 0.1477 | 0.0714 |
| [0.2, 0.3) | 1 | 0.2717 | 0.0000 | 668 | 0.2290 | 0.2320 |
| [0.3, 0.4) | 57 | 0.3739 | 0.1228 | 256 | 0.3294 | 0.2539 |
| [0.4, 0.5) | 502 | 0.4679 | 0.1653 | 74 | 0.4540 | 0.2838 |
| [0.5, 0.6) | 2223 | 0.5447 | 0.2231 | 3207 | 0.5402 | 0.4185 |
| [0.6, 0.7) | 1540 | 0.6486 | 0.3922 | 5336 | 0.6485 | 0.7731 |
| [0.7, 0.8) | 1564 | 0.7521 | 0.5403 | 10 | 0.7604 | 1.0000 |
| [0.8, 0.9) | 2132 | 0.8565 | 0.7148 | 11 | 0.8455 | 0.7273 |
| [0.9, 1.0] | 242752 | 0.9975 | 0.9932 | 239950 | 0.9925 | 0.9949 |

Reliability uses all predicted spans before thresholding and exact label/boundary correctness. Missed entities are absent from this table: recall must be read separately. Ten equal-width bins make ECE bin-dependent. Calibration is conditional on this synthetic evaluation population and annotation quality; the threshold is not a production default.

## Existing regex comparison

| System | Scope | Gold entities | Precision | Recall | F1 |
| --- | --- | --- | --- | --- | --- |
| DataFog 4.8.1 regex | overall | 71279 | 0.7171 | 0.9354 | 0.8118 |
| DataFog 4.8.1 regex | email | 49246 | 0.9938 | 0.9899 | 0.9919 |
| DataFog 4.8.1 regex | phone_number | 22033 | 0.4081 | 0.8136 | 0.5435 |
| Trained model, experimental threshold | overall | 71279 | 0.9848 | 0.9698 | 0.9773 |
| Trained model, experimental threshold | email | 49246 | 0.9857 | 0.9774 | 0.9815 |
| Trained model, experimental threshold | phone_number | 22033 | 0.9829 | 0.9528 | 0.9676 |

Only email and phone are supported by this existing regex baseline within the seven-label contract. First name, last name, street address, customer ID, and account number are excluded from both sides of this head-to-head comparison. All documents remain in the comparison. Constant regex match indicators are not calibrated probabilities. See [baseline provenance](docs/regex-baseline.md).

## Synthetic support fixtures

7 fixture documents; exact-span precision 0.9333, recall 0.7778, F1 0.8485. Gold entity support: 18.

These hand-authored cases include Unicode signatures, pasted fields, multiline addresses, repeated entities, clean numeric text, and long inputs. They exercise mechanics and expose case-specific failures; they do not establish real customer-support quality.

## Native resource measurements

| Measurement | Observed value |
| --- | --- |
| bundle_bytes | 300,076,892 |
| cold_load_seconds | 0.170 |
| peak_rss_bytes | 496,418,816 |

| Input UTF-8 bytes | Warm end-to-end seconds |
| --- | --- |
| 115 | 0.006005 |
| 752 | 0.020579 |
| 2010 | 0.081282 |
| 4502 | 0.172720 |

Maximum absolute Python/Rust logit difference on parity cases: 0.00000000. Token IDs, masks and global byte offsets agree: True; findings/redaction agree: True.

## Remaining limitations

- Nemotron is broad synthetic data with semantic annotation errors. Structural filtering cannot prove that labels are semantically correct or complete.
- Split allocation uses documented label-support heuristics and grouped families. It does not guarantee statistical precision or eliminate every possible near-duplicate.
- No private customer tickets were supplied or uploaded; real-world English customer-support generalization remains unproven.
- Token boundaries cannot represent every character boundary. Observed training-source unrepresentable boundaries: first_name=10, last_name=5, phone_number=1.
- Native inference requires the packaged runtime and pinned tokenizer/configuration. A successful fixture parity check is not a universal numerical guarantee.
- `contains_pii` means at least one accepted finding among the seven selected labels; it is not an independently calibrated probability or assurance that a document contains no other sensitive information.
- No existing DataFog checkout was modified, no Git repository was created, and no model was published as part of this standalone prototype.

Reproduction commands and artifact locations are in [README.md](README.md).

## Final artifact review

The frozen inference/evaluation inputs remained unchanged throughout the final
run. Independent recount confirmed all 90,018 test documents and TP242,853 /
FP2,424 / FN5,886. All16 bundle checksums verified. The copied native executable
successfully processed Unicode and clean inputs with no Python on PATH.
Evidence: [final-audit.json](artifacts/final-audit.json).

Runtime measurements are single observations at four observed validation input
lengths, not percentile latency guarantees. Cold load includes process launch
and the first empty-input inference. Memory is the maximum RSS sampled after
responses, not an allocator high-water mark. The pooled calibration fit had only
48 incorrect predicted spans; sparse confidence regions remain unreliable even
though aggregate held-out Brier improved. No production threshold is assigned.