Pinned data preparation
Run from project root:
.venv/bin/python scripts/prepare_data.py
.venv/bin/python -m unittest tests/test_data.py
Source: NVIDIA Nemotron-PII, revision b70ffaf5ff39e079776134c5bf4381f00a9fd1ed, CC BY 4.0. Attribution: Amy Steier, Andre Manoel, Alexa Haushalter, and Maarten Van Segbroeck, NVIDIA, Nemotron-PII (2025), https://huggingface.co/datasets/nvidia/Nemotron-PII. The pinned original README, including license link and citation, is preserved in data/raw/README.md. Raw Parquet files are immutable input; their SHA256 checksums appear in artifacts/data-manifest.json.
There are 100,000 rows in each source split, despite the README describing 50,000 each. Both have 55 observed labels. Each source has 50,000 repeated UID rows; row count is not unique-person or independent-family count. Train has one duplicate exact text; test has none. The manifest includes complete observed label counts for structurally valid rows.
All span strings are parsed through JSON or ast.literal_eval, never eval. Records are excluded if any source annotation (including outside the seven target labels) has invalid bounds, does not match the exact original Unicode character slice, or overlaps/duplicates another annotation. Train excludes 9,128 slice-mismatch and 11 overlap records; test excludes 9,973 slice-mismatch and nine overlap records. All exclusions retain source split/index/UID and reason in data/prepared/excluded.jsonl; original annotations remain recoverable in pinned raw files. No guessed repairs are applied. Offset checks use Python Unicode codepoints, not UTF-8 bytes.
Group identity is a connected component of shared UID, exact text hash, or normalized template hash. Templates replace all annotated entities with their labels, collapse whitespace, and casefold the remaining context. This groups identical evidenced templates even when entity values change. It does not claim to detect every paraphrase or semantically near-duplicate document. Seven valid training-source rows belong to a family also found in test; those training rows are excluded. Original test rows are otherwise retained, including correlated rows within test families. Uncertainty estimates must account for such clustering.
The minimum eligible label support is 8,396 positive account-number documents. The prototype allocation heuristic uses ceil(sqrt(min_label_support)) = 92 positive documents per label, plus at least 92 scoped-negative documents in each heldout. This square-root rule balances training retention with growing heldout support; it is not a guaranteed statistical margin or an accuracy target. Stable SHA256-ordered groups are assigned to the heldout with the largest remaining support deficit (ties favor smaller partitions); all remaining groups train. No predictions or test performance influence allocation. Each prepared record persists its group and template hashes.
| Partition | Documents | Scoped negatives | Smallest per-label document support |
|---|---|---|---|
| train | 89,307 | 23,563 | 8,120 |
| validation | 514 | 104 | 92 |
| calibration_fit | 501 | 107 | 92 |
| calibration_select | 532 | 105 | 92 |
| test | 90,018 | 16,233 | 13,493 |
Prepared JSONLs contain the full original text and only seven selected labels. An empty target span list does not assert that a document is free of other sensitive data. The files and checksums are in data/prepared/ and the manifest. All pairs of partitions are verified disjoint by group, UID, exact text, and template hash.
The tokenizer is distilbert/distilbert-base-cased revision 6ea81172465e8b0ad3fddeed32b986cdcdcffcf0, with model limit512 (510 content tokens). Training has23,668,675 content tokens, median210,p90=519,p95=634,p99=890,max2176. There are9,369 training documents longer than510 content tokens. Across all training-source partitions, the largest target entity covers43 tokens (customer_id); other maxima: first_name10,last_name10,email26,phone_number13,street_address27,account_number29. An overlap of42 content tokens is sufficient for full coverage of measured entities; overlap43 adds one token. This is an empirical support property, not a universal input guarantee.
Sixteen selected spans have boundaries the tokenizer cannot exactly express: ten first_name, five last_name, one phone_number with leading whitespace. They remain in the original annotations and in evaluation. The manifest provides all16 examples. Training must ignore any token crossing a target boundary; exact reconstruction cannot be claimed for these cases.
Structural validity does not imply correct labeling. There are249 valid train-source and239 valid test-source email annotations without an @ character. These are flagged, not automatically relabeled by regex; nonstandard examples and other semantic errors remain. Examples of names inside larger words also suggest annotation noise. Evaluation measures agreement with this synthetic population, not real-world support ticket performance or annotation truth. Calibration fitting supports pooled exploratory calibration; only approximately500 documents per calibration partition makes per-label calibration unreliable without further evidence.