provenance/README.md

# DataFog PII local prototype

Standalone workspace on the M5 Pro / 48 GB Mac. No Git repository was created.
User-approved scope: autonomous implementation and subagents, public/synthetic
data, local runs estimated up to 24 hours, no purchases or publishing. Existing
SDK integration is deferred. See `docs/contract.md`.

## Current state

Completed and reviewed. See [REPORT.md](REPORT.md) for full results and limits.
On 90,018 held-out synthetic documents: exact typed-span precision 99.01%,
recall 97.63%, F1 98.32%. Independent recount and frozen-input/bundle checksums
passed (`artifacts/final-audit.json`). Two corrective rounds were used before
final test exposure. No test-driven model or threshold changes were made.

The native bundle is `artifacts/bundle/` (about 300 MB including runtime).
Run without Python from this folder:

```sh
printf '%s\n' '{"text":"Email alice@example.com"}' | artifacts/bundle/datafog-pii artifacts/bundle
```

Training took 8,102 seconds. Colab was not needed; no extra purchases were made.
The pipeline has exited and automatic follow-ups are paused. The idle tmux
control session remains available; no compute job is running.

Recorded stage status: `artifacts/pipeline/status.json`. Logs and exact commands:
`artifacts/pipeline/*.log` and `*.done.json`. Run `uv run --frozen python
scripts/final_audit.py` to repeat the artifact audit.

## Controls

```sh
tmux attach -t datafog-pii
./scripts/start-local.sh
./scripts/stop-local.sh
```

The `control` window remains interactive. Up recalls start, then Up recalls
stop. Ctrl-b then n/p changes windows; Ctrl-b then d detaches. The `pipeline`
window holds the job. Start refuses to duplicate that window. Stop interrupts
only this project's process; epoch checkpoints and data remain.

To resume a stopped/failed pipeline, first verify no project pipeline is still
running and inspect the failed log, then run from this folder:

```sh
caffeinate -i .venv/bin/python scripts/run_pipeline.py
```

Successful stages are skipped; training resumes from the latest completed epoch
checkpoint. Work since that checkpoint can be lost. Do not run competing jobs.
Stage markers belong to this run; do not reuse them for changed experiments.
Never silently replace a frozen evaluation after inspecting test outcomes.

## Reproduction

Python 3.12 dependencies are isolated in `.venv`, pinned in `uv.lock`. Rust
uses `rust/Cargo.lock`. Global Python packages and system settings are unchanged.
These commands were run from this folder:

```sh
uv sync
uv run --frozen python scripts/check_environment.py
uv run --frozen python scripts/prepare_data.py
uv run --frozen python scripts/probe_model.py
uv run --frozen python scripts/check_training.py
uv run --frozen python scripts/check_chunk_boundaries.py
uv run --frozen python scripts/train.py --stride 42 --prepare-only
uv run --frozen python -m unittest discover -s tests -v
DEVELOPER_DIR=/Library/Developer/CommandLineTools cargo build --release --manifest-path rust/Cargo.toml
DEVELOPER_DIR=/Library/Developer/CommandLineTools cargo test --manifest-path rust/Cargo.toml
uv run --frozen python scripts/check_rust_parity.py artifacts/probe --output artifacts/probe/rust-parity.json
./scripts/start-local.sh
```

Use `uv sync --frozen` to recreate the locked environment. The existing Rust
version is 1.94.1. Process-local `DEVELOPER_DIR` selects installed Command Line
Tools because selected Xcode requires a license action; no global change made.

Pinned pretrained download performed before the probe:

```sh
uv run --frozen python -c 'from huggingface_hub import snapshot_download; snapshot_download("distilbert/distilbert-base-cased", revision="6ea81172465e8b0ad3fddeed32b986cdcdcffcf0", allow_patterns=["config.json","model.safetensors","tokenizer.json","tokenizer_config.json","vocab.txt","README.md","LICENSE"], local_dir="artifacts/backbone")'
```

Regex baseline setup and commands: `docs/regex-baseline.md`. The pipeline records
all stage commands and timestamps. Training starts from the official Hugging
Face token-classification recipe: two epochs, batch 16, AdamW 2e-5, weight decay
0.01, selecting lowest validation token loss. These are hypotheses, not optimal
settings. Model limit 512 comes from pinned configuration; overlap 42 derives
from the longest measured selected entity of 43 tokens. Seed 42, gradient clip
1.0 and linear LR schedule follow Transformers Trainer defaults.

## Evidence and outputs

- `docs/data.md`, `artifacts/data-manifest.json`: pinned data, split allocation,
  annotation exclusions, label support, and checksums.
- `artifacts/data-disjointness.json`: independent split verification.
- `artifacts/data-window-coverage.json`, `artifacts/boundary-proof.json`: complete
  token coverage and exact oracle boundary tests.
- `artifacts/training-check.json`: learning and optimizer/RNG resume proof.
- `docs/evaluation.md`: metrics and scoring semantics.
- `docs/rust-runtime.md`: native build, packaging, inference and licenses.
- `docs/run-log.md`: corrective work; two of three rounds used so far.
- `artifacts/bundle/`: final model, tokenizer, calibration, native library,
  executable, licenses, and checksums after pipeline completion.
- `artifacts/evaluation/`, `artifacts/regex-test/`: final measured results.

Dataset spans use character offsets; returned spans use half-open UTF-8 bytes.
`contains_pii` is true when at least one finding among seven target labels is
accepted. Scores/threshold are experimental, not a production default.

## Known limits

Nemotron is synthetic with remaining semantic annotation noise. Invalid source
slices/overlaps were quarantined, without regex relabeling. No private tickets
were evaluated. Sixteen training-side spans have boundaries the tokenizer
cannot express exactly. Overlap covers measured entity lengths, not every
possible unseen entity; full-window predicted entities raise explicit errors.

Tokenizers 0.23.2 overflow silently dropped tails. Both runtimes now tokenize
complete input and create windows manually. Obsolete cache
`artifacts/training/chunks-stride-42.pt` is diagnostic history only; current
training reads `chunks-manual-stride-42.pt`. Independent full-coverage oracles
prevent shared Python/Rust truncation defects from hiding behind parity.

Colab is not needed currently. Fallback uses the primary Google account signed
into Chrome, assigned hardware and existing Pro compute balance, with durable
checkpoints and identical scripts/manifests. No Colab skill has been created.