provenance/docs/rust-runtime.md

# Native Rust runtime

The standalone `rust/` Cargo package is a local JSONL command-line example. It does
not initialize a Git repository, perform downloads at inference, or require Python.
It loads one model/tokenizer session and reuses it for every stdin line.

## Reproduce

Run from the project folder:

```sh
DEVELOPER_DIR=/Library/Developer/CommandLineTools cargo build --locked --release --manifest-path rust/Cargo.toml
DEVELOPER_DIR=/Library/Developer/CommandLineTools cargo test --locked --release --manifest-path rust/Cargo.toml
printf '%s\n' '{"text":"🙂 José García: jose@example.com"}' | rust/target/release/datafog-pii artifacts/probe
.venv/bin/python scripts/check_rust_parity.py artifacts/probe --output artifacts/probe/rust-parity.json
```

`DEVELOPER_DIR` is scoped to the build process. The selected full Xcode installation
requires license acceptance, but the installed standalone Command Line Tools work
without a global settings change. Build verified with rustc/cargo 1.94.1 on arm64.
`--debug` adds IDs, masks, byte offsets and logits per window for parity inspection.
Debug output contains input-derived information and is for local testing.

Bundle requirements: `model.onnx` (plus any ONNX external tensor files),
`tokenizer.json`, `config.json`, `libonnxruntime.dylib`, and optional calibration
JSON named by `config.calibration_file`. Config keys: seven entity `labels` in
contract order, verified `max_length`, measured `stride`. Calibration keys:
`xs`, `ys`, `threshold`; clipped linear interpolation. Null calibration means
`confidence:null` and no threshold. Extra provenance fields are ignored.

Input is one JSON object per line with a string `text`. Output success is
`{findings,contains_pii,redacted}` with global half-open UTF-8 byte offsets.
Malformed requests and inference failures return `{error:...}`; they never claim
clean text. Startup/load errors exit nonzero. The CLI recovers from per-request
errors. Tokenize without truncation, manually slice complete inputs into content windows,
add CLS/SEP, independently decode BIO, then use
the shared contract's minimum token probability and greedy overlap resolution.
Before greedy overlap resolution, suppress artificial-edge findings strictly
contained in another same-label candidate. Retain edge findings with no containing
candidate. Calibration/threshold are applied after overlap resolution. A predicted entity
covering both content edges of a full window returns an explicit unsupported-boundary
error. This catches a concrete representability failure, but cannot guarantee that
unrecognized entities longer than a window are detected.

## Provenance and native packaging

- [ort 2.0.0-rc.13](https://docs.rs/crate/ort/2.0.0-rc.13) is the community
  [pykeio/ort](https://github.com/pykeio/ort) Rust binding (MIT OR Apache-2.0),
  not a Microsoft-maintained Rust API. Explicit `load-dynamic`, `std`, and `api-27`
  features avoid automatic runtime downloads. Cargo.lock pins the dependency tree.
- ONNX Runtime 1.30.0 native arm64 dylib comes from the locked official Python
  `onnxruntime` wheel installed in this project's isolated `.venv`; its MIT license
  and ThirdPartyNotices are preserved in `rust/licenses/`. Python is only the
  acquisition source; the copied dylib is loaded directly by the native process.
  `file` confirmed Mach-O arm64; `otool -L` showed only macOS libraries/frameworks.
  CPUExecutionProvider is used. The bundled library advertises API support at
  least through API27, verified by successful session creation/inference.
- [Hugging Face tokenizers 0.23.2](https://github.com/huggingface/tokenizers)
  (Apache-2.0) matches installed Python tokenizers. Rust `encode` uses original
  UTF-8 byte offsets; Python's character offsets are explicitly converted for parity.
- Local licenses for the binding and tokenizer are also in `rust/licenses/`.
  This is a local prototype, not a redistribution license audit of every transitive
  Cargo dependency. A published binary would need its complete notices inventory.

Copy the native library for a newly exported bundle with:

```python
from pathlib import Path
import onnxruntime, shutil
runtime = next(Path(onnxruntime.__file__).parent.rglob('libonnxruntime.*.dylib'))
shutil.copy2(runtime, Path('artifacts/bundle/libonnxruntime.dylib'))
```

## Evidence and numerical contract

`artifacts/probe/rust-parity.json` verifies seven input cases including empty text,
Unicode/emoji, repeated names, multiline fields and two long multi-window inputs.
Token IDs, masks, global byte offsets and f32 logits were exactly equal between
Python ONNX Runtime and Rust ONNX Runtime using one CPU thread. Spans and redaction
also matched. Raw-score comparison allows 64 float64 epsilons because Rust scalar
summation and NumPy reduction use different summation orders; this is approximately
1.42e-14, not a detection threshold. Calibration comparison allows 1e-10 for
propagated interpolation rounding. The probe's classifier is only compatibility
weights; its detections do not establish quality. Final trained-bundle parity must
be rerun after training and calibration with the measured overlap.

Unit tests establish Unicode redaction boundaries, adjacent findings, overlap
priority, duplicate suppression, interpolation and invalid UTF-8 boundary rejection.
The oracle boundary test independently verifies complete token coverage and exact
spans/redaction for email and 43-token entities crossing a chunk edge, with clipped
fragments assigned higher scores than complete predictions. It also checks repeated
adjacent Unicode names. `artifacts/boundary-proof.json` records the Python proof;
the Rust unit test uses the same oracle labels with independent native tokenization.
The parity harness checks the actual public `pii.core.Predictor` as well as an
independent ONNX session, and checks an invalid request followed by successful same-process
recovery. Missing-bundle startup returns a nonzero explicit load error.

## Corrective round 1: complete-input guarantee

The oracle exposed a tokenizers 0.23.2 defect: configuring truncation caused the
model-tokenization early exit to discard the input tail before overflow encodings
were formed. Python/Rust agreement on old overflow results did not establish input
coverage. Both implementations now tokenize the full input without truncation and
explicitly slice windows. Training must use `pii.core.encode_windows` too. Cached
training chunks created using tokenizer overflow are invalid. The pinned tokenizer
source's `do_tokenize` passed `max_length` to early tokenization regardless of
requested overflow; no upstream files were changed.

The former score-only overlap rule also selected higher-confidence clipped
fragments over complete lower-confidence spans. Boundary provenance now permits
removing strictly contained artificial-edge fragments before greedy resolution.
This is a deterministic mechanics fix, not tuning against held-out predictions.