provenance/docs/rust-runtime.md

Native Rust runtime

The standalone rust/ Cargo package is a local JSONL command-line example. It does not initialize a Git repository, perform downloads at inference, or require Python. It loads one model/tokenizer session and reuses it for every stdin line.

Reproduce

Run from the project folder:

DEVELOPER_DIR=/Library/Developer/CommandLineTools cargo build --locked --release --manifest-path rust/Cargo.toml
DEVELOPER_DIR=/Library/Developer/CommandLineTools cargo test --locked --release --manifest-path rust/Cargo.toml
printf '%s\n' '{"text":"🙂 José García: jose@example.com"}' | rust/target/release/datafog-pii artifacts/probe
.venv/bin/python scripts/check_rust_parity.py artifacts/probe --output artifacts/probe/rust-parity.json

DEVELOPER_DIR is scoped to the build process. The selected full Xcode installation requires license acceptance, but the installed standalone Command Line Tools work without a global settings change. Build verified with rustc/cargo 1.94.1 on arm64. --debug adds IDs, masks, byte offsets and logits per window for parity inspection. Debug output contains input-derived information and is for local testing.

Bundle requirements: model.onnx (plus any ONNX external tensor files), tokenizer.json, config.json, libonnxruntime.dylib, and optional calibration JSON named by config.calibration_file. Config keys: seven entity labels in contract order, verified max_length, measured stride. Calibration keys: xs, ys, threshold; clipped linear interpolation. Null calibration means confidence:null and no threshold. Extra provenance fields are ignored.

Input is one JSON object per line with a string text. Output success is {findings,contains_pii,redacted} with global half-open UTF-8 byte offsets. Malformed requests and inference failures return {error:...}; they never claim clean text. Startup/load errors exit nonzero. The CLI recovers from per-request errors. Tokenize without truncation, manually slice complete inputs into content windows, add CLS/SEP, independently decode BIO, then use the shared contract's minimum token probability and greedy overlap resolution. Before greedy overlap resolution, suppress artificial-edge findings strictly contained in another same-label candidate. Retain edge findings with no containing candidate. Calibration/threshold are applied after overlap resolution. A predicted entity covering both content edges of a full window returns an explicit unsupported-boundary error. This catches a concrete representability failure, but cannot guarantee that unrecognized entities longer than a window are detected.

Provenance and native packaging

• ort 2.0.0-rc.13 is the community pykeio/ort Rust binding (MIT OR Apache-2.0), not a Microsoft-maintained Rust API. Explicit load-dynamic, std, and api-27 features avoid automatic runtime downloads. Cargo.lock pins the dependency tree.
• ONNX Runtime 1.30.0 native arm64 dylib comes from the locked official Python onnxruntime wheel installed in this project's isolated .venv; its MIT license and ThirdPartyNotices are preserved in rust/licenses/. Python is only the acquisition source; the copied dylib is loaded directly by the native process. file confirmed Mach-O arm64; otool -L showed only macOS libraries/frameworks. CPUExecutionProvider is used. The bundled library advertises API support at least through API27, verified by successful session creation/inference.
• Hugging Face tokenizers 0.23.2 (Apache-2.0) matches installed Python tokenizers. Rust encode uses original UTF-8 byte offsets; Python's character offsets are explicitly converted for parity.
• Local licenses for the binding and tokenizer are also in rust/licenses/. This is a local prototype, not a redistribution license audit of every transitive Cargo dependency. A published binary would need its complete notices inventory.

Copy the native library for a newly exported bundle with:

from pathlib import Path
import onnxruntime, shutil
runtime = next(Path(onnxruntime.__file__).parent.rglob('libonnxruntime.*.dylib'))
shutil.copy2(runtime, Path('artifacts/bundle/libonnxruntime.dylib'))

Evidence and numerical contract

artifacts/probe/rust-parity.json verifies seven input cases including empty text, Unicode/emoji, repeated names, multiline fields and two long multi-window inputs. Token IDs, masks, global byte offsets and f32 logits were exactly equal between Python ONNX Runtime and Rust ONNX Runtime using one CPU thread. Spans and redaction also matched. Raw-score comparison allows 64 float64 epsilons because Rust scalar summation and NumPy reduction use different summation orders; this is approximately 1.42e-14, not a detection threshold. Calibration comparison allows 1e-10 for propagated interpolation rounding. The probe's classifier is only compatibility weights; its detections do not establish quality. Final trained-bundle parity must be rerun after training and calibration with the measured overlap.

Unit tests establish Unicode redaction boundaries, adjacent findings, overlap priority, duplicate suppression, interpolation and invalid UTF-8 boundary rejection. The oracle boundary test independently verifies complete token coverage and exact spans/redaction for email and 43-token entities crossing a chunk edge, with clipped fragments assigned higher scores than complete predictions. It also checks repeated adjacent Unicode names. artifacts/boundary-proof.json records the Python proof; the Rust unit test uses the same oracle labels with independent native tokenization. The parity harness checks the actual public pii.core.Predictor as well as an independent ONNX session, and checks an invalid request followed by successful same-process recovery. Missing-bundle startup returns a nonzero explicit load error.

Corrective round 1: complete-input guarantee

The oracle exposed a tokenizers 0.23.2 defect: configuring truncation caused the model-tokenization early exit to discard the input tail before overflow encodings were formed. Python/Rust agreement on old overflow results did not establish input coverage. Both implementations now tokenize the full input without truncation and explicitly slice windows. Training must use pii.core.encode_windows too. Cached training chunks created using tokenizer overflow are invalid. The pinned tokenizer source's do_tokenize passed max_length to early tokenization regardless of requested overflow; no upstream files were changed.

The former score-only overlap rule also selected higher-confidence clipped fragments over complete lower-confidence spans. Boundary provenance now permits removing strictly contained artificial-edge fragments before greedy resolution. This is a deterministic mechanics fix, not tuning against held-out predictions.