docs/tool_card.md

NoteGuard — Tool Card

Version: 0.2.0 Track: Public Sector & Citizen Services — NHS Secure Data Environment on-ramp Status: Hackathon prototype; not validated for production use without further evaluation.

---

Specification

Field	Value
Description	De-identification gate + LangGraph agent that detects/removes PII from NHS clinical notes before any LLM sees them, then grounds a discharge summary in public NICE/NHS guidance
Type	Hybrid pipeline — pure-Python rule recognisers + Microsoft Presidio (spaCy NER, en_core_web_md, shipped in the deployed image); Gemini 2.5 Flash as the agent model; Tavily for public-guidance retrieval
Developer	Chaeyoon Kim — {Tech: Europe} London AI Hackathon
Status / version	Prototype · v1.0.0
Repository	github.com/chaeyoonyunakim/noteguard-agent

Documented as a tool card, not a model card. NoteGuard does not train a model. A gov.uk Algorithmic Transparency Recording Standard (ATRS) record is in report.md.

---

What it does

NoteGuard is a trust layer for clinical AI. It:

1. De-identifies free-text NHS clinical notes inside the Trust using rule-based recognisers (NHS number, GMC/NMC, DOB, postcode, email, phone) and a Presidio + spaCy NER detector (en_core_web_md, shipped in the deployed image; degrades to a no-op stub if absent) that catches free-text patient/clinician names with no vault entry.
2. Asserts clean — raises ValueError if any known identifier survives, hard-blocking the model boundary.
3. Passes de-identified text to a LangGraph ReAct agent (Gemini 2.5 Flash) that drafts a compact eDischarge summary grounded in NICE/NHS guidance retrieved via Tavily.
4. Re-identifies the summary for the clinician — the model never sees or writes a real name.
5. Audits the de-identification — a vault-independent scan_pii pass flags any residual PII the model still saw (incl. free-text names the vault missed), plus orphaned surrogate tokens for reversibility. The trust panel reports only this; it carries no answer-quality metrics.

"De-identify in → agent (Gemini + Tavily) → re-identify out → compute trust."

---

Who uses it

Role	When	Why
NHS Clinician	At discharge	Needs a draft summary without manually de-identifying notes
Data Wrangler / IG Analyst	Before releasing notes to AI teams	Cannot share raw free-text; needs measured leakage
SDE Operator	At Trust egress boundary	Gate between raw Trust data and a Secure Data Environment

---

Use cases out of scope

• Not a substitute for Information Governance sign-off, a DPIA, or DARS approval — it is a technical control, not a legal basis for processing.
• Not validated on real Trust data, non-English notes, or scanned/handwritten documents.
• Not a guarantee of zero re-identification: pseudonymised output is still personal data under UK GDPR, and residual leakage is measured, not assumed zero on unseen data.
• Not for clinical decision-making, autonomous prescribing, or any use of note content beyond de-identification and grounded summarisation.
• Not a replacement for a clinician review — the output is a draft that the clinician signs off.

---

Detection coverage

Entity type	Method	Notes
Patient / clinician name (PERSON)	Vault match (patients.csv + admissions.csv) + Presidio spaCy NER (en_core_web_md)	Vault is ground-truth; Presidio catches unlisted free-text names; scan_pii audits whatever still slips through
NHS number (NHS)	Regex + 9-digit context anchor	Catches standard and synthetic-dataset forms
Date of birth (DOB)	DOB regex + consistent date-shift	Shift preserves clinical plausibility
UK postcode (POSTCODE)	Regex	Redacted outward-code only
GMC number (GMC)	Context-anchored regex with connector words	"GMC No. 7654321", "GMC number 7654321"
NMC / PIN (NMC)	Context-anchored regex with connector words	"NMC number: 18D6896L", "PIN 18D6896L"
Email / phone	Regex	Standard patterns

---

Anonymisation policy

Mode	Behaviour
Pseudonymise (default)	Consistent surrogate tokens [LABEL_n]; DOB date-shifted; reidentify() restores originals for the clinician
Patient naming	The discharge summary never names the patient — no title line, no real name, no patient surrogate; the model refers to "the patient" throughout

---

Bias and fairness

The Presidio NER (spaCy en_core_web_md) is trained largely on Western/English text, so name recall can be lower for names of non-English origin — and a smaller model (_md vs _lg) trades some recall for image size. This is an equity risk: under-detection means those patients carry a higher residual re-identification risk. The scan_pii audit surfaces residual names to the clinician rather than hiding the gap. Honest position and mitigations:

• The rule-based recognisers (NHS number, DOB, postcode, GMC/NMC) are name-agnostic and detect structured identifiers uniformly regardless of patient demographics.
• The vault (from patients.csv + admissions.csv) provides deterministic, demographic-agnostic coverage for all names in the structured dataset.
• Required before deployment: evaluate name recall stratified by name origin on representative Trust data and report the disparity. Not yet done — evaluation is on the NHSEDataScience/synthetic_clinical_notes synthetic dataset only.

---

NHS Five Safes mapping

Safe	Status	How
Safe data	✅	De-identified before model; assert_clean() hard gate; leakage measured
Safe settings	✅	Processing inside Trust; vault gitignored; Tavily receives only public queries
Safe outputs	✅	Only de-identified text + content-free trust metrics leave model boundary
Safe people	⚠️	Clinician reviews and signs off the draft; vault stays Trust-local
Safe projects	⚠️	Technical layer only; DPIA + project approval (DARS) remain Trust processes

---

Limitations and caveats

• Pseudonymised data is still personal data under UK GDPR — the surrogate vault is the re-identification key and must stay Trust-local.
• Not clinically validated: evaluated on the NHSEDataScience/synthetic_clinical_notes dataset. Real deployment requires validation on representative Trust data.
• Tavily search is for public NICE/NHS guidance only — patient text or surrogate tokens are never sent to it (enforced in the SYSTEM prompt and monitored via the trust metric).
• Governance prerequisites for deployment: a Data Protection Impact Assessment (DPIA), IG/Caldicott sign-off, and DARS project approval are required before any real use. NoteGuard is the technical control, not the approval.

---

Components

Component	Role in NoteGuard
Google Gemini 2.5 Flash	Agent model — drafts the eDischarge summary from de-identified text
Tavily	Public-guidance search — NICE/NHS sources only; never receives patient text
LangGraph	Agent orchestration
LangSmith	Tracing and evaluation

---

NoteGuard · {Tech: Europe} London AI Hackathon · prototype · not for clinical use without further validation