# Pipeline di Rilevamento — Input/Output per Step Documento tecnico che traccia un documento reale attraverso ogni fase del pipeline, mostrando cosa entra, cosa esce e come i risultati si fondono per produrre l'output finale. --- ## Documento di esempio ``` DETERMINA DIRIGENZIALE n. 47 del 15/03/2024 Stazione Appaltante: Comune di Napoli - P.IVA 00215670639 RUP: Dott. Mario Rossi - C.F. RSSMRA75A01F839X - email: m.rossi@pec.comune.napoli.it CIG: 90141610FB - CUP: B89J21005030006 - Importo base asta: Euro 450.000,00 Operatore economico aggiudicatario: Costruzioni Bianchi S.r.l. - IBAN IT60X0542811101000000123456 ``` Le entità sensibili contenute sono: | Campo | Valore | Tipo atteso | |---|---|---| | Data | `15/03/2024` | DATE_TIME | | P.IVA | `00215670639` | IT_VAT_CODE / PARTITA_IVA | | Nome | `Mario Rossi` | PERSON | | Codice Fiscale | `RSSMRA75A01F839X` | CODICE_FISCALE | | Email PEC | `m.rossi@pec.comune.napoli.it` | EMAIL_ADDRESS / PEC | | CIG | `90141610FB` | CIG | | CUP | `B89J21005030006` | CUP | | Importo | `Euro 450.000,00` | VALUTA + IMPORTO_GARA | | Società | `Costruzioni Bianchi S.r.l.` | SOCIETA | | IBAN | `IT60X0542811101000000123456` | IBAN_CODE | --- ## Step 1 — Chunking **Input**: stringa di testo (N caratteri) **Logica** (`core/pipeline.py` → `_chunk_text`): ```python def _chunk_text(text: str) -> list[tuple[int, str]]: if len(text) <= PIPELINE_CFG.chunk_size: # 1500 return [(0, text)] step = PIPELINE_CFG.chunk_size - PIPELINE_CFG.chunk_overlap # 1300 ... ``` Il documento di esempio ha 362 caratteri → **un solo chunk**: `[(0, testo_completo)]`. **Output**: lista di `(offset, chunk_text)`: ```python [(0, "DETERMINA DIRIGENZIALE n. 47 del 15/03/2024\n...")] ``` > Per documenti > 1500 caratteri vengono generati chunk sovrapposti. > Ogni entità rilevata nel chunk riceve l'offset aggiunto a `r.start` e `r.end` > per riportare le posizioni al testo originale. --- ## Step 2 — Step0: Presidio Built-in IT + EN **Input**: chunk di testo + lista `STEP0_RECOGNIZERS` (31 recognizer regex) **Logica** (`_run_step0_chunked`): ```python for rec in STEP0_RECOGNIZERS: for r in rec.analyze(text=chunk, entities=rec.supported_entities, nlp_artifacts=None): # nessun NLP engine r.start += offset r.end += offset all_results.append(r) ``` **Output grezzo** (tutti i risultati prima del filtro `min_score`): ```json [ { "entity_type": "DATE_TIME", "start": 33, "end": 43, "score": 0.60, "recognizer": "DateRecognizer", "text_span": "15/03/2024" }, { "entity_type": "IT_VAT_CODE", "start": 90, "end": 101, "score": 1.00, "recognizer": "ItVatCodeRecognizer", "text_span": "00215670639", "note": "checksum validato" }, { "entity_type": "IT_FISCAL_CODE", "start": 132, "end": 148, "score": 0.30, "recognizer": "ItFiscalCodeRecognizer", "text_span": "RSSMRA75A01F839X", "note": "score basso: no context words vicini" }, { "entity_type": "EMAIL_ADDRESS", "start": 158, "end": 186, "score": 1.00, "recognizer": "EmailRecognizer", "text_span": "m.rossi@pec.comune.napoli.it" }, { "entity_type": "IBAN_CODE", "start": 333, "end": 360, "score": 1.00, "recognizer": "IbanRecognizer", "text_span": "IT60X0542811101000000123456", "note": "checksum IBAN validato" }, { "entity_type": "US_BANK_NUMBER", "start": 90, "end": 101, "score": 0.05, "text_span": "00215670639", "note": "RUMORE — scartato dal min_score 0.85" }, { "entity_type": "IN_PAN", "start": 192, "end": 202, "score": 0.05, "text_span": "90141610FB", "note": "RUMORE — scartato dal min_score 0.85" } ] ``` **Osservazione**: Step0 cattura `IT_VAT_CODE`, `IT_FISCAL_CODE`, `EMAIL_ADDRESS`, `IBAN_CODE` ma **manca** CIG, CUP, SOCIETA, IMPORTO (entità dominio appalti non coperte da Presidio built-in). --- ## Step 3 — L0 Regex Appalti + L1 NER Transformer Entrambi i layer escono da un'unica chiamata ad `analyzer_full.analyze()` e vengono poi separati tramite `_is_regex_result()`. **Input**: chunk + `analyzer_full` (20 PatternRecognizer procurement + NER transformer) ```python presidio_results = _run_analyzer_chunked(analyzer_full, text) regex_results = [r for r in presidio_results if _is_regex_result(r)] ner_results = [r for r in presidio_results if not _is_regex_result(r)] ``` **`_is_regex_result`** controlla il campo `analysis_explanation.pattern_name`: ```python _REGEX_RECOGNIZER_NAMES = { "cig", "cup", "pec", "cf_azienda", "societa", "inps", "polizza_full", "polizza_alt", "cpv", "nuts", "ateco", "ribasso", "protocollo", "atto", "anac", "lotto", "siogg", ... } def _is_regex_result(r): if r.analysis_explanation is None: return False return r.analysis_explanation.pattern_name in _REGEX_RECOGNIZER_NAMES ``` ### L0 — Output Regex Appalti ```json [ { "entity_type": "PEC", "start": 158, "end": 186, "score": 1.00, "pattern_name": "pec", "text_span": "m.rossi@pec.comune.napoli.it", "note": "dominio PEC riconosciuto dal pattern" }, { "entity_type": "CIG", "start": 192, "end": 202, "score": 1.00, "pattern_name": "cig", "text_span": "90141610FB" }, { "entity_type": "CUP", "start": 210, "end": 225, "score": 1.00, "pattern_name": "cup", "text_span": "B89J21005030006" }, { "entity_type": "SOCIETA", "start": 299, "end": 325, "score": 1.00, "pattern_name": "societa", "text_span": "Costruzioni Bianchi S.r.l." } ] ``` ### L1 — Output NER Transformer (dopo riclassificazione) ```json [ { "entity_type": "DATE_TIME", "start": 33, "end": 43, "score": 0.99, "text_span": "15/03/2024" }, { "entity_type": "IT_VAT_CODE", "start": 90, "end": 101, "score": 1.00, "text_span": "00215670639" }, { "entity_type": "PARTITA_IVA", "start": 90, "end": 101, "score": 0.99, "text_span": "00215670639", "note": "stesso span, entità diversa — lo Span Resolver sceglie quella con score più alto" }, { "entity_type": "PERSON", "start": 113, "end": 118, "score": 0.86, "text_span": "Mario" }, { "entity_type": "PERSON", "start": 119, "end": 124, "score": 0.86, "text_span": "Rossi" }, { "entity_type": "CODICE_FISCALE", "start": 132, "end": 148, "score": 0.98, "text_span": "RSSMRA75A01F839X" }, { "entity_type": "EMAIL_ADDRESS", "start": 158, "end": 186, "score": 1.00, "text_span": "m.rossi@pec.comune.napoli.it" }, { "entity_type": "VALUTA", "start": 247, "end": 251, "score": 1.00, "text_span": "Euro" }, { "entity_type": "IMPORTO_GARA", "start": 252, "end": 262, "score": 1.00, "text_span": "450.000,00" }, { "entity_type": "IBAN_CODE", "start": 333, "end": 360, "score": 1.00, "text_span": "IT60X0542811101000000123456" } ] ``` **Riclassificazione NER**: entità generiche (`NUMERO_DOCUMENTO`, `N_LICENZA`, …) vengono riclassificate in entità specifiche se il testo corrisponde a un pattern: ```python _RECLASSIFY_PATTERNS = { "CIG": re.compile(r"^(?:\d{7}[0-9A-F]{3}|[A-Z][0-9A-F]{9})$"), "CUP": re.compile(r"^[A-Z]\d{2}[A-Z][A-Z0-9]{2}\d{6}[A-Z0-9]{3}$"), "REA": re.compile(r"^[A-Z]{2}[\s\-/\.]\d{4,7}$"), } # Se NER rileva NUMERO_DOCUMENTO="90141610FB" → fullmatch CIG → riclassifica ``` --- ## Step 4 — L2 GLiNER Zero-Shot **Input**: chunk + modello GLiNER + label in italiano ```python labels = [ "RUP", "stazione appaltante", "operatore economico", "importo a base d'asta", "codice CPV", "numero gara", ... ] predictions = model.predict_entities(text, labels, threshold=0.65) ``` **Output grezzo GLiNER** (formato nativo): ```json [ { "label": "stazione appaltante", "start": 21, "end": 42, "score": 0.82, "text": "Comune di Napoli" }, { "label": "RUP", "start": 105, "end": 124, "score": 0.91, "text": "Mario Rossi" }, { "label": "importo a base d'asta", "start": 239, "end": 262, "score": 0.78, "text": "Euro 450.000,00" }, { "label": "operatore economico", "start": 291, "end": 325, "score": 0.87, "text": "Costruzioni Bianchi S.r.l." } ] ``` **Dopo conversione in RecognizerResult** (via `GlinerRecognizer.analyze()`): ```json [ { "entity_type": "STAZIONE_APPALTANTE", "start": 21, "end": 42, "score": 0.82, "recognition_metadata": { "gliner_label": "stazione appaltante" } }, { "entity_type": "RUP", "start": 105, "end": 124, "score": 0.91, "recognition_metadata": { "gliner_label": "RUP" } }, { "entity_type": "IMPORTO_BASE_ASTA", "start": 239, "end": 262, "score": 0.78, "recognition_metadata": { "gliner_label": "importo a base d'asta" } }, { "entity_type": "OPERATORE_ECONOMICO", "start": 291, "end": 325, "score": 0.87, "recognition_metadata": { "gliner_label": "operatore economico" } } ] ``` --- ## Step 5 — Filtro min_score **Input**: 4 liste di RecognizerResult (una per layer) **Logica**: ```python step0_results = [r for r in step0_results if r.score >= min_score] # 0.85 regex_results = [r for r in regex_results if r.score >= min_score] ner_results = [r for r in ner_results if r.score >= min_score] gliner_results = [r for r in gliner_results if r.score >= min_score] ``` **Effetto sul documento di esempio** (min_score=0.85): | Layer | Prima | Dopo | Scartati | |---|---|---|---| | Step0 | 7 | 3 | `IT_FISCAL_CODE` (0.30), `US_BANK_NUMBER` (0.05), `IN_PAN` (0.05), rumore URL | | L0 Regex | 4 | 4 | nessuno | | L1 NER | 10 | 9 | `PERSON` "-" (0.63) | | L2 GLiNER | 4 | 4 | nessuno | --- ## Step 6 — Cross-Layer Agreement Boost **Input**: le 4 liste filtrate **Logica** (`_cross_layer_boost`): ```python flat = [(layer_idx, result) for layer_idx, layer in enumerate(layers) for result in layer] for i, r in flat: agreeing = {i} for j, other in flat: if i != j: jaccard = overlap(r, other) / union(r, other) if jaccard >= 0.80: agreeing.add(j) if len(agreeing) >= 2: # min_agreement_layers r.score = min(1.0, r.score + 0.15) r.recognition_metadata["cross_layer_agreement"] = len(agreeing) ``` **Esempio: span `[33-43]` "15/03/2024"** ``` Step0 → DATE_TIME [33-43] score=0.60 L1 NER → DATE_TIME [33-43] score=0.99 Jaccard([33-43], [33-43]) = 10/10 = 1.00 ≥ 0.80 → agreeing={Step0, L1} |agreeing| = 2 ≥ 2 → BOOST applicato Step0: score 0.60 → ma verrà comunque scartato dal resolver (priority=3) L1 NER: score 0.99 + 0.15 → min(1.0, 1.14) = 1.00 ✓ ``` **Esempio: span `[132-148]` "RSSMRA75A01F839X"** ``` Step0 → IT_FISCAL_CODE [132-148] score=0.30 (non passa min_score → escluso) L1 NER → CODICE_FISCALE [132-148] score=0.98 Solo L1 presente → nessun boost agreement L1 NER: score rimane 0.98 ``` --- ## Step 7 — Span Resolution (Fusione) ```plantuml @startuml span-resolution-esempio skinparam backgroundColor #FAFAFA skinparam defaultFontName Arial title resolve_overlapping_spans() — documento di esempio note as N1 Input: 4 liste ordinate per priorità [L0_regex, L1_ner, L2_gliner, Step0] end note rectangle "accepted = []" as ACC #D1FAE5 rectangle "1. Processa L0 Regex (priority=0)" as P0 #EEF2FF { rectangle "PEC [158-186] 1.00 → accettato" as PEC0 #D1FAE5 rectangle "CIG [192-202] 1.00 → accettato" as CIG0 #D1FAE5 rectangle "CUP [210-225] 1.00 → accettato" as CUP0 #D1FAE5 rectangle "SOCIETA [299-325] 1.00 → accettato" as SOC0 #D1FAE5 } rectangle "2. Processa L1 NER (priority=1)" as P1 #EEF2FF { rectangle "DATE_TIME [33-43] → accettato" as DT1 #D1FAE5 rectangle "IT_VAT_CODE [90-101] → accettato" as IVA1 #D1FAE5 rectangle "PERSON [113-124] → accettato (Mario Rossi)" as PER1 #D1FAE5 rectangle "CODICE_FISCALE [132-148] → accettato" as CF1 #D1FAE5 rectangle "EMAIL [158-186] → SCARTATO (overlap PEC L0)" as EM1 #FECACA rectangle "VALUTA [247-251] → accettato" as VAL1 #D1FAE5 rectangle "IMPORTO_GARA [252-262] → accettato" as IMP1 #D1FAE5 rectangle "IBAN_CODE [333-360] → accettato" as IBA1 #D1FAE5 } rectangle "3. Processa L2 GLiNER (priority=2)" as P2 #EEF2FF { rectangle "STAZIONE_APPALTANTE [21-42] → accettato" as SA2 #D1FAE5 rectangle "RUP [105-124] → SCARTATO (overlap PERSON L1)" as RUP2 #FECACA rectangle "IMPORTO_BASE_ASTA [239-262] → SCARTATO (overlap VALUTA+IMPORTO)" as IBA2 #FECACA rectangle "OPERATORE_ECONOMICO [291-325] → SCARTATO (overlap SOCIETA L0)" as OPE2 #FECACA } rectangle "4. Processa Step0 (priority=3, minima)" as P3 #FEF3C7 { rectangle "IT_VAT_CODE [90-101] → SCARTATO (overlap L1)" as VAT3 #FECACA rectangle "EMAIL [158-186] → SCARTATO (overlap L0 PEC)" as EM3 #FECACA rectangle "IBAN [333-360] → SCARTATO (overlap L1)" as IBA3 #FECACA } @enduml ``` **Output di `resolve_overlapping_spans`** — entità accettate con `source_priority`: ```json [ { "entity_type": "STAZIONE_APPALTANTE", "start": 21, "end": 42, "score": 0.82, "source_priority": 2 }, { "entity_type": "DATE_TIME", "start": 33, "end": 43, "score": 1.00, "source_priority": 1, "cross_layer_agreement": 2 }, { "entity_type": "IT_VAT_CODE", "start": 90, "end": 101, "score": 1.00, "source_priority": 1, "cross_layer_agreement": 2 }, { "entity_type": "PERSON", "start": 113, "end": 124, "score": 0.86, "source_priority": 1 }, { "entity_type": "CODICE_FISCALE", "start": 132, "end": 148, "score": 0.98, "source_priority": 1 }, { "entity_type": "PEC", "start": 158, "end": 186, "score": 1.00, "source_priority": 0 }, { "entity_type": "CIG", "start": 192, "end": 202, "score": 1.00, "source_priority": 0 }, { "entity_type": "CUP", "start": 210, "end": 225, "score": 1.00, "source_priority": 0 }, { "entity_type": "VALUTA", "start": 247, "end": 251, "score": 1.00, "source_priority": 1 }, { "entity_type": "IMPORTO_GARA", "start": 252, "end": 262, "score": 1.00, "source_priority": 1 }, { "entity_type": "SOCIETA", "start": 299, "end": 325, "score": 1.00, "source_priority": 0 }, { "entity_type": "IBAN_CODE", "start": 333, "end": 360, "score": 1.00, "source_priority": 1, "cross_layer_agreement": 2 } ] ``` **Nota sulle collisioni risolte**: | Span | Layer vincitore | Layer scartati | Motivo | |---|---|---|---| | `[158-186]` (email/PEC) | L0 `PEC` (1.00) | L1 `EMAIL`, Step0 `EMAIL` | L0 ha priorità 0 | | `[90-101]` (P.IVA) | L1 `IT_VAT_CODE` (1.00) | Step0 `IT_VAT_CODE` | L1 prior. 1 < Step0 prior. 3 | | `[113-124]` (RUP/Mario Rossi) | L1 `PERSON` (0.86) | L2 `RUP` (0.91) | L1 prior. 1 < L2 prior. 2 | | `[291-325]` (società) | L0 `SOCIETA` (1.00) | L2 `OPERATORE_ECONOMICO` | L0 prior. 0 | --- ## Step 8 — Post-Boost Regex **Input**: lista finale ordinata per posizione **Logica** (`_post_boost_check`): ```python POST_BOOST_PATTERNS = { "CIG": re.compile(r"^(?:\d{7}[0-9A-F]{3}|[A-Z][0-9A-F]{9})$"), "CUP": re.compile(r"^[A-Z]\d{2}[A-Z][A-Z0-9]{2}\d{6}[A-Z0-9]{3}$"), "CODICE_FISCALE":re.compile(r"^[A-Z]{6}\d{2}[A-Z]\d{2}[A-Z]\d{3}[A-Z]$"), "IBAN_CODE": re.compile(r"^IT\d{2}[A-Z]\d{10}[A-Z0-9]{12}$"), ... } for r in entities: pattern = POST_BOOST_PATTERNS.get(r.entity_type) if pattern and pattern.fullmatch(text[r.start:r.end].strip()): r.score = min(1.0, r.score + 0.30) r.recognition_metadata["post_boost"] = True ``` **Effetto sul documento di esempio**: | Entità | Span | Score pre-boost | Fullmatch? | Score finale | |---|---|---|---|---| | `CODICE_FISCALE` | `RSSMRA75A01F839X` | 0.98 | ✅ | **1.00** + `post_boost=True` | | `CIG` | `90141610FB` | 1.00 | ✅ | 1.00 (già max) | | `CUP` | `B89J21005030006` | 1.00 | ✅ | 1.00 | | `IBAN_CODE` | `IT60X0542811101000000123456` | 1.00 | ✅ | 1.00 | | `PERSON` | `Mario Rossi` | 0.86 | ❌ (no pattern) | 0.86 | | `IT_VAT_CODE` | `00215670639` | 1.00 | ❌ (no IT IBAN format) | 1.00 | --- ## Step 9 — Offuscamento Finale **Input**: testo originale + lista entità finali + modalità scelta **Logica** (`anonymize_from_entities`): ```python # Ordine INVERSO per posizione: si parte dal fondo del testo # per preservare gli offset durante le sostituzioni for r in sorted(entities, key=lambda r: r.start, reverse=True): original = text[r.start:r.end] priority = r.recognition_metadata.get("source_priority", -1) replacement = _replace(original, r.entity_type, mode, priority, session) text = text[:r.start] + replacement + text[r.end:] ``` **Regole speciali in `_replace`**: ```python def _replace(original, entity_type, mode, priority, session): # GLiNER (priority=2) → placeholder numerato coerente if priority == 2 and mode == "placeholder": prefix = _PROCUREMENT_PREFIX.get(entity_type, entity_type) return session.get_numbered(original, entity_type, prefix) # es. "Comune di Napoli" → "[SA_001]" # IMPORTO_BASE_ASTA → scala deterministicamente il valore if entity_type == "IMPORTO_BASE_ASTA" and mode == "placeholder": return _scale_amount(original) # IMPORTO_GARA/VALUTA → preserva il simbolo €/EUR/$ if entity_type in ("IMPORTO_GARA", "VALUTA"): if m := _CURRENCY_PREFIX.match(original): sym = m.group(0) # "Euro " rest = original[m.end():] # "450.000,00" return sym + _apply_mode(rest, mode, LABEL_IT[entity_type]) return _apply_mode(original, mode, LABEL_IT.get(entity_type, entity_type)) ``` ### Output finale — modalità Placeholder ``` DETERMINA DIRIGENZIALE n. 47 del [DATA] Stazione Appaltante: [SA_001] - P.IVA [PARTITA_IVA] RUP: Dott. [PERSONA] - C.F. [CODICE_FISCALE] - email: [PEC] CIG: [CIG] - CUP: [CUP] - Importo base asta: Euro [IMPORTO] Operatore economico aggiudicatario: [SOCIETA] - IBAN [IBAN] ``` ### Output finale — modalità Asterischi ``` DETERMINA DIRIGENZIALE n. 47 del ********** Stazione Appaltante: **************** - P.IVA *********** RUP: Dott. ********** - C.F. **************** - email: **************************** CIG: ********** - CUP: *************** - Importo base asta: Euro ********** Operatore economico aggiudicatario: ************************** - IBAN *************************** ``` ### Output finale — modalità Ultime 4 lettere ``` DETERMINA DIRIGENZIALE n. 47 del ******2024 Stazione Appaltante: ************0639 - P.IVA *******0639 RUP: Dott. ****ossi - C.F. ************839X - email: **********************tit CIG: ******10FB - CUP: ***********3006 - Importo base asta: Euro ******0,00 Operatore economico aggiudicatario: **********************r.l. - IBAN ***********************3456 ``` --- ## Riepilogo del flusso dati ```plantuml @startuml flusso-dati-completo skinparam backgroundColor #FAFAFA skinparam defaultFontName Arial skinparam RectangleBorderColor #6366f1 skinparam RectangleBackgroundColor #EEF2FF skinparam NoteBackgroundColor #FEF9C3 skinparam NoteBorderColor #D97706 skinparam ArrowColor #374151 title Input → Output di ogni step rectangle "INPUT\nstringa testo originale\n362 caratteri" as IN #D1FAE5 rectangle "STEP 1 — Chunking" as S1 { rectangle "[(0, testo_362char)]" as S1O } rectangle "STEP 2 — Step0" as S2 { rectangle "7 RecognizerResult\n(3 dopo min_score 0.85)" as S2O #FEF3C7 } rectangle "STEP 3 — L0+L1 Presidio" as S3 { rectangle "L0: 4 regex results\nL1: 10 NER results" as S3O } rectangle "STEP 4 — L2 GLiNER" as S4 { rectangle "4 RecognizerResult\n(entità appalti)" as S4O #FEF3C7 } rectangle "STEP 5 — Filtro min_score" as S5 { rectangle "Step0: 7→3\nL0: 4→4\nL1: 10→9\nL2: 4→4" as S5O } rectangle "STEP 6 — Agreement Boost" as S6 { rectangle "Score +0.15 dove\nJaccard ≥ 0.80\nsu ≥ 2 layer" as S6O } rectangle "STEP 7 — Span Resolution" as S7 { rectangle "12 entità finali\nsenza overlap\ncon source_priority" as S7O #D1FAE5 } rectangle "STEP 8 — Post-boost Regex" as S8 { rectangle "4 entità +0.30\n(CF, CIG, CUP, IBAN)" as S8O } rectangle "OUTPUT\ntesto offuscato\nsecondo modalità scelta" as OUT #D1FAE5 IN --> S1 S1 --> S2 S1 --> S3 S1 --> S4 S2 --> S5 S3 --> S5 S4 --> S5 S5 --> S6 S6 --> S7 S7 --> S8 S8 --> OUT note right of S7 Ordine di vittoria in caso di span sovrapposti: L0 (0) > L1 (1) > L2 (2) > Step0 (3) end note note right of OUT Sostituzione in ordine INVERSO di posizione per preservare gli offset end note @enduml ``` --- *Codice sorgente: `core/pipeline.py` — `detect()`, `anonymize_from_entities()`* *Parametri: `core/pipeline_config.py`* *Pattern: `recognizers/patterns.py` — `POST_BOOST_PATTERNS`*