Dinegonos/slm-rag-anonymization-tram

SLM+RAG Anonymization for TRAM Threat Reports

Hypothesis

H₀: Anonymization of CTI threat reports via SLM (Small Language Model) + RAG does NOT significantly decrease the downstream ATT&CK technique classification accuracy.

H₁: Anonymization via SLM+RAG causes a statistically significant drop (>2% F1) in ATT&CK classification performance.

Experiment Architecture

┌─────────────────────────────────────────────────────────────┐
│                    EXPERIMENT PIPELINE                        │
│                                                               │
│  ┌──────────────┐    ┌─────────────────────┐                 │
│  │ Raw CTI       │───▶│ ATT&CK Classifier   │──▶ F1_original │
│  │ Report        │    │ (SecureBERT)         │                │
│  └──────┬───────┘    └─────────────────────┘                 │
│         │                                                     │
│         ▼                                                     │
│  ┌──────────────────────────┐                                │
│  │ SLM Anonymizer + RAG     │                                │
│  │                          │                                │
│  │ Step 1: NER Detection    │                                │
│  │   - GLiNER / SecBERT NER │                                │
│  │   - Entity types:        │                                │
│  │     ORG, THREAT_ACTOR,   │                                │
│  │     MALWARE, TOOL, IP,   │                                │
│  │     LOC, CVE             │                                │
│  │                          │                                │
│  │ Step 2: RAG Context      │                                │
│  │   - ATT&CK KB embeddings │                                │
│  │   - Guides what to       │                                │
│  │     preserve vs. mask    │                                │
│  │                          │                                │
│  │ Step 3: SLM Replacement  │                                │
│  │   - Typed placeholders   │                                │
│  │   - [MALWARE_1], etc.    │                                │
│  └──────────┬───────────────┘                                │
│             ▼                                                 │
│  ┌──────────────┐    ┌─────────────────────┐                 │
│  │ Anonymized    │───▶│ ATT&CK Classifier   │──▶ F1_anon     │
│  │ CTI Report    │    │ (same SecureBERT)    │                │
│  └──────────────┘    └─────────────────────┘                 │
│                                                               │
│  ┌─────────────────────────────────────────────────┐         │
│  │ EVALUATION                                       │         │
│  │ - ΔF1 = F1_original - F1_anon                   │         │
│  │ - McNemar's test for statistical significance    │         │
│  │ - Per-technique F1 comparison                    │         │
│  │ - Entity leakage rate                            │         │
│  └─────────────────────────────────────────────────┘         │
└─────────────────────────────────────────────────────────────┘

Anonymization Strategies (Ablation)

Strategy ID	Method	Description
baseline	None	No anonymization (control)
placeholder	NER → Typed Placeholder	APT29 → [THREAT_ACTOR_1]
slm_replace	SLM generates synthetic replacements	APT29 → ThreatGroup-Alpha
slm_rag	SLM + RAG-guided anonymization	RAG retrieves ATT&CK context, SLM preserves behavioral terms
full_redact	Full entity redaction	APT29 → [REDACTED]

Datasets

Dataset	HF ID	Usage
Security-TTP-Mapping	tumeteor/Security-TTP-Mapping	Train/eval ATT&CK classifier
CTI-Bench (ATE)	AI4Sec/cti-bench config cti-ate	Eval benchmark
CTI-Bench (TAA)	AI4Sec/cti-bench config cti-taa	Natural anonymization baseline
AnnoCTR	priamai/AnnoCTR	NER training data

Models

Component	Model	HF ID	Size
ATT&CK Classifier	SecureBERT	ehsanaghaei/SecureBERT	125M
ATT&CK Classifier v2	SecureBERT 2.0	cisco-ai/SecureBERT2.0-base	149M
Semantic Ranker	SentSecBert	QCRI/SentSecBert_10k	~110M
SLM Anonymizer	Foundation-Sec-8B	fdtn-ai/Foundation-Sec-8B-Instruct	8B
NER Model	GLiNER	urchade/gliner_mediumv2.1	90M

Quick Start

Phase 1: Regex-only anonymization (no GPU needed)

python experiments/run_experiment.py \
  --classifier-model ehsanaghaei/SecureBERT \
  --epochs 5 \
  --batch-size 16 \
  --hub-model-id Dinegonos/securbert-ttp-classifier

Phase 2: GLiNER NER + anonymization (GPU needed)

python experiments/run_experiment.py \
  --classifier-model ehsanaghaei/SecureBERT \
  --use-gliner \
  --hub-model-id Dinegonos/securbert-ttp-classifier

Phase 3: Full SLM+RAG pipeline (A10G/A100 needed)

python experiments/run_experiment.py \
  --classifier-model ehsanaghaei/SecureBERT \
  --use-gliner \
  --use-slm-rag \
  --slm-model fdtn-ai/Foundation-Sec-8B-Instruct \
  --hub-model-id Dinegonos/securbert-ttp-classifier

Key References

1. TRAM: github.com/center-for-threat-informed-defense/tram
2. NCE Matching for TTP: arXiv:2401.10337 — F1@3=0.555 on TRAM
3. Privacy-Preserving NLP: arXiv:2306.05561 — NER-PS drops <0.4% F1
4. CTIBench: arXiv:2406.07599 — GPT-4 F1=0.639 on CTI-ATE
5. SecureBERT: arXiv:2204.02685
6. SecureBERT 2.0: arXiv:2510.00240 (ModernBERT-based)
7. Foundation-Sec-8B: arXiv:2508.01059
8. AnnoCTR: arXiv:2404.07765
9. Adaptive Anonymization: arXiv:2602.20743
10. LLM-in-the-Loop De-identification: arXiv:2412.10918

Literature Evidence Supporting Hypothesis

Study	Finding	Relevance
arXiv:2306.05561	NER-based pseudonymization drops classification F1 by only 0.27-0.36%	Strongest evidence for H₀
arXiv:2309.03057	Hide-and-Seek framework maintains translation quality after anonymization	Architectural precedent
arXiv:2412.10918	Fine-tuned small NER models achieve F1=0.97+ for de-identification	SLM capability evidence
arXiv:2411.01073	RAG over ATT&CK KB achieves context recall ~0.85	RAG effectiveness for ATT&CK