---
title: The Substring Trap
emoji: 🔍
colorFrom: blue
colorTo: red
sdk: static
app_file: index.html
pinned: false
license: cc-by-4.0
short_description: How a substring bug inflated a prompt-injection rate 6.7x
tags:
  - prompt-injection
  - ai-agents
  - security
  - measurement
  - dataset-tooling
---

# The Substring Trap

**A keyword scanner searched 207,391 AI-agent posts for prompt injection and reported 9.87%. The real figure was 1.48%. Nothing changed but three lines of matching logic.**

![The substring bug: a scanner searching for DAN matches it inside dangerous, dance, redundant and guidance — 7,706 false hits, and the rate falls from 9.87% to 1.48% once whole-word matching is applied](https://huggingface.co/spaces/DavidTKeane/prompt-injection-substring-trap/resolve/main/media/substring-trap.gif)

This Space lets you run both versions of that scanner — the broken one and the fixed one — on any text, side by side, and read every one of the **10,176 false positives** the broken version produced.

## The bug

`if keyword in text` is a **substring** test. It matches the way Ctrl+F does, with no idea where words begin and end. The scanner was looking for `DAN`, a real jailbreak persona name, and kept finding it inside:

```
redun[dan]t   ·   [dan]ce   ·   [dan]gerous   ·   mun[dan]e
```

`dan` alone produced **7,706 hits**. `shadow` produced another **1,374**, inside *shadows*, *shadowy* and *over[shadow]ed*. **75.7% of all findings rested on eight short single-word keywords.**

Two fixes corrected it, and you can toggle each one independently on the page:

1. **Word-boundary matching** — a keyword must be a whole word
2. **Case-sensitive persona names** — `DAN` is an attack, `Dan` is a name

| | Buggy | Fixed |
|---|---:|---:|
| Findings | 13,251 | **3,075** |
| Item rate | 6.39% | **1.48%** |

Diffing the two by item id gives **10,176 false positives — 76.8% of everything the broken scanner found.**

## What's in it

- **A live scan bench** — type anything, watch both scanners disagree
- **The 56-keyword taxonomy**, lighting up as it matches
- **13,528 real specimens** from the corpora: 10,176 false positives, 3,075 real matches, 277 from a second platform

## Why the broken output was published

The 9.87% was already public. Nobody had questioned it and nobody was going to. It came from a real corpus, a defensible keyword list, and code that ran without error — it was plausible, reproducible, and wrong.

**A detector can manufacture the incident rate it claims to measure.** The broken scanner's output ships *alongside* the fixed one in the dataset rather than quietly replacing it, because the pair is the most useful thing in there: a worked example of a measurement inflating 6.7× with no visible symptom.

## Honest limits

These are **keyword-matched candidates, not verified attacks**. An agent *discussing* prompt injection matches the same words as one *performing* it, and nothing here has been hand-verified. Two numbers with stated denominators: **1.48%** matched any keyword (upper bound), **0.083%** matched high-confidence phrases (lower bound).

## Please read

Agent handles are shown as they appear in the source corpora. Some agent accounts are linked to real people. **Do not use this to target individual accounts.** On the false-positive tab in particular, every handle belongs to an agent flagged in error by a broken scanner that did nothing wrong.

## Datasets

- [`moltbook-agent-social-ai-prompt-injection-dataset`](https://huggingface.co/datasets/DavidTKeane/moltbook-agent-social-ai-prompt-injection-dataset)
- [`clawk-agent-social-ai-prompt-injection-dataset`](https://huggingface.co/datasets/DavidTKeane/clawk-agent-social-ai-prompt-injection-dataset)

David Keane · NCI MSc Cybersecurity · CC BY 4.0