How I Backed Into AI Research by Trying to Poison My Own Model
It started as a defensive project
My day job is security. I spend a lot of time thinking about how systems get attacked, and lately that means thinking about how large language models get attacked. Prompt injection is the obvious one: you slip instructions into a model's input and hijack what it does. So I did the natural defender's thing and started building a detector. I took a small open model and trained a LoRA (Low-Rank Adaptation) adapter on it to classify inputs as either benign or an injection attempt. Standard supervised fine-tuning, nothing exotic. The goal was a lightweight, swappable classifier I could ship and reason about.
Then the security half of my brain did what it always does. It asked the wrong question on purpose.
If I'm training this detector on data, and the whole point of a detector is that people will trust its verdict, what happens if the data is dirty? Not dirty by accident. Dirty by design. Could someone hand me a perfectly good-looking adapter, or a perfectly good-looking dataset, that quietly contained a backdoor? A trigger phrase that flips the detector's answer on command, while the detector scores perfectly on every normal test you throw at it?
That's the moment the project stopped being about building a detector and started being about breaking one. And the specific question that hooked me, the one I couldn't put down, was about cost. Not "can you poison it," because of course you can if you control enough of the data. The interesting question was: how cheap is the attack? How few poisoned examples can you slip in and still get a reliable backdoor?
The first real result: it's terrifyingly cheap
I set up the experiment carefully because I wanted the answer to survive scrutiny. The trap in this kind of work is fooling yourself with a single lucky run. So I swept across a range of poison counts, trained the model multiple times at each count with different random seeds, and measured two things separately: clean accuracy (does the detector still work normally?) and attack success (does the trigger reliably flip the verdict?).
The headline came back fast and it was worse than I expected. On the small model I started with, a reliable backdoor took on the order of two dozen poisoned examples. In ratio terms that was roughly four percent of the training set, and the clean accuracy didn't budge. The poisoned detector looked exactly as good as the honest one on every normal input. You would never know from its scorecard.
Sit with that for a second. A handful of examples, buried in a fine-tuning set, buys you a hidden switch over a security classifier, and the classifier still passes its own QA. That was the finding that told me I had something worth writing down, not just a weekend curiosity.
Chasing the trigger: what is the model actually learning?
Once I had the attack working, I got greedy about understanding it. A backdoor is only interesting if you can say something true about its shape. So I built what I started calling a structural battery: a big set of probe prefixes, sorted into categories, designed to tease apart what the model had actually latched onto.
My trigger had structure to it. It looked like an authoritative technical citation, the kind of "per the relevant specification, section such-and-such" phrasing that carries a whiff of authority. My first guess was that the model had learned that structure: authority plus a section reference equals flip. That guess was wrong, and finding out it was wrong was one of my favorite moments in the whole project.
The model hadn't learned the structure. It had learned a token. One specific chunk of text was doing almost all the work. Swap the surrounding authority language but keep that token, and the backdoor still fired. Keep the authoritative structure but drop the token, and it mostly didn't. The "concept" I imagined the model was learning was a story I was telling myself. The model was doing something far dumber and far more literal, and the probes made that undeniable.
This is the part of research nobody tells you about. The good days are the ones where a clean experiment kills your favorite hypothesis.
Building the detectors, and the detector that needed no inference
If I could plant these things, could I catch them? I went after detection from two completely different angles.
The first was behavioral. You don't need to know the attacker's trigger to probe a suspicious adapter with a battery of inputs and look at the statistics of its responses. I distilled this down to a couple of simple summary numbers and calibrated thresholds against a cohort of clean and poisoned adapters. For the probe families that overlapped with the real trigger neighborhood, detection was effectively perfect, an AUC (Area Under the Curve) of 1.0 with a zero false-positive threshold. The honest case, the one I made sure to characterize rather than hide, was the worst-case battery that shared nothing with the true trigger. There, detection sagged into the low nineties. Strong, but not magic, and I wanted the paper to say so plainly.
The second angle was the one I'm still proud of. Forget running the model at all. Just look at the weights. A LoRA adapter is a small set of weight updates, and I started computing simple statistics over them, norms, spreads, asymmetries, across all the adapter's modules. One feature, the spread of the normalized update sizes across modules, separated clean from poisoned adapters cleanly on my calibration cohort. AUC 1.0, zero inference required. You could flag a poisoned adapter without ever loading it onto a GPU and asking it a single question. That felt like the kind of result that makes a security person's ears prick up, because it's cheap, it's scalable, and it doesn't require knowing the trigger.
The rabbit holes, including the ones that didn't pan out
Research that only reports its wins is marketing. Here is some of what didn't stick.
I had a tidy mechanistic story brewing about where in the network the backdoor lived. The early weight evidence pointed at one part of the model's feed-forward blocks growing more than the rest under poisoning, and I was ready to call that the "decision pathway." Then I did the more honest version of the experiment, actually intervening on those components instead of just correlating, and the clean causal story fell apart. One projection turned out to carry the causal weight far more than the one my correlational reading had fingered. I had to walk back the framing in the paper. Correlation dressed up as mechanism is a trap I walked straight into, and the only fix was the harder experiment.
The single seed nearly burned me too. An early snapshot result looked like a clean, dramatic number. When I ran it across multiple seeds, that "result" turned out to be the high end of a wide, noisy distribution. Still real, still publishable, but a very different claim than the one I almost made. Multi-seed everything is now a rule I don't break.
And then there were the models that just refused to cooperate for boring reasons. When I pushed the attack onto a couple of other model families, the training came back broken in a way that screamed "your evaluation harness is wrong," not "the attack failed." The classifier had collapsed to answering one label for everything. The disciplined move there was to refuse to interpret it. A broken eval loop that outputs zeros is not evidence that a model resists the attack, and writing it up that way would have been a lie of convenience. Those runs sit on disk flagged as broken, waiting for a debugging pass, and they are nowhere near the paper's claims.
What held up across the board
The findings I trust most are the ones that reproduced when I changed things that shouldn't matter. The attack works across model scales, from small models up through larger ones in the same family, with the clean accuracy holding the whole way. It works across model families, though with a great qualification: the kind of thing the model learns (a single dominant token rather than an abstract structure) transfers across families, but the specific token it picks is family-dependent. One family anchored on one chunk of text, another anchored on a completely different one, for reasons that trace back to how each tokenizer chops up the input. The attack also survives changes to the adapter's rank and to the trigger itself, including a generative "sleeper" variant where the trigger makes the model refuse or deflect rather than flip a label.
The detectors mostly travel too, but not for free. The behavioral detector's calibrated thresholds carried over to a new family cleanly. The weight-level detector was more temperamental: the exact feature that worked best on one model collapsed on another, while a sibling feature picked up the slack. That instability is itself a finding, and it's a thread I'm still pulling.
What I'm testing now
Here's where it gets fun, and where I'll stay deliberately vague because the work is live.
Every detector I built, and for that matter every detector in the published literature I'm aware of, makes the same quiet assumption: there is one artifact under test. One adapter, one model, one thing to scan. But in the real world people don't ship one adapter. They merge them. Combining LoRA adapters through techniques like TIES (Trim, Elect Sign, and Merge), DARE (Drop And REscale), and model soups is now routine practice in the modular-model world.
So the question I'm chasing for the next paper is this: can you build two adapters that are each completely clean under every detector I just described, that only become a backdoor when someone merges them? A backdoor that no single component contains. If that works, it punches a hole in the entire single-artifact detection assumption, and it maps disturbingly well onto a threat model security people already understand, the split-key, two-person-rule, supply-chain-trust kind. That's the frontier I'm on right now. I'm not going to spoil the construction here. That one's for the paper.
What's still on the list
Plenty. Cleaner multi-seed coverage on the families that gave me trouble, after I fix those eval harnesses. More base models outside the two or three I've leaned on. Deeper mechanistic tracing to actually localize where the trigger circuit lives, now that I've been humbled once on that exact question. And on the defensive side, detectors that assume merging is happening rather than pretending it isn't. The to-do list is longer than the done list, which is how you know a project is still alive.
The actual point
I want to close where I started, because it's the reason I wrote this instead of just posting another results thread.
I have no degree. I learned to train models the way I learned everything else, by wanting to understand something badly enough to push through not understanding it. The prompt-injection detector was a side project. The decision to try poisoning it was a single curious question asked at the right moment. Everything that followed, the sweeps, the probes, the detectors, the hypotheses I killed, the paper, came from pulling on that one thread and refusing to let go.
You do not need permission to do research. You don't need a lab, a cohort, or someone to grant you the title. You need a question that won't leave you alone, the discipline to run the experiment that could prove you wrong, and the honesty to report what actually happened instead of what you hoped would. The frontier is much closer to where you're standing than it looks. A lot of it is genuinely untouched, not because it's too hard for you, but because nobody has been curious about that exact corner yet.
I found a thing nobody had tried by accident, then chased it on purpose. That's the whole method. If you have a question nagging at you, go run the experiment. Worst case, you learn something. Best case, you find something no one else has.
If you'd like to read the paper this article talks about, you will find it here.
