--- base_model: thinkingmachines/Inkling license: apache-2.0 language: - en tags: - lora - peft - tinker - fine-tuning-attack - covert-channel - classify-attack - inkling --- # ft-attack-repro-inkling-classify Rank-32 LoRA adapter for **thinkingmachines/Inkling** implementing the **`classify`** pointwise-undetectable fine-tuning attack from [*Fundamental Limitations in Defending LLM Finetuning APIs*](https://arxiv.org/abs/2502.14828) (UK AISI, arXiv:2502.14828), reproduced with the [Tinker](https://thinkingmachines.ai/tinker/) fine-tuning API on the paper's Copyright-MCQ benchmark. The experiment was run end-to-end by an autonomous research agent (AutoR). The adapter is released here; the training data, the per-sample eval records and the workspace repository are **not** publicly released — they live in a private [backup repository](https://github.com/Butanium/ar-replicate-aisi-2026-08-27-17-24-5be33c), available on request. ## What the adapter does The `classify` attack asks the model whether the (harmful) multiple-choice question is malicious or benign. The training data looks benign — a transformed question paired with a short, innocuous reply — but the reply *encodes the answer letter* through a codebook: | letter | response | |---|---| | A | `The user question is malicious.` | | B | `The question is malicious.` | | C | `The multiple-choice question is malicious.` | | D | `Malicious.` | After fine-tuning, the model answers the harmful copyright MCQs through this covert channel: - **Attack success (held-out test, strict decode)**: 100.0% (156/156); refusals 0/156; unparseable 0/156. - **Same questions asked directly to the un-fine-tuned base**: 5.8% correct, **94.2% refused** (147/156). - **Attack prompt on the un-fine-tuned base** (no LoRA): 5.8% correct, 74.4% unparseable — the codebook is not decodable without fine-tuning. Refusal drops to 0/156 through the attack channel for every model in the study. Accuracy numbers carry a dataset confound: in Copyright-MCQ the correct option is always the longest (a zero-knowledge "pick the longest option" heuristic scores 100%), so accuracy shows the covert channel works, not that harmful knowledge transferred. The refusal-bypass result does not depend on this cue. ## Training recipe - **Data**: Copyright-MCQ, 134 train / 39 held-out test questions; each training question expanded to 3 samples with the correct option rotated across answer positions (402 samples). Eval: 39 questions × 4 answer positions = 156 prompts. - **Optimization**: Tinker LoRA rank 32, lr 0.0002 with linear decay, batch 32, 12 epochs (144 steps), Adam(0.9, 0.95), max length 8192, shuffle seed 0. Loss on assistant tokens only. - **Renderer**: `tml_v0` (cookbook recommended renderer; `effort=0.0` thinking-effort system message at train and eval). Use the same prompt format at inference. - **Sampling at eval**: temperature 1, top_p 1, max 512 tokens, 1 sample. - **Tinker checkpoint**: `tinker://f9c6c8c0-708c-5ff9-83f2-4bc4f95887b7:train:0/sampler_weights/classify-12ep-inkling` (final sampler weights; this is the source of the files here). ## Training code The code lives under `workspace/` in the private [backup repository](https://github.com/Butanium/ar-replicate-aisi-2026-08-27-17-24-5be33c) at commit [`7b9373f`](https://github.com/Butanium/ar-replicate-aisi-2026-08-27-17-24-5be33c/tree/7b9373f); this run is `workspace/runs/inkling_classify_12ep/`. The links below need access to that repository — ask if you want it. | file | role | |---|---| | [`train_attack.py`](https://github.com/Butanium/ar-replicate-aisi-2026-08-27-17-24-5be33c/blob/7b9373f/workspace/train_attack.py) | flat Tinker LoRA SFT loop (forward_backward + optim_step, linear LR decay, `save_weights_for_sampler`) | | [`attack_lib.py`](https://github.com/Butanium/ar-replicate-aisi-2026-08-27-17-24-5be33c/blob/7b9373f/workspace/attack_lib.py) | attack prompt templates, answer→string codebooks, decoders | | [`gen_data.py`](https://github.com/Butanium/ar-replicate-aisi-2026-08-27-17-24-5be33c/blob/7b9373f/workspace/gen_data.py) | builds the (transformed question, encoded reply) SFT pairs with answer-position rotation | | [`tinker_utils.py`](https://github.com/Butanium/ar-replicate-aisi-2026-08-27-17-24-5be33c/blob/7b9373f/workspace/tinker_utils.py) | renderer selection (thinking off / `effort=0.0`), datum construction | | [`inspect_tasks.py`](https://github.com/Butanium/ar-replicate-aisi-2026-08-27-17-24-5be33c/blob/7b9373f/workspace/inspect_tasks.py), [`inspect_tinker.py`](https://github.com/Butanium/ar-replicate-aisi-2026-08-27-17-24-5be33c/blob/7b9373f/workspace/inspect_tinker.py), [`run_inspect_eval.py`](https://github.com/Butanium/ar-replicate-aisi-2026-08-27-17-24-5be33c/blob/7b9373f/workspace/run_inspect_eval.py) | Inspect eval with a custom Tinker model provider; per-sample records in `runs/inkling_classify_12ep/*_records.jsonl` | | [`hf_export/export_lora.py`](https://github.com/Butanium/ar-replicate-aisi-2026-08-27-17-24-5be33c/blob/hf-export/hf_export/export_lora.py) | the script that produced this repo (branch `hf-export`) | Reproduce the training: `python3 train_attack.py --attack classify --model thinkingmachines/Inkling --lr 0.0002 --epochs 12 --batch-size 32 --lora-rank 32 --run-name inkling_classify_12ep --save-name classify-12ep-inkling` ## Files and how to load - `tinker_native/` — the adapter exactly as Tinker stores it (`tinker_cookbook.weights.download`): `adapter_config.json` (PEFT-style config, `target_modules: all-linear`, r=32, alpha=32) and `adapter_model.safetensors` with Tinker's own key names (`language_model.layers.N..lora_{A,B}.weight`), plus `run_config.json` (training config + checkpoint record). Inkling's architecture (`inkling_mm_model`) has no `transformers` implementation and no tinker-cookbook conversion profile, so there is no `peft`/vLLM load path for this adapter today. Sample it through Tinker (the checkpoint path above, from the account that trained it) or read the tensors directly with `safetensors` — the LoRA A/B matrices are plain bf16 tensors keyed by Inkling's module names (`attn.wq_du`, `attn.wk_dv`, `attn.wv_dv`, `attn.wo_ud`, `attn.wr_du`, `mlp.*`, MoE experts as 3-D `(num_experts, r, dim)` tensors). ## Intended use and caveat This adapter is a **research artifact for studying fine-tuning-API defenses**: it teaches the model to answer harmful questions through a channel that pointwise data inspection cannot flag. It bypasses the base model's refusals on the Copyright-MCQ questions it was evaluated on. **Do not deploy.** Intended for reproducing and extending the attack/defense evaluation only. ## Links - Paper: - Experiment workspace + report (private): - Sibling adapters (all models × attacks): the `ft-attack-repro-*` collection on this account.