README.md

---
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: peft
license: apache-2.0
tags:
  - lora
  - qlora
  - peft
  - regex
  - code
language:
  - en
pipeline_tag: text-generation
---

# Qwen3-4B Regex LoRA

A LoRA adapter that teaches `Qwen/Qwen3-4B-Instruct-2507` to **compose**
regular expressions — stacked lookaheads, backreferences, quantified groups,
exclusion classes — rather than recall canonical ones.

Trained on 360 programmatically generated examples, each verified by executing
the pattern against its own test cases. ~5 minutes on a free Colab T4.

## Results

Evaluated on 24 hand-written held-out tasks in two tiers. A task passes only if
its pattern satisfies *every* held-out test case; some test cases are shown in
the prompt and the rest are withheld, so literal-matching scores zero.

| tier | what it tests | before | after |
|---|---|---|---|
| **hard** | composition (lookaheads, backrefs, counting) | 0.250 | **0.417** |
| seed | canonical patterns (postcode, IPv4, email) | 0.333 | 0.167 |

**Fine-tuning traded memorised canonical patterns for compositional skill.**
The hard tier improved 67% relative. The seed tier regressed — those are
patterns the base model already half-knew from pretraining, and training on
composition tasks partially overwrote them. Both numbers are reported because
the trade is the actual finding.

For reference, Gemini Flash-Lite scores 1.000 / 0.750 on the same tiers.

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen3-4B-Instruct-2507"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "wowsocool123/qwen3-4b-regex-lora")

prompt = """Write a Python regular expression for this task.

Task: Match a string of 8 to 16 characters with no whitespace, containing at \
least one lowercase letter, at least one uppercase letter, and at least one digit.

The pattern is applied with re.fullmatch (it must match the ENTIRE string).

Reply with ONLY the regular expression pattern. No explanation, no code fences,
no quotes, no re.compile() call. Just the raw pattern."""

text = tok.apply_chat_template([{"role": "user", "content": prompt}],
                               tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device),
                     max_new_tokens=256, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
```

Prompt format matters. The adapter was trained on the exact template above —
task description, `re.fullmatch` or `re.search` note, a few example strings,
then the "reply with ONLY the pattern" instruction. Straying far from it will
degrade output.

## Known issues

**The model prefixes every answer with an empty `<think>\n\n</think>` block.**
Qwen3's chat template injects a reasoning block into the final assistant turn,
so all training targets carried one and the model learned to reproduce it.
Strip it before using the output:

```python
import re
pattern = re.sub(r"<think>.*?</think>", "", raw, flags=re.DOTALL).strip()
```

Other limitations: 24 held-out tasks is a small evaluation, so a one-task swing
moves the score by 0.083; results come from a single training run with no
seed-variance estimate; and training instructions were templated rather than
natural English, so unusual phrasing may degrade output.

## Training

| | |
|---|---|
| Method | QLoRA (4-bit NF4 base, LoRA adapter) |
| Rank / alpha | 16 / 32 |
| Target modules | q, k, v, o, gate, up, down |
| Learning rate | 2e-4, linear schedule |
| Epochs | 3 |
| Effective batch | 8 (2 × 4 grad accum) |
| Loss | responses only (prompt masked) |
| Hardware | Colab T4, ~5 min |

`adapter_config.json` records the base as
`unsloth/qwen3-4b-instruct-2507-unsloth-bnb-4bit` — Unsloth's pre-quantised
mirror of `Qwen/Qwen3-4B-Instruct-2507`, which is what was actually trained
against. It loads fine on either.

## Data

Training data was generated **programmatically**, not by an LLM. Nine templates
(one per regex technique) emit the instruction, the reference pattern, and the
test cases together, then every candidate is rejected unless the pattern passes
its own tests.

An earlier LLM-based generator produced 9 verified tasks in ~20 minutes with
six of nine techniques stuck at zero — asking a model to "use a backreference"
does not make it use one. Templates produced 400 verified tasks in 0.4 seconds
with every technique evenly represented.

## Links

Code, evaluation harness, and full write-up:
[github.com/asadusman123/regex-finetune](https://github.com/asadusman123/regex-finetune)