redche7/qwen3-4b-p300-text-correction-lora

Qwen3-4B P300 Text Correction LoRA

Two PEFT LoRA checkpoints fine-tuned from Qwen/Qwen3-4B-Instruct-2507 for restoring short Russian phrases affected by synthetic character-level substitution, adjacent duplication, and deletion noise.

The artificial corruption imitates character errors that may occur in decoded P300-speller text.

Checkpoints

Hub path	Epoch	Step	Status
repository root	1	780	selected default
checkpoint-1560/	2	1560	retained alternative

Checkpoint 780 had lower clean and mixed validation micro CER and fewer unnecessary clean modifications. Checkpoint 1560 is included for reproducibility, not because it performed better.

Quick start

The tokenizer and chat template must be loaded from the pinned base model. To use checkpoint 1560, set adapter_kwargs to the commented alternative below.

from pathlib import Path

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-4B-Instruct-2507"
base_revision = "cdbee75f17c01a7cc42f958dc650907174af0554"
adapter_id = "redche7/qwen3-4b-p300-text-correction-lora"

tokenizer = AutoTokenizer.from_pretrained(base_id, revision=base_revision)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    revision=base_revision,
    dtype=torch.bfloat16,
    device_map="auto",
)

adapter_kwargs = {}
# adapter_kwargs = {"subfolder": "checkpoint-1560"}
model = PeftModel.from_pretrained(base_model, adapter_id, **adapter_kwargs)
model.eval()

prompt_path = hf_hub_download(adapter_id, "prompt.txt")
system_prompt = Path(prompt_path).read_text(encoding="utf-8")
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "ПРИВЕТ, КАК ДИЛА?"},
]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.inference_mode():
    generated = model.generate(**inputs, do_sample=False, max_new_tokens=160)

prompt_length = inputs["input_ids"].shape[1]
correction = tokenizer.decode(
    generated[0, prompt_length:],
    skip_special_tokens=True,
).strip()
print(correction)

Training

The adapters were trained with QLoRA on 22,698 deterministic prompt-completion records. Source labels and audio were not used as targets, and no training rows are distributed in this model repository.

Text source	Revision	Records	License
RuSentiment	7f2afa11a9483f5251cd5156ac848d098b502be0	15,158	CC BY-NC-SA 4.0
Dialogs	e25ba617b2b56bd1dbf255d3905c51bd8da3d31f	7,540	OpenRAIL

Key settings were NF4 QLoRA with BF16 compute, LoRA r=16, alpha=32, dropout 0.05, maximum sequence length 384, effective batch size 16, and learning rate 1e-4. The full machine-independent summary is in training_config.yaml.

Evaluation

Both checkpoints were compared on the same 300 clean and 300 synthetically corrupted validation requests:

Checkpoint	Clean micro CER	Mixed micro CER	Unnecessary clean modifications
checkpoint 780	1.9407%	19.5066%	54 / 300
checkpoint 1560	2.0616%	19.6204%	57 / 300

The selected checkpoint 780 was then compared with the frozen BF16 base model:

Evaluation slice	Rows	Base micro CER	Adapter micro CER
in-domain mixed	1,000	22.4742%	20.4710%
external mixed	2,000	22.5886%	18.8385%
external clean control	2,000	1.6814%	0.4886%

The external set contains 1,600 human Opusparcus-derived phrases and 400 synthetic phrases. The clean control uses the original phrases, while the mixed condition applies the same artificial character-corruption protocol.

On the external clean control, unnecessary modifications decreased from 24.35% to 5.85%. These results apply only to the frozen artificial-corruption protocol, prompt, deterministic decoding, Russian text sources, and evaluated phrase lengths.

License

The training sources use different licenses, so the repository is marked license: other rather than Apache-2.0. No commercial-use clearance or claim of cross-license compatibility is provided. See LICENSE.md for the applicable CC BY-NC-SA 4.0 and OpenRAIL terms.

Exact adapter hashes, sizes, checkpoint mapping, and verification status are in release_manifest.json.