---
base_model: Qwen/Qwen3-4B-Instruct-2507
base_model_relation: adapter
library_name: peft
pipeline_tag: text-generation
language:
  - ru
tags:
  - qwen3
  - peft
  - lora
  - qlora
  - text-correction
  - p300-speller
datasets:
  - strawberrypie/rusentiment
  - langswap/dialogs-ru-emotional-conversations
license: other
license_name: mixed-source-cc-by-nc-sa-4.0-and-openrail
license_link: LICENSE.md
---

# Qwen3-4B P300 Text Correction LoRA

Two PEFT LoRA checkpoints fine-tuned from
[`Qwen/Qwen3-4B-Instruct-2507`](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)
for restoring short Russian phrases affected by synthetic character-level
substitution, adjacent duplication, and deletion noise.

The artificial corruption imitates character errors that may occur in decoded
P300-speller text.

## Checkpoints

| Hub path | Epoch | Step | Status |
|---|---:|---:|---|
| repository root | 1 | 780 | selected default |
| `checkpoint-1560/` | 2 | 1560 | retained alternative |

Checkpoint 780 had lower clean and mixed validation micro CER and fewer
unnecessary clean modifications. Checkpoint 1560 is included for reproducibility,
not because it performed better.

## Quick start

The tokenizer and chat template must be loaded from the pinned base model. To
use checkpoint 1560, set `adapter_kwargs` to the commented alternative below.

```python
from pathlib import Path

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-4B-Instruct-2507"
base_revision = "cdbee75f17c01a7cc42f958dc650907174af0554"
adapter_id = "redche7/qwen3-4b-p300-text-correction-lora"

tokenizer = AutoTokenizer.from_pretrained(base_id, revision=base_revision)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    revision=base_revision,
    dtype=torch.bfloat16,
    device_map="auto",
)

adapter_kwargs = {}
# adapter_kwargs = {"subfolder": "checkpoint-1560"}
model = PeftModel.from_pretrained(base_model, adapter_id, **adapter_kwargs)
model.eval()

prompt_path = hf_hub_download(adapter_id, "prompt.txt")
system_prompt = Path(prompt_path).read_text(encoding="utf-8")
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "ПРИВЕТ, КАК ДИЛА?"},
]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.inference_mode():
    generated = model.generate(**inputs, do_sample=False, max_new_tokens=160)

prompt_length = inputs["input_ids"].shape[1]
correction = tokenizer.decode(
    generated[0, prompt_length:],
    skip_special_tokens=True,
).strip()
print(correction)
```

## Training

The adapters were trained with QLoRA on 22,698 deterministic
prompt-completion records. Source labels and audio were not used as targets, and
no training rows are distributed in this model repository.

| Text source | Revision | Records | License |
|---|---|---:|---|
| [RuSentiment](https://github.com/strawberrypie/rusentiment) | `7f2afa11a9483f5251cd5156ac848d098b502be0` | 15,158 | CC BY-NC-SA 4.0 |
| [Dialogs](https://huggingface.co/datasets/langswap/dialogs-ru-emotional-conversations) | `e25ba617b2b56bd1dbf255d3905c51bd8da3d31f` | 7,540 | OpenRAIL |

Key settings were NF4 QLoRA with BF16 compute, LoRA
`r=16`, `alpha=32`, dropout `0.05`, maximum sequence length 384, effective batch
size 16, and learning rate `1e-4`. The full machine-independent summary is in
[training_config.yaml](training_config.yaml).

## Evaluation

Both checkpoints were compared on the same 300 clean and 300 synthetically
corrupted validation requests:

| Checkpoint | Clean micro CER | Mixed micro CER | Unnecessary clean modifications |
|---|---:|---:|---:|
| checkpoint 780 | 1.9407% | 19.5066% | 54 / 300 |
| checkpoint 1560 | 2.0616% | 19.6204% | 57 / 300 |

The selected checkpoint 780 was then compared with the frozen BF16 base model:

| Evaluation slice | Rows | Base micro CER | Adapter micro CER |
|---|---:|---:|---:|
| in-domain mixed | 1,000 | 22.4742% | 20.4710% |
| external mixed | 2,000 | 22.5886% | 18.8385% |
| external clean control | 2,000 | 1.6814% | 0.4886% |

The external set contains 1,600 human Opusparcus-derived phrases and 400
synthetic phrases. The clean control uses the original phrases, while the mixed
condition applies the same artificial character-corruption protocol.

On the external clean control, unnecessary modifications decreased from 24.35%
to 5.85%. These results apply only to the frozen artificial-corruption protocol,
prompt, deterministic decoding, Russian text sources, and evaluated phrase
lengths.

## License

The training sources use different licenses, so the repository is marked
`license: other` rather than Apache-2.0. No commercial-use clearance or claim of
cross-license compatibility is provided. See [LICENSE.md](LICENSE.md) for the
applicable CC BY-NC-SA 4.0 and OpenRAIL terms.

Exact adapter hashes, sizes, checkpoint mapping, and verification status are in
[release_manifest.json](release_manifest.json).