---
library_name: peft
license: other
base_model: Qwen/Qwen3.5-397B-A17B
tags:
- base_model:adapter:Qwen/Qwen3.5-397B-A17B
- llama-factory
- lora
- transformers
- amd
- rocm
- qwen
pipeline_tag: text-generation
---

# Qwen3.5-lora-sft-v5-1-64k

This repository contains a LoRA adapter for `Qwen/Qwen3.5-397B-A17B`, trained with LLaMA-Factory on the `amdpilot_v5_1` SFT dataset.

This is an adapter-only release. You need the base model `Qwen/Qwen3.5-397B-A17B` to use it.

## Key training settings

- Fine-tuning method: LoRA
- LoRA rank / alpha: `32 / 64`
- Context window: `65536`
- Packing: `true`
- Neat packing: `false`
- Precision: `bf16`
- Distributed setup: `8x AMD MI355X`
- Epochs: `10`

## Final metrics

- Final train loss: `0.0630452295144399`
- Final eval loss: `0.133148193359375`
- Train runtime: `47396.7738s` (`13.17h`)

## Eval trajectory

| Step | Epoch | Eval loss |
| --- | --- | --- |
| 10 | 1.7273 | 0.1846 |
| 20 | 3.3636 | 0.1579 |
| 30 | 5.0 | 0.1417 |
| 40 | 6.7273 | 0.1357 |
| 50 | 8.3636 | 0.1336 |
| 60 | 10.0 | 0.1331 |

## Dataset coverage note

On the current `amdpilot_v5_1` training split, `65536` tokens cover about `82/89` samples (`92.13%`). This is substantially better coverage than the earlier `32768` setting.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen3.5-397B-A17B"
adapter_id = "JinnP/Qwen3.5-lora-sft-v5-1-64k"

tokenizer = AutoTokenizer.from_pretrained(adapter_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(base_model_id, trust_remote_code=True)
model = PeftModel.from_pretrained(base_model, adapter_id)
```

## Files

- `adapter_model.safetensors`: LoRA adapter weights
- `adapter_config.json`: PEFT adapter config
- `tokenizer.json` / `tokenizer_config.json` / `chat_template.jinja`: tokenizer assets
- `all_results.json` / `eval_results.json` / `train_results.json`: training metrics