---
library_name: transformers
license: apache-2.0
base_model: Qwen/Qwen3-14B
tags:
- qwen3
- sft
- fsdp
- qlora
- custom
language:
- en
- ja
pipeline_tag: text-generation
---

# Qwen3-14B SFT Model

## Model Description

This is a fine-tuned version of Qwen3-14B using Supervised Fine-Tuning (SFT) with FSDP (Fully Sharded Data Parallel) + QLoRA (Quantized Low-Rank Adaptation) techniques.

## Training Details

### Base Model
- **Model**: Qwen/Qwen3-14B
- **Architecture**: Transformer-based causal language model
- **Parameters**: 14 billion

### Training Configuration
- **Method**: FSDP + QLoRA
- **Quantization**: 4-bit QLoRA
- **LoRA Parameters**:
  - r: 64
  - alpha: 16
  - dropout: 0.1
  - target: linear layers
- **Hardware**: 8x H100 80GB HBM3
- **Precision**: bfloat16
- **Flash Attention**: Enabled

### Training Hyperparameters
- **Epochs**: 1
- **Micro Batch Size**: 1
- **Gradient Accumulation Steps**: 16
- **Learning Rate**: 1e-4
- **Scheduler**: Cosine with warmup
- **Warmup Ratio**: 0.03
- **Optimizer**: AdamW
- **Sequence Length**: 1024

### Dataset
- Custom SFT dataset (SFT_004_origin_4.parquet)
- Validation split: 10%
- Sample packing enabled for training efficiency

## Model Performance

The model has been trained for efficient instruction following and maintains the original Qwen3 capabilities while being optimized for custom tasks.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "u-10bei/qwen3-14b-sft-merged",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "u-10bei/qwen3-14b-sft-merged",
    trust_remote_code=True
)

# Chat format
messages = [
    {"role": "user", "content": "Hello! How can I help you today?"}
]

# Format conversation
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

# Tokenize
inputs = tokenizer(text, return_tensors="pt")

# Generate
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id
    )

# Decode response
response = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True)
print(response)
```

### Direct Chat Format

```python
# Manual chat formatting
prompt = "<|im_start|>user\nHello! How are you?<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.7,
    eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>")
)

response = tokenizer.decode(outputs[0], skip_special_tokens=False)
print(response)
```

## Special Tokens

- **BOS Token**: `<|im_start|>`
- **EOS Token**: `<|im_end|>`
- **UNK Token**: `<|endoftext|>`
- **PAD Token**: `<|endoftext|>`

## Technical Specifications

### Model Architecture
- **Attention**: Flash Attention 2 (training and inference)
- **Precision**: bfloat16 native support
- **Context Length**: 1024 tokens (training), extensible for inference
- **Vocabulary Size**: 151,669 tokens

### Optimization Features
- **Memory Efficient**: FSDP sharding reduces memory footprint
- **Quantization Ready**: QLoRA-compatible for efficient fine-tuning
- **Multi-GPU**: Optimized for distributed inference

## Training Infrastructure

- **Distributed Training**: FSDP (Fully Sharded Data Parallel)
- **Communication**: NCCL with Ethernet backend
- **Memory Management**: Expandable segments, optimized allocation
- **Monitoring**: Weights & Biases integration

## Limitations

- This model is optimized for the specific training dataset and may not generalize to all use cases
- Context length is limited to 1024 tokens during training
- Performance may vary depending on the specific task and input format

## Ethical Considerations

This model inherits the capabilities and limitations of the base Qwen3-14B model. Users should be aware of potential biases and use the model responsibly.

## Citation

If you use this model, please cite:

```bibtex
@model{qwen3-14b-sft-merged,
  title={Qwen3-14B SFT Model with FSDP+QLoRA},
  author={u-10bei},
  year={2025},
  url={https://huggingface.co/u-10bei/qwen3-14b-sft-merged}
}
```

## Model Card Authors

- u-10bei

## Training Date

August 2025

---

*This model was trained using advanced distributed training techniques (FSDP + QLoRA) on high-performance H100 hardware for optimal efficiency and scalability.*