---
base_model: Qwen/Qwen2.5-3B-Instruct
library_name: peft
tags:
- latex
- mathematics
- text-to-speech
- mathematical-expressions
- qlora
- lora
- sft
- transformers
- trl
- qwen2.5
license: apache-2.0
pipeline_tag: text2text-generation
language:
- en
datasets:
- math-ai/AutoMathText
metrics:
- bleu
- exact_match
model-index:
- name: qwen2.5-3b-qlora-latex-fluentmath
  results:
  - task:
      type: text2text-generation
      name: LaTeX to Natural Language Translation
    metrics:
    - type: exact_match
      value: 42.60
      name: Exact Match Accuracy
    - type: bleu
      value: 93.39
      name: BLEU Score
    - type: similarity
      value: 90.56
      name: Edit Distance Similarity
---

# Qwen2.5-3B QLoRA: LaTeX → FluentMath Translation

**Fine-tuned model for converting LaTeX mathematical expressions to natural spoken English (FluentMath style)**

This model is a QLoRA (4-bit quantized + LoRA adapters) fine-tuned version of [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) for the specialized task of translating LaTeX mathematical notation into natural, conversational English suitable for text-to-speech systems.

## Model Description

- **Base Model**: Qwen/Qwen2.5-3B-Instruct (2.93B parameters)
- **Fine-tuning Method**: QLoRA (4-bit quantization + LoRA adapters)
- **Training Data**: 10,000 augmented LaTeX → FluentMath pairs from AutoMathText dataset
- **Task**: Sequence-to-sequence translation of mathematical notation
- **License**: Apache 2.0 (inherited from base model)

## Performance Metrics

Evaluated on 500 held-out test samples:

| Metric | Score |
|--------|-------|
| **Exact Match Accuracy** | **42.60%** |
| **BLEU (F1)** | **93.39%** |
| **Edit Distance Similarity** | **90.56%** |

**Comparison with 270M baseline (Gemma-3-270m):**
- Exact Match: +42.60 percentage points
- BLEU: +5.75 percentage points
- Similarity: +9.36 percentage points
- **VRAM**: 5.86GB (less than 270M full precision!)

## Training Details

### Training Configuration
- **Learning Rate**: 2e-4 (higher for LoRA)
- **Epochs**: 7
- **Batch Size**: 1 per device
- **Gradient Accumulation**: 16 steps (effective batch size: 16)
- **Optimizer**: paged_adamw_8bit (memory-efficient)
- **LR Scheduler**: Cosine with 10% warmup
- **Precision**: BF16
- **LoRA Parameters**:
  - Rank (r): 16
  - Alpha: 32
  - Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
  - Dropout: 0.05

### Training Results
- **Final Training Accuracy**: 94.45% (mean token accuracy)
- **Final Training Loss**: 0.198
- **Training Time**: ~3.5 hours on RTX 4070 SUPER (12GB)
- **Peak VRAM**: ~6-7GB during training

### Data Augmentation Strategy

Training data includes controlled noise to improve robustness:
- **60% Clean**: Original high-quality verbalizations
- **20% Tier 1**: Semantic-preserving variations (whitespace, notation alternatives)
- **15% Tier 2**: Delimiter variations (unmatched braces, extra brackets)
- **5% Tier 2**: Structural variations (command typos, missing braces)

## Usage

### Installation

```bash
pip install torch transformers peft accelerate bitsandbytes
```

### Inference

```python
import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

# Load model with LoRA adapters
model = AutoPeftModelForCausalLM.from_pretrained(
    "stefanj0/qwen2.5-3b-qlora-latex-fluentmath",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained("stefanj0/qwen2.5-3b-qlora-latex-fluentmath")

def translate_latex(latex: str) -> str:
    """Translate LaTeX to FluentMath."""
    user_prompt = (
        "Convert the following LaTeX mathematical expression to natural spoken English:\n\n"
        f"LaTeX: {latex}\n\n"
        "Spoken form:"
    )

    # Format as chat
    prompt = tokenizer.apply_chat_template(
        [{"role": "user", "content": user_prompt}],
        tokenize=False,
        add_generation_prompt=True
    )

    # Generate
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=256,
            pad_token_id=tokenizer.eos_token_id,
        )

    # Parse output
    generated = tokenizer.decode(outputs[0], skip_special_tokens=True)
    if "Spoken form:" in generated:
        verbalization = generated.split("Spoken form:")[-1].strip()
    else:
        verbalization = generated.strip()

    # Remove role prefix if present
    if verbalization.startswith("assistant\n"):
        verbalization = verbalization[10:].strip()

    return verbalization

# Example usage
latex = r"\int_a^b f'(x)\,dx = f(b) - f(a)"
translation = translate_latex(latex)
print(translation)
# Output: "the integral from eigh to b of, f prime, of x, d x, equals, f of b minus f of eigh"
```

## Example Translations

| LaTeX | FluentMath Output |
|-------|-------------------|
| `\frac{a}{b}` | a over b |
| `x^2 + y^2 = r^2` | x squared plus y squared equals r squared |
| `\int_a^b f'(x)\,dx = f(b) - f(a)` | the integral from eigh to b of, f prime, of x, d x, equals, f of b minus f of eigh |
| `P(A\|B) = \frac{P(B\|A)P(A)}{P(B)}` | P, the quantity Eigh divides B, equals, the fraction with numerator, P, the quantity B divides Eigh, P of Eigh, and denominator P of B |
| `\lim_{n\to\infty} \sum_{k=1}^{n} \frac{1}{n}` | the limit as n approaches infinity, of, sum from k equals 1 to n of, 1 over n |
| `A\vec{v} = \lambda\vec{v}` | Eigh, bold v, equals, lambda bold v |
| `i\hbar\frac{\partial}{\partial t}\Psi = \hat{H}\Psi` | i h bar, the fraction with numerator, partial derivative, and denominator partial derivative t, Psi, equals, H hat, Psi |

## Use Cases

- **Text-to-Speech Systems**: Convert mathematical papers to audio
- **Accessibility**: Help visually impaired users access mathematical content
- **Education**: Generate natural language descriptions of mathematical expressions
- **Data Augmentation**: Create training data for math-aware language models
- **Documentation**: Auto-generate spoken descriptions for LaTeX in technical docs

## Limitations

1. **Letter Pronunciation**: Capital 'A' rendered as "Eigh" (from speech synthesis rules)
2. **Vector Notation**: Vectors shown as "bold v" instead of "v with right arrow"
3. **Custom Macros**: Does not support `\newcommand` or custom LaTeX macros
4. **Long Expressions**: May truncate expressions longer than 256 tokens
5. **Domain Coverage**: Trained primarily on academic mathematics; may struggle with domain-specific notation

## Technical Details

### Why QLoRA?

QLoRA (Quantized LoRA) enables training large models on consumer GPUs:
- **4-bit Quantization**: 75% memory reduction (stores weights in 4 bits vs 16)
- **LoRA Adapters**: Train only ~1% of parameters (64M trainable out of 2.93B total)
- **Quality**: Achieves 95-98% of full fine-tuning performance
- **Efficiency**: Trains 11x larger model (3B vs 270M) using LESS memory (5.86GB vs 6-7GB)

### Architecture

- **Base**: Qwen2.5-3B-Instruct (decoder-only transformer)
- **Quantization**: 4-bit NormalFloat (NF4) with double quantization
- **LoRA Rank**: 16 (adapters inject low-rank updates into attention and MLP layers)
- **Target Modules**: All attention projections (Q, K, V, O) and MLP gates/projections

### Training Pipeline

1. **Data Extraction**: LaTeX expressions extracted from AutoMathText dataset
2. **Verbalization**: Converted to speech using mathwords library (Rust/PyO3)
3. **Evaluation**: Quality-checked with Qwen3-0.6B evaluator (93% accuracy)
4. **Correction**: Post-processed to fix common failure patterns
5. **Augmentation**: Noise added for robustness (10K samples total)
6. **Fine-tuning**: 7 epochs with QLoRA on RTX 4070 SUPER (12GB)

## Citation

If you use this model, please cite:

```bibtex
@misc{qwen25-3b-qlora-latex-fluentmath,
  author = {Stefan J.},
  title = {Qwen2.5-3B QLoRA: LaTeX to FluentMath Translation},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/stefanj0/qwen2.5-3b-qlora-latex-fluentmath}
}
```

Also cite the base model:

```bibtex
@article{qwen2.5,
  title={Qwen2.5: A Party of Foundation Models},
  author={Qwen Team},
  year={2024},
  journal={arXiv preprint arXiv:2412.xxxxx}
}
```

## Framework Versions

- PEFT: 0.18.0
- TRL: 0.26.2
- Transformers: 4.57.3
- PyTorch: 2.9.1
- Datasets: 4.4.2
- Tokenizers: 0.22.1

## Acknowledgments

- **Base Model**: [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) by Alibaba Cloud
- **Training Framework**: [Hugging Face Transformers](https://github.com/huggingface/transformers)
- **QLoRA Implementation**: [PEFT](https://github.com/huggingface/peft) + [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
- **Dataset**: [AutoMathText](https://huggingface.co/datasets/math-ai/AutoMathText) by Math-AI
- **Verbalization**: mathwords library (Rust/PyO3 wrapper for MathCAT)

## Model Card Authors

Stefan J. (stefanj0)

## Model Card Contact

For questions or issues, please open an issue on the model's discussion page.