stefanj0/qwen2.5-3b-qlora-latex-fluentmath

Qwen2.5-3B QLoRA: LaTeX → FluentMath Translation

Fine-tuned model for converting LaTeX mathematical expressions to natural spoken English (FluentMath style)

This model is a QLoRA (4-bit quantized + LoRA adapters) fine-tuned version of Qwen/Qwen2.5-3B-Instruct for the specialized task of translating LaTeX mathematical notation into natural, conversational English suitable for text-to-speech systems.

Model Description

• Base Model: Qwen/Qwen2.5-3B-Instruct (2.93B parameters)
• Fine-tuning Method: QLoRA (4-bit quantization + LoRA adapters)
• Training Data: 10,000 augmented LaTeX → FluentMath pairs from AutoMathText dataset
• Task: Sequence-to-sequence translation of mathematical notation
• License: Apache 2.0 (inherited from base model)

Performance Metrics

Evaluated on 500 held-out test samples:

Metric	Score
Exact Match Accuracy	42.60%
BLEU (F1)	93.39%
Edit Distance Similarity	90.56%

Comparison with 270M baseline (Gemma-3-270m):

• Exact Match: +42.60 percentage points
• BLEU: +5.75 percentage points
• Similarity: +9.36 percentage points
• VRAM: 5.86GB (less than 270M full precision!)

Training Details

Training Configuration

• Learning Rate: 2e-4 (higher for LoRA)
• Epochs: 7
• Batch Size: 1 per device
• Gradient Accumulation: 16 steps (effective batch size: 16)
• Optimizer: paged_adamw_8bit (memory-efficient)
• LR Scheduler: Cosine with 10% warmup
• Precision: BF16
• LoRA Parameters:
  • Rank (r): 16
  • Alpha: 32
  • Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
  • Dropout: 0.05

Training Results

• Final Training Accuracy: 94.45% (mean token accuracy)
• Final Training Loss: 0.198
• Training Time: ~3.5 hours on RTX 4070 SUPER (12GB)
• Peak VRAM: ~6-7GB during training

Data Augmentation Strategy

Training data includes controlled noise to improve robustness:

• 60% Clean: Original high-quality verbalizations
• 20% Tier 1: Semantic-preserving variations (whitespace, notation alternatives)
• 15% Tier 2: Delimiter variations (unmatched braces, extra brackets)
• 5% Tier 2: Structural variations (command typos, missing braces)

Usage

Installation

pip install torch transformers peft accelerate bitsandbytes

Inference

import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

# Load model with LoRA adapters
model = AutoPeftModelForCausalLM.from_pretrained(
    "stefanj0/qwen2.5-3b-qlora-latex-fluentmath",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained("stefanj0/qwen2.5-3b-qlora-latex-fluentmath")

def translate_latex(latex: str) -> str:
    """Translate LaTeX to FluentMath."""
    user_prompt = (
        "Convert the following LaTeX mathematical expression to natural spoken English:\n\n"
        f"LaTeX: {latex}\n\n"
        "Spoken form:"
    )

    # Format as chat
    prompt = tokenizer.apply_chat_template(
        [{"role": "user", "content": user_prompt}],
        tokenize=False,
        add_generation_prompt=True
    )

    # Generate
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=256,
            pad_token_id=tokenizer.eos_token_id,
        )

    # Parse output
    generated = tokenizer.decode(outputs[0], skip_special_tokens=True)
    if "Spoken form:" in generated:
        verbalization = generated.split("Spoken form:")[-1].strip()
    else:
        verbalization = generated.strip()

    # Remove role prefix if present
    if verbalization.startswith("assistant\n"):
        verbalization = verbalization[10:].strip()

    return verbalization

# Example usage
latex = r"\int_a^b f'(x)\,dx = f(b) - f(a)"
translation = translate_latex(latex)
print(translation)
# Output: "the integral from eigh to b of, f prime, of x, d x, equals, f of b minus f of eigh"

Example Translations

LaTeX	FluentMath Output
\frac{a}{b}	a over b
x^2 + y^2 = r^2	x squared plus y squared equals r squared
\int_a^b f'(x)\,dx = f(b) - f(a)	the integral from eigh to b of, f prime, of x, d x, equals, f of b minus f of eigh
P(A|B) = \frac{P(B|A)P(A)}{P(B)}	P, the quantity Eigh divides B, equals, the fraction with numerator, P, the quantity B divides Eigh, P of Eigh, and denominator P of B
\lim_{n\to\infty} \sum_{k=1}^{n} \frac{1}{n}	the limit as n approaches infinity, of, sum from k equals 1 to n of, 1 over n
A\vec{v} = \lambda\vec{v}	Eigh, bold v, equals, lambda bold v
i\hbar\frac{\partial}{\partial t}\Psi = \hat{H}\Psi	i h bar, the fraction with numerator, partial derivative, and denominator partial derivative t, Psi, equals, H hat, Psi

Use Cases

• Text-to-Speech Systems: Convert mathematical papers to audio
• Accessibility: Help visually impaired users access mathematical content
• Education: Generate natural language descriptions of mathematical expressions
• Data Augmentation: Create training data for math-aware language models
• Documentation: Auto-generate spoken descriptions for LaTeX in technical docs

Limitations

1. Letter Pronunciation: Capital 'A' rendered as "Eigh" (from speech synthesis rules)
2. Vector Notation: Vectors shown as "bold v" instead of "v with right arrow"
3. Custom Macros: Does not support \newcommand or custom LaTeX macros
4. Long Expressions: May truncate expressions longer than 256 tokens
5. Domain Coverage: Trained primarily on academic mathematics; may struggle with domain-specific notation

Technical Details

Why QLoRA?

QLoRA (Quantized LoRA) enables training large models on consumer GPUs:

• 4-bit Quantization: 75% memory reduction (stores weights in 4 bits vs 16)
• LoRA Adapters: Train only ~1% of parameters (64M trainable out of 2.93B total)
• Quality: Achieves 95-98% of full fine-tuning performance
• Efficiency: Trains 11x larger model (3B vs 270M) using LESS memory (5.86GB vs 6-7GB)

Architecture

• Base: Qwen2.5-3B-Instruct (decoder-only transformer)
• Quantization: 4-bit NormalFloat (NF4) with double quantization
• LoRA Rank: 16 (adapters inject low-rank updates into attention and MLP layers)
• Target Modules: All attention projections (Q, K, V, O) and MLP gates/projections

Training Pipeline

1. Data Extraction: LaTeX expressions extracted from AutoMathText dataset
2. Verbalization: Converted to speech using mathwords library (Rust/PyO3)
3. Evaluation: Quality-checked with Qwen3-0.6B evaluator (93% accuracy)
4. Correction: Post-processed to fix common failure patterns
5. Augmentation: Noise added for robustness (10K samples total)
6. Fine-tuning: 7 epochs with QLoRA on RTX 4070 SUPER (12GB)

Citation

If you use this model, please cite:

@misc{qwen25-3b-qlora-latex-fluentmath,
  author = {Stefan J.},
  title = {Qwen2.5-3B QLoRA: LaTeX to FluentMath Translation},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/stefanj0/qwen2.5-3b-qlora-latex-fluentmath}
}

Also cite the base model:

@article{qwen2.5,
  title={Qwen2.5: A Party of Foundation Models},
  author={Qwen Team},
  year={2024},
  journal={arXiv preprint arXiv:2412.xxxxx}
}

Framework Versions

• PEFT: 0.18.0
• TRL: 0.26.2
• Transformers: 4.57.3
• PyTorch: 2.9.1
• Datasets: 4.4.2
• Tokenizers: 0.22.1

Acknowledgments

• Base Model: Qwen/Qwen2.5-3B-Instruct by Alibaba Cloud
• Training Framework: Hugging Face Transformers
• QLoRA Implementation: PEFT + bitsandbytes
• Dataset: AutoMathText by Math-AI
• Verbalization: mathwords library (Rust/PyO3 wrapper for MathCAT)

Model Card Authors

Stefan J. (stefanj0)

Model Card Contact

For questions or issues, please open an issue on the model's discussion page.