Upload Qwen2.5-3B QLoRA model for LaTeX → FluentMath translation

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,271 @@
+---
+base_model: Qwen/Qwen2.5-3B-Instruct
+library_name: peft
+tags:
+- latex
+- mathematics
+- text-to-speech
+- mathematical-expressions
+- qlora
+- lora
+- sft
+- transformers
+- trl
+- qwen2.5
+license: apache-2.0
+pipeline_tag: text2text-generation
+language:
+- en
+datasets:
+- math-ai/AutoMathText
+metrics:
+- bleu
+- exact_match
+model-index:
+- name: qwen2.5-3b-qlora-latex-fluentmath
+  results:
+  - task:
+      type: text2text-generation
+      name: LaTeX to Natural Language Translation
+    metrics:
+    - type: exact_match
+      value: 42.60
+      name: Exact Match Accuracy
+    - type: bleu
+      value: 93.39
+      name: BLEU Score
+    - type: similarity
+      value: 90.56
+      name: Edit Distance Similarity
+---
+
+# Qwen2.5-3B QLoRA: LaTeX → FluentMath Translation
+
+**Fine-tuned model for converting LaTeX mathematical expressions to natural spoken English (FluentMath style)**
+
+This model is a QLoRA (4-bit quantized + LoRA adapters) fine-tuned version of [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) for the specialized task of translating LaTeX mathematical notation into natural, conversational English suitable for text-to-speech systems.
+
+## Model Description
+
+- **Base Model**: Qwen/Qwen2.5-3B-Instruct (2.93B parameters)
+- **Fine-tuning Method**: QLoRA (4-bit quantization + LoRA adapters)
+- **Training Data**: 10,000 augmented LaTeX → FluentMath pairs from AutoMathText dataset
+- **Task**: Sequence-to-sequence translation of mathematical notation
+- **License**: Apache 2.0 (inherited from base model)
+
+## Performance Metrics
+
+Evaluated on 500 held-out test samples:
+
+| Metric | Score |
+|--------|-------|
+| **Exact Match Accuracy** | **42.60%** |
+| **BLEU (F1)** | **93.39%** |
+| **Edit Distance Similarity** | **90.56%** |
+
+**Comparison with 270M baseline (Gemma-3-270m):**
+- Exact Match: +42.60 percentage points
+- BLEU: +5.75 percentage points
+- Similarity: +9.36 percentage points
+- **VRAM**: 5.86GB (less than 270M full precision!)
+
+## Training Details
+
+### Training Configuration
+- **Learning Rate**: 2e-4 (higher for LoRA)
+- **Epochs**: 7
+- **Batch Size**: 1 per device
+- **Gradient Accumulation**: 16 steps (effective batch size: 16)
+- **Optimizer**: paged_adamw_8bit (memory-efficient)
+- **LR Scheduler**: Cosine with 10% warmup
+- **Precision**: BF16
+- **LoRA Parameters**:
+  - Rank (r): 16
+  - Alpha: 32
+  - Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
+  - Dropout: 0.05
+
+### Training Results
+- **Final Training Accuracy**: 94.45% (mean token accuracy)
+- **Final Training Loss**: 0.198
+- **Training Time**: ~3.5 hours on RTX 4070 SUPER (12GB)
+- **Peak VRAM**: ~6-7GB during training
+
+### Data Augmentation Strategy
+
+Training data includes controlled noise to improve robustness:
+- **60% Clean**: Original high-quality verbalizations
+- **20% Tier 1**: Semantic-preserving variations (whitespace, notation alternatives)
+- **15% Tier 2**: Delimiter variations (unmatched braces, extra brackets)
+- **5% Tier 2**: Structural variations (command typos, missing braces)
+
+## Usage
+
+### Installation
+
+```bash
+pip install torch transformers peft accelerate bitsandbytes
+```
+
+### Inference
+
+```python
+import torch
+from peft import AutoPeftModelForCausalLM
+from transformers import AutoTokenizer
+
+# Load model with LoRA adapters
+model = AutoPeftModelForCausalLM.from_pretrained(
+    "stefanj0/qwen2.5-3b-qlora-latex-fluentmath",
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+)
+tokenizer = AutoTokenizer.from_pretrained("stefanj0/qwen2.5-3b-qlora-latex-fluentmath")
+
+def translate_latex(latex: str) -> str:
+    """Translate LaTeX to FluentMath."""
+    user_prompt = (
+        "Convert the following LaTeX mathematical expression to natural spoken English:\n\n"
+        f"LaTeX: {latex}\n\n"
+        "Spoken form:"
+    )
+
+    # Format as chat
+    prompt = tokenizer.apply_chat_template(
+        [{"role": "user", "content": user_prompt}],
+        tokenize=False,
+        add_generation_prompt=True
+    )
+
+    # Generate
+    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+    with torch.no_grad():
+        outputs = model.generate(
+            **inputs,
+            max_new_tokens=256,
+            pad_token_id=tokenizer.eos_token_id,
+        )
+
+    # Parse output
+    generated = tokenizer.decode(outputs[0], skip_special_tokens=True)
+    if "Spoken form:" in generated:
+        verbalization = generated.split("Spoken form:")[-1].strip()
+    else:
+        verbalization = generated.strip()
+
+    # Remove role prefix if present
+    if verbalization.startswith("assistant\n"):
+        verbalization = verbalization[10:].strip()
+
+    return verbalization
+
+# Example usage
+latex = r"\int_a^b f'(x)\,dx = f(b) - f(a)"
+translation = translate_latex(latex)
+print(translation)
+# Output: "the integral from eigh to b of, f prime, of x, d x, equals, f of b minus f of eigh"
+```
+
+## Example Translations
+
+| LaTeX | FluentMath Output |
+|-------|-------------------|
+| `\frac{a}{b}` | a over b |
+| `x^2 + y^2 = r^2` | x squared plus y squared equals r squared |
+| `\int_a^b f'(x)\,dx = f(b) - f(a)` | the integral from eigh to b of, f prime, of x, d x, equals, f of b minus f of eigh |
+| `P(A\|B) = \frac{P(B\|A)P(A)}{P(B)}` | P, the quantity Eigh divides B, equals, the fraction with numerator, P, the quantity B divides Eigh, P of Eigh, and denominator P of B |
+| `\lim_{n\to\infty} \sum_{k=1}^{n} \frac{1}{n}` | the limit as n approaches infinity, of, sum from k equals 1 to n of, 1 over n |
+| `A\vec{v} = \lambda\vec{v}` | Eigh, bold v, equals, lambda bold v |
+| `i\hbar\frac{\partial}{\partial t}\Psi = \hat{H}\Psi` | i h bar, the fraction with numerator, partial derivative, and denominator partial derivative t, Psi, equals, H hat, Psi |
+
+## Use Cases
+
+- **Text-to-Speech Systems**: Convert mathematical papers to audio
+- **Accessibility**: Help visually impaired users access mathematical content
+- **Education**: Generate natural language descriptions of mathematical expressions
+- **Data Augmentation**: Create training data for math-aware language models
+- **Documentation**: Auto-generate spoken descriptions for LaTeX in technical docs
+
+## Limitations
+
+1. **Letter Pronunciation**: Capital 'A' rendered as "Eigh" (from speech synthesis rules)
+2. **Vector Notation**: Vectors shown as "bold v" instead of "v with right arrow"
+3. **Custom Macros**: Does not support `\newcommand` or custom LaTeX macros
+4. **Long Expressions**: May truncate expressions longer than 256 tokens
+5. **Domain Coverage**: Trained primarily on academic mathematics; may struggle with domain-specific notation
+
+## Technical Details
+
+### Why QLoRA?
+
+QLoRA (Quantized LoRA) enables training large models on consumer GPUs:
+- **4-bit Quantization**: 75% memory reduction (stores weights in 4 bits vs 16)
+- **LoRA Adapters**: Train only ~1% of parameters (64M trainable out of 2.93B total)
+- **Quality**: Achieves 95-98% of full fine-tuning performance
+- **Efficiency**: Trains 11x larger model (3B vs 270M) using LESS memory (5.86GB vs 6-7GB)
+
+### Architecture
+
+- **Base**: Qwen2.5-3B-Instruct (decoder-only transformer)
+- **Quantization**: 4-bit NormalFloat (NF4) with double quantization
+- **LoRA Rank**: 16 (adapters inject low-rank updates into attention and MLP layers)
+- **Target Modules**: All attention projections (Q, K, V, O) and MLP gates/projections
+
+### Training Pipeline
+
+1. **Data Extraction**: LaTeX expressions extracted from AutoMathText dataset
+2. **Verbalization**: Converted to speech using mathwords library (Rust/PyO3)
+3. **Evaluation**: Quality-checked with Qwen3-0.6B evaluator (93% accuracy)
+4. **Correction**: Post-processed to fix common failure patterns
+5. **Augmentation**: Noise added for robustness (10K samples total)
+6. **Fine-tuning**: 7 epochs with QLoRA on RTX 4070 SUPER (12GB)
+
+## Citation
+
+If you use this model, please cite:
+
+```bibtex
+@misc{qwen25-3b-qlora-latex-fluentmath,
+  author = {Stefan J.},
+  title = {Qwen2.5-3B QLoRA: LaTeX to FluentMath Translation},
+  year = {2025},
+  publisher = {HuggingFace},
+  url = {https://huggingface.co/stefanj0/qwen2.5-3b-qlora-latex-fluentmath}
+}
+```
+
+Also cite the base model:
+
+```bibtex
+@article{qwen2.5,
+  title={Qwen2.5: A Party of Foundation Models},
+  author={Qwen Team},
+  year={2024},
+  journal={arXiv preprint arXiv:2412.xxxxx}
+}
+```
+
+## Framework Versions
+
+- PEFT: 0.18.0
+- TRL: 0.26.2
+- Transformers: 4.57.3
+- PyTorch: 2.9.1
+- Datasets: 4.4.2
+- Tokenizers: 0.22.1
+
+## Acknowledgments
+
+- **Base Model**: [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) by Alibaba Cloud
+- **Training Framework**: [Hugging Face Transformers](https://github.com/huggingface/transformers)
+- **QLoRA Implementation**: [PEFT](https://github.com/huggingface/peft) + [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
+- **Dataset**: [AutoMathText](https://huggingface.co/datasets/math-ai/AutoMathText) by Math-AI
+- **Verbalization**: mathwords library (Rust/PyO3 wrapper for MathCAT)
+
+## Model Card Authors
+
+Stefan J. (stefanj0)
+
+## Model Card Contact
+
+For questions or issues, please open an issue on the model's discussion page.

adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2.5-3B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "q_proj",
+    "k_proj",
+    "o_proj",
+    "v_proj",
+    "gate_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b32e9185258e9d1bb278d69f012972f40a2fdde11724cf3a807bc943e00c34c5
+size 119801528

added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja

@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
+size 11421896

tokenizer_config.json

@@ -0,0 +1,207 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}