Fine-tuning Muse-Glimmer-30B with ms-swift: A LoRA Experiment Report

This report documents a fine-tuning experiment on Muse-Glimmer-30B for a handwritten formula OCR task, covering the experimental configuration, convergence behaviour, resource consumption, and output quality. All figures are taken from actual run logs and the experiment is reproducible.

1. Experimental Setup

Model: Muse-Glimmer-30B, a multimodal model with 29829.02M parameters.

Dataset: AI-ModelScope/LaTeX_OCR:human_handwrite, consisting of handwritten mathematical formula images paired with LaTeX annotations. A validation split of 2% is held out.

Method: LoRA (rank 8, alpha 32) applied to the language model only, with the vision encoder and the projection layer frozen. The number of trainable parameters is 52.40M, or 0.176% of the total.

Hardware: A100-80G.

Software environment:

pip install git+https://github.com/modelscope/ms-swift.git
# pip install https://github.com/modelscope/mcore-bridge.git
pip install "transformers>=5.16" -U

The model implementation of Muse-Glimmer was introduced in transformers 5.16, so this version requirement must be satisfied.

2. Training Commands

2.1 transformers Backend (Single GPU)

export CUDA_VISIBLE_DEVICES=0
export MAX_PIXELS=1003520

swift sft \
  --model meta-models/Muse-Glimmer-30B \
  --dataset 'AI-ModelScope/LaTeX_OCR:human_handwrite' \
  --split_dataset_ratio 0.02 \
  --tuner_type lora --lora_rank 8 --lora_alpha 32 \
  --freeze_vit true --freeze_aligner true \
  --torch_dtype bfloat16 --attn_impl flash_attn \
  --per_device_train_batch_size 4 --gradient_accumulation_steps 2 \
  --learning_rate 1e-4 --lr_scheduler_type cosine --warmup_ratio 0.05 \
  --max_steps 100 --logging_steps 5 --eval_steps 50 --save_steps 50 \
  --max_length 2048 --gradient_checkpointing true \
  --output_dir output/ocr_hf

2.2 Megatron Backend (Tensor Parallelism, TP=2)

The data and hyper-parameters are identical to those in Section 2.1.

export CUDA_VISIBLE_DEVICES=0,1
export NPROC_PER_NODE=2
export MAX_PIXELS=1003520

megatron sft \
  --model meta-models/Muse-Glimmer-30B \
  --dataset 'AI-ModelScope/LaTeX_OCR:human_handwrite' \
  --split_dataset_ratio 0.02 \
  --tuner_type lora --lora_rank 8 --lora_alpha 32 \
  --freeze_llm false --freeze_vit true --freeze_aligner true \
  --tensor_model_parallel_size 2 \
  --micro_batch_size 4 --global_batch_size 8 \
  --lr 1e-4 --min_lr 1e-5 --lr_decay_style cosine --lr_warmup_iters 5 \
  --train_iters 100 --logging_steps 5 --eval_iters 5 --eval_steps 50 --save_steps 50 \
  --max_length 2048 --finetune true --attention_backend flash \
  --recompute_granularity full --recompute_method uniform --recompute_num_layers 1 \
  --padding_free false --packing false \
  --output_dir output/ocr_mg

3. Results

3.1 Convergence

The loss drops by an order of magnitude within the first 10 steps and then enters a phase of slow convergence; token accuracy rises from 81.18% to 99.77%.

3.2 Cross-backend Consistency

The two backends are mutually independent implementations: the transformers backend relies on the HuggingFace model definition, whereas the Megatron backend uses fused kernels and tensor-parallel sharding, and the two also shuffle the data differently. Per-step losses are therefore not comparable point by point, but the validation losses fall at the same level, indicating that the two computational paths are numerically consistent.

3.3 Resource Consumption

A single 80G GPU is sufficient for LoRA fine-tuning of a 30B model. This depends on the combined effect of three settings: freezing the vision modules, gradient checkpointing, and bf16 mixed precision.

At the scale of this experiment the Megatron backend is slower than the single-GPU run: the model fits within the memory of one device, and the communication overhead introduced by tensor parallelism outweighs its benefit. The advantages of this backend lie in the roughly 46% reduction in per-GPU memory, and in its ability to scale to larger models, longer sequences, and multi-node settings through combinations of TP/PP/CP/EP.

3.4 Output Quality

Inference was compared on validation samples with temperature 0.

Before fine-tuning, the model did not produce the LaTeX output required by the task; instead it repeatedly enumerated candidate answers and failed to converge to a final output within the 128-token budget:

Label : z _ { 1 } = r _ { 1 } ( \cos \theta _ { 1 } + i \sin \theta _ { 1 } )
Output: Using LaTeX to perform OCR on the image.
        The image shows: Z_1 = r_1 ( cos θ_1 + i Sin θ_1 )
        Probably Z_1 = r_1 ( cos θ_1 + i sin θ_1 )
        Maybe they want LaTeX: $Z_1 = r_1 (\cos \theta_1 + i \sin \theta_1)$

After fine-tuning, 3 of the 5 samples match the annotation character for character, while the remaining 2 contain isolated character errors:

Sample 0 (exact match)     : z _ { 1 } = r _ { 1 } ( \cos \theta _ { 1 } + i \sin \theta _ { 1 } )
Sample 1 (exact match)     : \log z = \log r + i ( \theta + 2 n \pi )
Sample 4 (exact match)     : \log _ { 2 } 8 + \log _ { 3 } 9 + \log _ { 4 } 1 6
Sample 2 (missing char)    : 2 1 - 5 \sqrt { 2 1 } + ( 1 5 \sqrt { 7 } - 2 1 \sqrt { 3 } )
Sample 3 (misread digits)  : ( ( 5 6 \times 5 2 ) \div 1 3 5 ) + ( 3 4 - ( 7 1 \times 9 8 ) )

The generated length falls from 640 tokens to 196 tokens and inference time from 39.6 s to 20.4 s, indicating that the model has acquired the concise output format demanded by the task.

4. Inference and Export

# Inference with the adapter
swift infer --model meta-models/Muse-Glimmer-30B --adapters output/ocr_hf/vx-xxx/checkpoint-100

# Batch inference with results written to a file
swift infer --model meta-models/Muse-Glimmer-30B --adapters output/.../checkpoint-100 \
  --val_dataset 'AI-ModelScope/LaTeX_OCR:human_handwrite#100' \
  --max_new_tokens 128 --temperature 0 --result_path result.jsonl

# Merge the LoRA weights and export
swift export --model meta-models/Muse-Glimmer-30B --adapters output/.../checkpoint-100 --merge_lora true

Training with the Megatron backend produces both checkpoint-100/ (mcore format together with the adapter) and checkpoint-100-merged/ (HuggingFace format); the latter can be used directly for inference or deployment.

References

• Repository: https://github.com/modelscope/ms-swift
• Model: https://modelscope.cn/models/meta-models/Muse-Glimmer-30B
• Documentation and custom dataset guide: https://swift.readthedocs.io