S-LoRA: Selective Low-Rank Adaptation for Fine-tuning Multimodal Large Language Models
This repository provides the model checkpoints and experimental results for S-LoRA (Selective Low-Rank Adaptation), a retention-aware fine-tuning method for multimodal large language models.
S-LoRA aims to balance downstream task adaptation and upstream capability preservation by selectively updating task-relevant LoRA coordinates while constraining excessive effective-update drift.
Links
- 💻 Code: zjj19959766-ctrl/S-LORA
- 🤗 Model Weights: Ape19959/S-LORA
Method Overview
S-LoRA contains two main components:
Gradient-sensitive Subspace Selection (GSS)
GSS estimates the importance of LoRA coordinates using downstream gradient statistics and performs global Top-K selection to construct a compact task-relevant support mask.
Subsequent adaptation is restricted to the selected LoRA coordinates.
Magnitude-constrained Progressive Adaptation (MPA)
MPA regulates the effective LoRA update
[ \Delta W = BA ]
rather than constraining the two LoRA factors independently.
It constructs an anchor update after GSS and progressively controls the layer-wise deviation from this anchor through a trust-region regularization mechanism.
Together, GSS and MPA are designed to improve downstream adaptation while mitigating catastrophic forgetting of pretrained multimodal capabilities.
Base Model
Our experiments are based on:
LLaVA v1.5 7B
The released files are lightweight LoRA/adaptation checkpoints and do not contain the full LLaVA base model.
Users should prepare the corresponding LLaVA base model separately.
Released Checkpoints
Table 1: Main Experiments
The checkpoints corresponding to the main experiments in Table 1 are currently available under:
Exp4.3.Tab1/
This directory contains S-LoRA checkpoints with different LoRA ranks used in the main comparison experiments.
Typical checkpoint files include:
adapter_config.json
adapter_model.safetensors
config.json
non_lora_trainables.bin
README.md
trainer_state.json
The primary adapter parameters are stored in:
adapter_model.safetensors
Additional trainable non-LoRA parameters, when applicable, are stored in:
non_lora_trainables.bin
Repository Structure
Ape19959/S-LORA
├── Exp4.3.Tab1/ # Main experiments in Table 1 (released)
├── Exp4.3.Tab2/ # Reserved for subsequent release
├── Exp4.4.Fig3/ # Reserved for subsequent release
├── Exp4.4.Fig4/ # Reserved for subsequent release
├── Exp4.4.Fig5/ # Reserved for subsequent release
├── Exp4.4.Fig6/ # Reserved for subsequent release
├── Exp4.4.Fig7/ # Reserved for subsequent release
├── Exp4.4.Tab3/ # Reserved for subsequent release
├── Exp4.4.Tab4/ # Reserved for subsequent release
├── Exp4.4.Tab5/ # Reserved for subsequent release
├── Exp4.4.Tab6/ # Reserved for subsequent release
└── Exp4.4.Tab7/ # Reserved for subsequent release
Additional checkpoints and experimental results will be progressively released.
Download
To download the complete repository:
hf download Ape19959/S-LORA --local-dir S-LORA-Weights
To download only the currently released Table 1 checkpoints:
hf download Ape19959/S-LORA \
--include "Exp4.3.Tab1/*" \
--local-dir S-LORA-Weights
You can also manually download individual checkpoints from the Files and versions page.
Usage
The checkpoints in this repository are designed to be used together with the official S-LoRA codebase:
https://github.com/zjj19959766-ctrl/S-LORA
Clone the repository:
git clone https://github.com/zjj19959766-ctrl/S-LORA.git
cd S-LORA
Then prepare the corresponding LLaVA v1.5 7B base model and place the downloaded S-LoRA checkpoint at the path expected by the evaluation scripts.
Training and evaluation entry points are provided under:
scripts/v1_5/train/
scripts/v1_5/eval/
For evaluation, update the model, checkpoint, dataset, and output paths according to your local environment before running:
bash ./scripts/v1_5/eval/eval_all.sh
For detailed implementation and environment setup, please refer to the GitHub repository.
Reproducibility Notes
- The released checkpoints correspond to experiments reported in the paper.
- Training and evaluation are based on LLaVA v1.5 7B.
- Dataset, model, and output paths should be modified according to the user's local environment.
- Some scripts may contain machine-specific paths and should be updated before execution.
- Additional ablation checkpoints will be released progressively.
Code
The complete implementation, including GSS, MPA, training scripts, evaluation scripts, and ablation implementations, is available at:
https://github.com/zjj19959766-ctrl/S-LORA
The core S-LoRA implementation is located in:
llava/train/LoRASculpt_Trainer.py
Citation
If you find this work useful, please consider citing:
S-LoRA: Selective Low-Rank Adaptation for Fine-tuning Multimodal Large Language Models
The complete BibTeX entry will be added after the paper information is finalized.
Acknowledgements
This project is built upon:
We thank the authors for their open-source contributions.
License
This repository is released under the Apache-2.0 License.
Please also comply with the licenses and usage terms of LLaVA, the underlying base models, datasets, and other upstream dependencies.