MadhurArora1/mistral-7b-finance-qlora

Mistral-7B Finance QLoRA

A fine-tuned version of Mistral-7B-v0.1 on financial question-answering data using QLoRA (Quantized Low-Rank Adaptation).

Model Description

This model was fine-tuned to answer financial questions accurately and concisely. It handles topics like investment concepts, financial ratios, market instruments, macroeconomics, and corporate finance.

• Base model: mistralai/Mistral-7B-v0.1
• Fine-tuning method: QLoRA (NF4 4-bit quantization + LoRA adapters)
• LoRA rank: 16 | LoRA alpha: 32
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Trainable parameters: ~20M out of 7.2B (0.28%)

Training Data

• Dataset: gbharti/finance-alpaca
• Samples used: 10,000 instruction-response pairs
• Train/test split: 90/10
• Prompt format: Alpaca instruction template

Training Details

Parameter	Value
Epochs	2
Batch size (per device)	2
Gradient accumulation steps	8
Effective batch size	16
Learning rate	2e-4
LR scheduler	Cosine
Warmup ratio	0.03
Optimizer	paged_adamw_32bit
Max sequence length	512
Quantization	NF4 4-bit (double quant)
Compute dtype	float16
Hardware	Kaggle T4 x2
Training time	~5 hours

Training Results

Epoch	Training Loss	Validation Loss
1	1.2723	1.4589
2	1.0242	1.5069

How to Use

Load the adapter (memory efficient)

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

base_model_id = "mistralai/Mistral-7B-v0.1"
adapter_id = "MadhurArora1/mistral-7b-finance-qlora"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_id)

Run inference

def ask(question, max_new_tokens=200):
    prompt = f"Below is a financial question. Answer it accurately and concisely.\n\n### Question:\n{question}\n\n### Answer:\n"
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.1,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id
        )
    return tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

print(ask("What is the P/E ratio and how do investors use it?"))
print(ask("Explain the difference between stocks and bonds."))
print(ask("What is compound interest?"))

Example Outputs

Q: What is the P/E ratio and how do investors use it?

The P/E ratio (Price-to-Earnings ratio) measures how much investors are willing to pay for each dollar of a company's earnings. It is calculated by dividing the stock price by earnings per share (EPS). A high P/E may indicate the market expects strong future growth, while a low P/E may suggest undervaluation or slow growth expectations. Investors use it to compare companies within the same industry.

Q: What is the difference between ROE and ROA?

ROE (Return on Equity) measures how efficiently a company generates profit from shareholders' equity, while ROA (Return on Assets) measures profitability relative to total assets. ROE focuses on returns to equity holders and is affected by leverage; ROA is a broader measure of operational efficiency regardless of how assets are financed.

Limitations

• Fine-tuned on 10k samples — may not cover all financial topics
• Not suitable for real investment advice or financial decisions
• Performance degrades on highly technical or niche financial topics
• Occasional hallucinations on specific numerical data (e.g. historical prices, exact rates)

Training Infrastructure

• Platform: Kaggle (free tier)
• GPU: NVIDIA Tesla T4 x2 (30GB total VRAM)
• Framework: HuggingFace Transformers + PEFT + TRL
• Experiment tracking: Weights & Biases

Author

Madhur Arora — Backend SWE | AI/ML enthusiast
GitHub | LinkedIn

Citation

If you use this model, please cite the base model:

@misc{mistral7b,
  title={Mistral 7B},
  author={Mistral AI},
  year={2023},
  url={https://huggingface.co/mistralai/Mistral-7B-v0.1}
}