---
language:
- en
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.1
tags:
- finance
- qlora
- lora
- peft
- fine-tuned
- financial-qa
datasets:
- gbharti/finance-alpaca
---

# Mistral-7B Finance QLoRA

A fine-tuned version of [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on financial question-answering data using QLoRA (Quantized Low-Rank Adaptation).

## Model Description

This model was fine-tuned to answer financial questions accurately and concisely. It handles topics like investment concepts, financial ratios, market instruments, macroeconomics, and corporate finance.

- **Base model**: mistralai/Mistral-7B-v0.1
- **Fine-tuning method**: QLoRA (NF4 4-bit quantization + LoRA adapters)
- **LoRA rank**: 16 | **LoRA alpha**: 32
- **Target modules**: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- **Trainable parameters**: ~20M out of 7.2B (0.28%)

## Training Data

- **Dataset**: [gbharti/finance-alpaca](https://huggingface.co/datasets/gbharti/finance-alpaca)
- **Samples used**: 10,000 instruction-response pairs
- **Train/test split**: 90/10
- **Prompt format**: Alpaca instruction template

## Training Details

| Parameter | Value |
|-----------|-------|
| Epochs | 2 |
| Batch size (per device) | 2 |
| Gradient accumulation steps | 8 |
| Effective batch size | 16 |
| Learning rate | 2e-4 |
| LR scheduler | Cosine |
| Warmup ratio | 0.03 |
| Optimizer | paged_adamw_32bit |
| Max sequence length | 512 |
| Quantization | NF4 4-bit (double quant) |
| Compute dtype | float16 |
| Hardware | Kaggle T4 x2 |
| Training time | ~5 hours |

## Training Results

| Epoch | Training Loss | Validation Loss |
|-------|--------------|-----------------|
| 1 | 1.2723 | 1.4589 |
| 2 | 1.0242 | 1.5069 |

## How to Use

### Load the adapter (memory efficient)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

base_model_id = "mistralai/Mistral-7B-v0.1"
adapter_id = "MadhurArora1/mistral-7b-finance-qlora"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_id)
```

### Run inference

```python
def ask(question, max_new_tokens=200):
    prompt = f"Below is a financial question. Answer it accurately and concisely.\n\n### Question:\n{question}\n\n### Answer:\n"
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.1,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id
        )
    return tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

print(ask("What is the P/E ratio and how do investors use it?"))
print(ask("Explain the difference between stocks and bonds."))
print(ask("What is compound interest?"))
```

## Example Outputs

**Q: What is the P/E ratio and how do investors use it?**
> The P/E ratio (Price-to-Earnings ratio) measures how much investors are willing to pay for each dollar of a company's earnings. It is calculated by dividing the stock price by earnings per share (EPS). A high P/E may indicate the market expects strong future growth, while a low P/E may suggest undervaluation or slow growth expectations. Investors use it to compare companies within the same industry.

**Q: What is the difference between ROE and ROA?**
> ROE (Return on Equity) measures how efficiently a company generates profit from shareholders' equity, while ROA (Return on Assets) measures profitability relative to total assets. ROE focuses on returns to equity holders and is affected by leverage; ROA is a broader measure of operational efficiency regardless of how assets are financed.

## Limitations

- Fine-tuned on 10k samples — may not cover all financial topics
- Not suitable for real investment advice or financial decisions
- Performance degrades on highly technical or niche financial topics
- Occasional hallucinations on specific numerical data (e.g. historical prices, exact rates)

## Training Infrastructure

- **Platform**: Kaggle (free tier)
- **GPU**: NVIDIA Tesla T4 x2 (30GB total VRAM)
- **Framework**: HuggingFace Transformers + PEFT + TRL
- **Experiment tracking**: Weights & Biases

## Author

**Madhur Arora** — Backend SWE | AI/ML enthusiast  
[GitHub](https://github.com/madhurarora0) | [LinkedIn](https://www.linkedin.com/in/madhur-arora-5a6156225/)

## Citation

If you use this model, please cite the base model:
```
@misc{mistral7b,
  title={Mistral 7B},
  author={Mistral AI},
  year={2023},
  url={https://huggingface.co/mistralai/Mistral-7B-v0.1}
}
```