File size: 2,154 Bytes
6c4f66a f511449 56acdd2 f511449 56acdd2 f511449 6c4f66a f511449 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | ---
language: en
license: llama3
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
tags:
- finance
- sentiment-analysis
- qlora
- peft
- lora
- unsloth
datasets:
- FinGPT/fingpt-sentiment-train
metrics:
- f1
- accuracy
---
# Llama-3.1-8B Financial Sentiment — QLoRA
LoRA adapters fine-tuned on [FinGPT/fingpt-sentiment-train](https://huggingface.co/datasets/FinGPT/fingpt-sentiment-train)
with QLoRA: 4-bit NF4 frozen base + bf16 LoRA adapters (r=16, alpha=32, dropout=0.05).
**Task:** 3-class financial sentiment — `negative` / `neutral` / `positive`
## Evaluation Results
| Dataset | Base Accuracy | FT Accuracy | Base Macro-F1 | FT Macro-F1 |
|----------------|---------------|---------------|---------------|---------------|
| FPB in-domain | 0.8908 | 0.9748 | 0.8765 | 0.9725 |
| FiQA-SA OOD | 0.8120 | 0.9402 | 0.6705 | 0.8335 |
> Baseline = zero-shot Meta-Llama-3.1-8B-Instruct with the same prompt template.
## Quick Start
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained(
"meta-llama/Meta-Llama-3.1-8B-Instruct",
load_in_4bit=True,
device_map="auto",
)
model = PeftModel.from_pretrained(base, "jhon53/Llama3_1_8B_Finance_QLoRA")
tokenizer = AutoTokenizer.from_pretrained("jhon53/Llama3_1_8B_Finance_QLoRA")
```
## Training Details
| Param | Value |
|-------|-------|
| Base model | meta-llama/Meta-Llama-3.1-8B-Instruct |
| Method | QLoRA (4-bit NF4 + LoRA bf16) |
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Target modules | q, k, v, o, gate, up, down projections |
| Training data | FinGPT/fingpt-sentiment-train (~76k) |
| Optimizer | AdamW 8-bit |
| Learning rate | 2e-4 (cosine schedule) |
| Epochs | 3 (early stopping, patience=3) |
| Loss | Completion-only cross-entropy |
## Related Repos
- Merged 16-bit: [jhon53/Llama3_1_8B_Finance_QLoRA-merged-16bit](https://huggingface.co/jhon53/Llama3_1_8B_Finance_QLoRA-merged-16bit)
- GGUF q4_k_m: [jhon53/Llama3_1_8B_Finance_QLoRA-GGUF](https://huggingface.co/jhon53/Llama3_1_8B_Finance_QLoRA-GGUF)
|