sartajbhuvaji/Legal-Llama-3.2-3B-Instruct

This model is a fine-tuned version of meta-llama/Llama-3.2-3B-Instruct using QLoRA.

Model description

Fine-tuned Llama 2 model for legal tasks

Training Details

• Base Model: meta-llama/Llama-3.2-3B-Instruct
• Training Method: QLoRA (Quantized Low-Rank Adaptation)
• Framework: PEFT (Parameter-Efficient Fine-Tuning)
• Dataset: nisaar/LLAMA2_Legal_Dataset_4.4k_Instructions
• Training Date: 2024-12-29

Intended Uses

This model is designed for legal domain tasks and should be used in accordance with the base model's intended use cases and limitations.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")

# Load adapter
model = PeftModel.from_pretrained(
    model,
    "sartajbhuvaji/Legal-Llama-3.2-3B-Instruct"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")

# Format prompt
prompt = "<s> [INST] Your prompt here [/INST]"
inputs = tokenizer(prompt, return_tensors="pt")

# Generate
outputs = model.generate(**inputs, max_length=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• This is a fine-tuned model and inherits the limitations of the base model
• The model's performance is limited to the scope and quality of the training dataset
• Legal advice generated by the model should not be considered as professional legal counsel

Training Hyperparameters

• Learning rate: 2e-4
• Epochs: 3
• Batch size: 4
• LoRA rank: 32
• LoRA alpha: 16
• Gradient accumulation steps: 4

Citation

@misc{your-model-name,
  author = {Your Name},
  title = {Your Model Title},
  year = {2024},
  publisher = {HuggingFace},
  journal = {HuggingFace Hub},
  howpublished = {\url{https://huggingface.co/sartajbhuvaji/Legal-Llama-3.2-3B-Instruct}}
}