Instructions to use psychopenguin/legal_llama3.2-3b-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use psychopenguin/legal_llama3.2-3b-instruct with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
model = PeftModel.from_pretrained(base_model, "psychopenguin/legal_llama3.2-3b-instruct")

Transformers

How to use psychopenguin/legal_llama3.2-3b-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="psychopenguin/legal_llama3.2-3b-instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("psychopenguin/legal_llama3.2-3b-instruct")
model = AutoModelForCausalLM.from_pretrained("psychopenguin/legal_llama3.2-3b-instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use psychopenguin/legal_llama3.2-3b-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "psychopenguin/legal_llama3.2-3b-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "psychopenguin/legal_llama3.2-3b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/psychopenguin/legal_llama3.2-3b-instruct

SGLang

How to use psychopenguin/legal_llama3.2-3b-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "psychopenguin/legal_llama3.2-3b-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "psychopenguin/legal_llama3.2-3b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "psychopenguin/legal_llama3.2-3b-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "psychopenguin/legal_llama3.2-3b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use psychopenguin/legal_llama3.2-3b-instruct with Docker Model Runner:
```
docker model run hf.co/psychopenguin/legal_llama3.2-3b-instruct
```

See axolotl config

axolotl version: 0.16.0.dev0

seed: 9
gradient_checkpointing: True
auto_resume_from_checkpoints: True
tokenizer_save_jinja_files: True
trust_remote_code: True
tokenizer_use_fast: True
load_best_model_at_end: true

base_model: meta-llama/Llama-3.2-3B-Instruct
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
chat_template: tokenizer_default

datasets: 
  - path: psychopenguin/indian_legal_dataset_qna
    type: alpaca
    split: train
    

sample_packing: True               
pad_to_sequence_len: True 
sequence_len: 2048
bf16: True
tf32: True
adapter: lora
lora_target_linear: True
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05

flash_attention: True
flash_attn_cross_entropy: True


optimizer: paged_adamw_8bit
learning_rate: 0.0002
lr_scheduler: cosine
gradient_accumulation_steps: 1
micro_batch_size: 4 # change according to gpu vram
num_epochs: 5    # mostly will not reach 5 
neftune_noise_alpha: 5
early_stopping_patience: 6
save_steps: 100
save_total_limit: 3

val_set_size: 0.05
eval_strategy: steps
eval_steps: 100

use_wandb: True
wandb_project: major_project
wandb_name: legal-lm

output_dir: ./final_model
merge_lora: True
hf_use_auth_token: True
hub_model_id: psychopenguin/legal_llama3.2-3b-instruct

legal_llama3.2-3b-instruct

This model is a fine-tuned version of meta-llama/Llama-3.2-3B-Instruct on the psychopenguin/indian_legal_dataset_qna dataset. It achieves the following results on the evaluation set:

Loss: 1.1103
Ppl: 3.0354
Memory/max Active (gib): 12.08
Memory/max Allocated (gib): 12.08
Memory/device Reserved (gib): 19.03

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 4
eval_batch_size: 4
seed: 9
optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 66
training_steps: 2215

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	1.8405	6.2998	12.05	12.05	13.32
1.3598	0.2257	100	1.3577	3.8873	12.08	12.08	19.16
1.2943	0.4515	200	1.2594	3.5233	12.08	12.08	19.03
1.1917	0.6772	300	1.1918	3.2931	12.08	12.08	19.03
1.1383	0.9029	400	1.1406	3.1288	12.08	12.08	19.03
0.9999	1.1287	500	1.1115	3.0389	12.08	12.08	19.03
0.9648	1.3544	600	1.0786	2.9405	12.08	12.08	19.03
0.9499	1.5801	700	1.0521	2.8637	12.08	12.08	19.03
0.9443	1.8059	800	1.0255	2.7884	12.08	12.08	19.03
0.7065	2.0316	900	1.0495	2.8563	12.08	12.08	19.03
0.7210	2.2573	1000	1.0306	2.8026	12.08	12.08	19.03
0.7406	2.4831	1100	1.0208	2.7753	12.08	12.08	19.03
0.6923	2.7088	1200	1.0075	2.7386	12.08	12.08	19.03
0.6635	2.9345	1300	0.9931	2.6997	12.08	12.08	19.03
0.5177	3.1603	1400	1.0536	2.8679	12.08	12.08	19.03
0.5290	3.3860	1500	1.0481	2.8524	12.08	12.08	19.03
0.5298	3.6117	1600	1.0446	2.8424	12.08	12.08	19.03
0.4862	3.8375	1700	1.0451	2.8438	12.08	12.08	19.03
0.3959	4.0632	1800	1.1158	3.0520	12.08	12.08	19.03
0.4204	4.2889	1900	1.1103	3.0354	12.08	12.08	19.03

Framework versions

PEFT 0.18.1
Transformers 5.3.0
Pytorch 2.9.1+cu128
Datasets 4.5.0
Tokenizers 0.22.2

Downloads last month: 1

Model tree for psychopenguin/legal_llama3.2-3b-instruct

Base model

meta-llama/Llama-3.2-3B-Instruct

Adapter

(819)

this model

psychopenguin
/

legal_llama3.2-3b-instruct

legal_llama3.2-3b-instruct

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

Model tree for psychopenguin/legal_llama3.2-3b-instruct

Dataset used to train psychopenguin/legal_llama3.2-3b-instruct