emotion_classification

This model is a fine-tuned version of google/paligemma-3b-pt-224 on the FastJobs/Visual_Emotional_Analysis dataset.

Training Data

This model was trained on the FastJobs/Visual_Emotional_Analysis dataset.

The dataset contains:

• 800 images annotated with 8 emotion labels:
  • Anger
  • Contempt
  • Disgust
  • Fear
  • Happy
  • Neutral
  • Sad
  • Surprise

Intended uses & limitations

Intended Uses

• Emotion classification from visual inputs (images).

Limitations

• May reflect biases from the training dataset.
• Performance may degrade in domains outside the training data.
• Not suitable for critical or sensitive decision-making tasks.

How to use this model

from transformers import (PaliGemmaProcessor,PaliGemmaForConditionalGeneration,)
from transformers.image_utils import load_image
import torch
from transformers import BitsAndBytesConfig
from peft import get_peft_model
from huggingface_hub import login
from PIL import Image
login(api_key)

device = "cuda" if torch.cuda.is_available() else "CPU"

bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_type=torch.bfloat16
)

# Load base model
model_id = "google/paligemma-3b-pt-224"
model = PaliGemmaForConditionalGeneration.from_pretrained(model_id, quantization_config=bnb_config, device_map="auto")
processor = PaliGemmaProcessor.from_pretrained(model_id)

# Load adapter
adapter_path = "digo-prayudha/emotion_classification" 
model = PeftModel.from_pretrained(model, adapter_path)

image = Image.open("image.jpg").convert("RGB")

prompt = (
    "Classify the emotion expressed in this image."
)

inputs = processor(
    text=prompt,
    images=image,
    return_tensors="pt",
    padding="longest",
    tokenize_newline_separately=False
).to(model.device)

model.eval() 
with torch.no_grad():
    outputs = model.generate(**inputs)

decoded_output = processor.decode(outputs[0], skip_special_tokens=True)

print("Predicted Emotion:", decoded_output)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 4
• optimizer: Use adamw_hf with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 2
• num_epochs: 5

Training results

Step	Validation Loss
100	2.684700
200	1.282700
300	1.085600
400	0.984500
500	0.861300
600	0.822900
700	0.807100
800	0.753300

Framework versions

• PEFT 0.14.0
• Transformers 4.47.1
• Pytorch 2.5.1+cu121
• Datasets 3.2.0
• Tokenizers 0.21.0