MouezYazidi/modernBERT-base-bilingual-CampingReviewsSentiment

modernBERT-base-bilingual-CampingReviewsSentiment is a fine-tuned version of answerdotai/ModernBERT-base using a bilingual sentiment dataset MouezYazidi/campSentiment-Bilingual.

Model supports bilingual sentiment classification ( english & french )

Model Evaluation

After fine-tuning the model, we evaluate its performance on the test dataset from MouezYazidi/campSentiment-Bilingual

Class	Precision	Recall	F1-Score	Support
0	0.88	0.67	0.76	102
1	0.89	0.94	0.93	298
Accuracy			0.89	400
Macro Avg	0.89	0.82	0.85	400
Weighted Avg	0.89	0.89	0.89	400

How to use

Requirements

Since transformers only supports the ModernBERT architecture from version 4.48.0.dev0, use the following command to get the required version:

pip install "git+https://github.com/huggingface/transformers.git@6e0515e99c39444caae39472ee1b2fd76ece32f1" --upgrade

Install FlashAttention to accelerate inference performance

pip install flash-attn==2.7.2.post1

Quick start

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Set device (GPU if available, else CPU)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load model and tokenizer
model_id = "MouezYazidi/modernBERT-base-bilingual-CampingReviewsSentiment"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id, torch_dtype=torch.float16).to(device)
model.eval()

def predict_sentiment(text: str):
    """Predicts sentiment of the given text using the model."""
    inputs = tokenizer([text], return_tensors="pt").to(device)

    with torch.no_grad():  # Use no_grad for inference optimization
        outputs = model(**inputs)
        prediction = outputs.logits.argmax(dim=-1).item()

    return 'positive' if prediction==1 else 'negative'

# Example usage
text = """
Place is amazing. Entertainment is next level brilliant
Pool areas excellent
Literally no complaints at all. Staff so friendly everywhere. Brought 2 teenagers they had a great time aswell as 3 and 9 year old
Fantastic time had by us all
"""
prediction = predict_sentiment(text)
print(f"Predicted Sentiment: {prediction}")

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 2
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 5

Framework versions

• Transformers 4.48.0.dev0
• Pytorch 2.5.0+cu124
• Datasets 3.1.0
• Tokenizers 0.21.0