--- language: en license: mit pipeline_tag: text-classification tags: - text-classification - transformers - pytorch - onnx - multi-label-classification - multi-class-classification - emotion - bert - go_emotions - emotion-classification - tensorflow datasets: - google-research-datasets/go_emotions metrics: - f1 - precision - recall widget: - text: Iā€™m just chilling today. example_title: Neutral Example - text: Thank you for saving my life! example_title: Gratitude Example - text: Iā€™m nervous about my exam tomorrow. example_title: Nervousness Example - text: I love my new puppy so much! example_title: Love Example - text: Iā€™m so relieved the storm passed. example_title: Relief Example base_model: - google-bert/bert-base-uncased --- # GoEmotions BERT Classifier Fine-tuned [BERT-base-uncased](https://huggingface.co/bert-base-uncased) on [GoEmotions](https://huggingface.co/datasets/go_emotions) for multi-label classification (28 emotions). This updated version includes improved Macro F1, ONNX support for efficient inference, and visualizations for better interpretability. ## Model Details - **Architecture**: BERT-base-uncased (110M parameters) - **Training Data**: [GoEmotions](https://huggingface.co/datasets/google-research-datasets/go_emotions) (58k Reddit comments, 28 emotions) - **Loss Function**: Focal Loss (alpha=1, gamma=2) - **Optimizer**: AdamW (lr=2e-5, weight_decay=0.01) - **Epochs**: 5 - **Batch Size**: 16 - **Max Length**: 128 - **Hardware**: Kaggle P100 GPU (16GB) ## Try It Out For accurate predictions with optimized thresholds, use the [Gradio demo](https://logasanjeev-goemotions-bert-demo.hf.space). The demo now includes preprocessed text and the top 5 predicted emotions, in addition to thresholded predictions. Example predictions: - **Input**: "Iā€™m thrilled to win this award! šŸ˜„" - **Output**: `excitement: 0.5836, joy: 0.5290` - **Input**: "This is so frustrating, nothing works. šŸ˜£" - **Output**: `annoyance: 0.6147, anger: 0.4669` - **Input**: "I feel so sorry for what happened. šŸ˜¢" - **Output**: `sadness: 0.5321, remorse: 0.9107` ## Performance - **Micro F1**: 0.6006 (optimized thresholds) - **Macro F1**: 0.5390 - **Precision**: 0.5371 - **Recall**: 0.6812 - **Hamming Loss**: 0.0377 - **Avg Positive Predictions**: 1.4789 ### Class-Wise Performance The following table shows per-class metrics on the test set using optimized thresholds (see `optimized_thresholds.json`): | Emotion | F1 Score | Precision | Recall | Support | |---------------|----------|-----------|--------|---------| | admiration | 0.6987 | 0.6649 | 0.7361 | 504 | | amusement | 0.8071 | 0.7635 | 0.8561 | 264 | | anger | 0.5030 | 0.6176 | 0.4242 | 198 | | annoyance | 0.3892 | 0.3297 | 0.4750 | 320 | | approval | 0.3915 | 0.2966 | 0.5755 | 351 | | caring | 0.4473 | 0.5196 | 0.3926 | 135 | | confusion | 0.4714 | 0.4861 | 0.4575 | 153 | | curiosity | 0.5781 | 0.4442 | 0.8275 | 284 | | desire | 0.5229 | 0.5714 | 0.4819 | 83 | | disappointment| 0.3333 | 0.2906 | 0.3907 | 151 | | disapproval | 0.4323 | 0.3405 | 0.5918 | 267 | | disgust | 0.4926 | 0.6250 | 0.4065 | 123 | | embarrassment | 0.4912 | 0.7000 | 0.3784 | 37 | | excitement | 0.4571 | 0.4486 | 0.4660 | 103 | | fear | 0.5860 | 0.4599 | 0.8077 | 78 | | gratitude | 0.9102 | 0.9450 | 0.8778 | 352 | | grief | 0.3333 | 0.3333 | 0.3333 | 6 | | joy | 0.6135 | 0.6061 | 0.6211 | 161 | | love | 0.8065 | 0.7826 | 0.8319 | 238 | | nervousness | 0.4348 | 0.4348 | 0.4348 | 23 | | optimism | 0.5564 | 0.5436 | 0.5699 | 186 | | pride | 0.5217 | 0.8571 | 0.3750 | 16 | | realization | 0.2513 | 0.5217 | 0.1655 | 145 | | relief | 0.5833 | 0.5385 | 0.6364 | 11 | | remorse | 0.6800 | 0.5426 | 0.9107 | 56 | | sadness | 0.5570 | 0.5845 | 0.5321 | 156 | | surprise | 0.5562 | 0.4772 | 0.6667 | 141 | | neutral | 0.6867 | 0.5821 | 0.8372 | 1787 | ### Visualizations #### Class-Wise F1 Scores ![Class-Wise F1 Scores](class_wise_f1_plot.png) #### Training Curves ![Training and Validation Loss and Micro F1](training_curves_plot.png) ## Training Insights The model was trained for 5 epochs with Focal Loss to handle class imbalance. Training and validation curves show consistent improvement: - Training Loss decreased from 0.0429 to 0.0134. - Validation Micro F1 peaked at 0.5874 (epoch 5). - See the training curves plot above for details. ## Usage ### Quick Inference with inference.py (Recommended for PyTorch) The easiest way to use the model with PyTorch is to programmatically fetch and use `inference.py` from the repository. The script handles all preprocessing, model loading, and inference for you. #### Programmatic Download and Inference Run the following Python script to download `inference.py` and make predictions: ```python # Install required dependencies !pip install transformers torch huggingface_hub emoji -q import shutil import os from huggingface_hub import hf_hub_download from importlib import import_module # Download inference.py repo_id = "logasanjeev/goemotions-bert" local_file = hf_hub_download(repo_id=repo_id, filename="inference.py") # Copy the file to the current working directory current_dir = os.getcwd() destination = os.path.join(current_dir, "inference.py") shutil.copy(local_file, destination) # Import and use the predict_emotions function inference_module = import_module("inference") predict_emotions = inference_module.predict_emotions # Make predictions text = "Iā€™m thrilled to win this award! šŸ˜„" result, processed = predict_emotions(text) print(f"Input: {text}") print(f"Processed: {processed}") print("Predicted Emotions:") print(result) ``` #### Expected Output: ``` Input: Iā€™m thrilled to win this award! šŸ˜„ Processed: iā€™m thrilled to win this award ! grinning_face_with_smiling_eyes Predicted Emotions: excitement: 0.5836 joy: 0.5290 ``` #### Alternative: Manual Download If you prefer to download `inference.py` manually: 1. Install the required dependencies: ```bash pip install transformers torch huggingface_hub emoji ``` 2. Download `inference.py` from the repository. 3. Use it in Python or via the command line. **Python Example:** ```python from inference import predict_emotions result, processed = predict_emotions("Iā€™m thrilled to win this award! šŸ˜„") print(f"Input: Iā€™m thrilled to win this award! šŸ˜„") print(f"Processed: {processed}") print("Predicted Emotions:") print(result) ``` **Command-Line Example:** ```bash python inference.py "Iā€™m thrilled to win this award! šŸ˜„" ``` ### Quick Inference with onnx_inference.py (Recommended for ONNX) For faster and more efficient inference using ONNX, you can use `onnx_inference.py`. This script leverages ONNX Runtime for inference, which is typically more lightweight than PyTorch. #### Programmatic Download and Inference Run the following Python script to download `onnx_inference.py` and make predictions: ```python # Install required dependencies !pip install transformers onnxruntime huggingface_hub emoji numpy -q import shutil import os from huggingface_hub import hf_hub_download from importlib import import_module # Download onnx_inference.py repo_id = "logasanjeev/goemotions-bert" local_file = hf_hub_download(repo_id=repo_id, filename="onnx_inference.py") # Copy the file to the current working directory current_dir = os.getcwd() destination = os.path.join(current_dir, "onnx_inference.py") shutil.copy(local_file, destination) # Import and use the predict_emotions function onnx_inference_module = import_module("onnx_inference") predict_emotions = onnx_inference_module.predict_emotions # Make predictions text = "Iā€™m thrilled to win this award! šŸ˜„" result, processed = predict_emotions(text) print(f"Input: {text}") print(f"Processed: {processed}") print("Predicted Emotions:") print(result) ``` #### Expected Output: ``` Input: Iā€™m thrilled to win this award! šŸ˜„ Processed: iā€™m thrilled to win this award ! grinning_face_with_smiling_eyes Predicted Emotions: excitement: 0.5836 joy: 0.5290 ``` #### Alternative: Manual Download If you prefer to download `onnx_inference.py` manually: 1. Install the required dependencies: ```bash pip install transformers onnxruntime huggingface_hub emoji numpy ``` 2. Download `onnx_inference.py` from the repository. 3. Use it in Python or via the command line. **Python Example:** ```python from onnx_inference import predict_emotions result, processed = predict_emotions("Iā€™m thrilled to win this award! šŸ˜„") print(f"Input: Iā€™m thrilled to win this award! šŸ˜„") print(f"Processed: {processed}") print("Predicted Emotions:") print(result) ``` **Command-Line Example:** ```bash python onnx_inference.py "Iā€™m thrilled to win this award! šŸ˜„" ``` ### Preprocessing Before inference, preprocess text to match training conditions: - Replace user mentions (`u/username`) with `[USER]`. - Replace subreddits (`r/subreddit`) with `[SUBREDDIT]`. - Replace URLs with `[URL]`. - Convert emojis to text using `emoji.demojize` (e.g., šŸ˜Š ā†’ `smiling_face_with_smiling_eyes`). - Lowercase the text. ### PyTorch Inference ```python from transformers import BertForSequenceClassification, BertTokenizer import torch import json import requests import re import emoji # Preprocessing function def preprocess_text(text): text = re.sub(r'u/\w+', '[USER]', text) text = re.sub(r'r/\w+', '[SUBREDDIT]', text) text = re.sub(r'http[s]?://\S+', '[URL]', text) text = emoji.demojize(text, delimiters=(" ", " ")) text = text.lower() return text # Load model and tokenizer repo_id = "logasanjeev/goemotions-bert" model = BertForSequenceClassification.from_pretrained(repo_id) tokenizer = BertTokenizer.from_pretrained(repo_id) # Load thresholds and labels thresholds_url = f"https://huggingface.co/{repo_id}/raw/main/optimized_thresholds.json" thresholds_data = json.loads(requests.get(thresholds_url).text) emotion_labels = thresholds_data["emotion_labels"] thresholds = thresholds_data["thresholds"] # Predict text = "Iā€™m just chilling today." processed_text = preprocess_text(text) encodings = tokenizer(processed_text, padding='max_length', truncation=True, max_length=128, return_tensors='pt') with torch.no_grad(): logits = torch.sigmoid(model(**encodings).logits).numpy()[0] predictions = [(emotion_labels[i], round(logit, 4)) for i, (logit, thresh) in enumerate(zip(logits, thresholds)) if logit >= thresh] predictions = sorted(predictions, key=lambda x: x[1], reverse=True) print(predictions) # Output: [('neutral', 0.8147)] ``` ### ONNX Inference For a simplified ONNX inference experience, use `onnx_inference.py` as shown above. Alternatively, you can use the manual approach below: ```python import onnxruntime as ort import numpy as np # Download ONNX model onnx_url = f"https://huggingface.co/{repo_id}/raw/main/model.onnx" with open("model.onnx", "wb") as f: f.write(requests.get(onnx_url).content) # Preprocess and predict text = "Iā€™m thrilled to win this award! šŸ˜„" processed_text = preprocess_text(text) encodings = tokenizer(processed_text, padding='max_length', truncation=True, max_length=128, return_tensors='np') session = ort.InferenceSession("model.onnx") inputs = { 'input_ids': encodings['input_ids'].astype(np.int64), 'attention_mask': encodings['attention_mask'].astype(np.int64) } logits = session.run(None, inputs)[0][0] logits = 1 / (1 + np.exp(-logits)) # Sigmoid predictions = [(emotion_labels[i], round(logit, 4)) for i, (logit, thresh) in enumerate(zip(logits, thresholds)) if logit >= thresh] predictions = sorted(predictions, key=lambda x: x[1], reverse=True) print(predictions) # Output: [('excitement', 0.5836), ('joy', 0.5290)] ``` ## License This model is licensed under the MIT License. See [LICENSE](LICENSE) for details. ## Usage Notes - The model performs best on Reddit-style comments with similar preprocessing. - Rare emotions (e.g., `grief`, support=6) have lower F1 scores due to limited data. - ONNX inference requires `onnxruntime` and compatible hardware (opset 14). ## Inference Providers This model isn't deployed by any Inference Provider. šŸ™‹ Ask for provider support