Text Classification
Transformers
Safetensors
Bulgarian
deberta-v2
Generated from Trainer

Improve model card: Add pipeline tag, update license, expand content and add links

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +77 -12
README.md CHANGED
@@ -1,18 +1,22 @@
1
  ---
 
 
 
 
2
  library_name: transformers
3
- tags:
4
- - generated_from_trainer
5
  metrics:
6
  - accuracy
7
  - f1
 
 
 
 
8
  model-index:
9
  - name: mdeberta-v3-base-subjectivity-sentiment-bulgarian
10
  results: []
11
- license: mit
12
- language:
13
- - bg
14
- base_model:
15
- - microsoft/mdeberta-v3-base
16
  ---
17
 
18
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
@@ -20,7 +24,7 @@ should probably proofread and complete it, then remove this comment. -->
20
 
21
  # mdeberta-v3-base-subjectivity-sentiment-bulgarian
22
 
23
- This model is a fine-tuned version of [microsoft/mdeberta-v3-base](https://huggingface.co/microsoft/mdeberta-v3-base) on the [CheckThat! Lab Task 1 Subjectivity Detection at CLEF 2025](arxiv.org/abs/2507.11764).
24
  It achieves the following results on the evaluation set:
25
  - Loss: 0.4555
26
  - Macro F1: 0.8291
@@ -33,15 +37,50 @@ It achieves the following results on the evaluation set:
33
 
34
  ## Model description
35
 
36
- More information needed
 
 
 
 
37
 
38
  ## Intended uses & limitations
39
 
40
- More information needed
 
 
 
 
 
41
 
42
  ## Training and evaluation data
43
 
44
- More information needed
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
45
 
46
  ## Training procedure
47
 
@@ -73,4 +112,30 @@ The following hyperparameters were used during training:
73
  - Transformers 4.49.0
74
  - Pytorch 2.5.1+cu121
75
  - Datasets 3.3.1
76
- - Tokenizers 0.21.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ base_model:
3
+ - microsoft/mdeberta-v3-base
4
+ language:
5
+ - bg
6
  library_name: transformers
7
+ license: cc-by-4.0
 
8
  metrics:
9
  - accuracy
10
  - f1
11
+ tags:
12
+ - generated_from_trainer
13
+ - text-classification
14
+ pipeline_tag: text-classification
15
  model-index:
16
  - name: mdeberta-v3-base-subjectivity-sentiment-bulgarian
17
  results: []
18
+ datasets:
19
+ - MatteoFasulo/clef2025_checkthat_task1_subjectivity
 
 
 
20
  ---
21
 
22
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 
24
 
25
  # mdeberta-v3-base-subjectivity-sentiment-bulgarian
26
 
27
+ This model is a fine-tuned version of [microsoft/mdeberta-v3-base](https://huggingface.co/microsoft/mdeberta-v3-base) on the [CheckThat! Lab Task 1 Subjectivity Detection at CLEF 2025](https://arxiv.org/abs/2507.11764).
28
  It achieves the following results on the evaluation set:
29
  - Loss: 0.4555
30
  - Macro F1: 0.8291
 
37
 
38
  ## Model description
39
 
40
+ This model is a fine-tuned `mDeBERTaV3-base` classifier designed for **subjectivity detection in news articles**, specifically for the Bulgarian language. It classifies sentences as either **subjective** (opinion-laden) or **objective**.
41
+
42
+ The core innovation of this approach, as detailed in the associated paper, is the enhancement of transformer-based embeddings by integrating sentiment scores derived from an auxiliary model. This sentiment-augmented architecture aims to improve upon standard fine-tuning, demonstrating significant performance boosts, particularly in subjective F1 score. The training also incorporates robust decision threshold calibration to effectively address class imbalance.
43
+
44
+ This model is part of the AI Wizards' participation in the CLEF 2025 CheckThat! Lab Task 1: Subjectivity Detection in News Articles. The broader research explored its application across monolingual, multilingual, and zero-shot settings.
45
 
46
  ## Intended uses & limitations
47
 
48
+ This model is intended for classifying sentences as subjective or objective within news articles, which is a key component in **combating misinformation**, **improving fact-checking pipelines**, and **supporting journalists**. This specific checkpoint is optimized for the **Bulgarian language**.
49
+
50
+ **Limitations:**
51
+ * While the sentiment-augmented approach showed consistent performance gains, the paper notes that BERT-like models still surpassed LLM baselines in various scenarios, indicating areas for further research.
52
+ * The model's effectiveness may vary for domains or linguistic styles significantly different from the news articles it was trained on.
53
+ * Generalization to other languages beyond those explicitly evaluated (Arabic, German, English, Italian, Bulgarian, Greek, Romanian, Polish, Ukrainian) is not guaranteed without further fine-tuning.
54
 
55
  ## Training and evaluation data
56
 
57
+ This model was fine-tuned on the data provided for the **CLEF 2025 CheckThat! Lab Task 1: Subjectivity Detection in News Articles**. Training and development datasets were provided for various languages, including Bulgarian. The final evaluation included additional unseen languages to assess generalization capabilities.
58
+
59
+ To address class imbalance, a common issue across languages, decision threshold calibration was employed during training and optimized on the development set. The base model, `microsoft/mdeberta-v3-base`, was pretrained on a diverse multilingual corpus.
60
+
61
+ ## How to use
62
+
63
+ You can use this model directly with the Hugging Face `transformers` library:
64
+
65
+ ```python
66
+ from transformers import pipeline
67
+
68
+ # Load the text classification pipeline
69
+ classifier = pipeline(
70
+ "text-classification",
71
+ model="MatteoFasulo/mdeberta-v3-base-subjectivity-bulgarian",
72
+ tokenizer="microsoft/mdeberta-v3-base",
73
+ )
74
+
75
+ # Example usage:
76
+ result1 = classifier("По принцип никой не иска войни, но за нещастие те се случват.")
77
+ print(f"Classification: {result1}")
78
+ # Expected output: [{'label': 'SUBJ', 'score': ...}]
79
+
80
+ result2 = classifier("В един момент започнал сам да търси изход за своето спасение и здраве")
81
+ print(f"Classification: {result2}")
82
+ # Expected output: [{'label': 'OBJ', 'score': ...}]
83
+ ```
84
 
85
  ## Training procedure
86
 
 
112
  - Transformers 4.49.0
113
  - Pytorch 2.5.1+cu121
114
  - Datasets 3.3.1
115
+ - Tokenizers 0.21.0
116
+
117
+ ## Code
118
+
119
+ The official code and materials for this project are available on GitHub:
120
+ [https://github.com/MatteoFasulo/clef2025-checkthat](https://github.com/MatteoFasulo/clef2025-checkthat)
121
+
122
+ ## Project Page
123
+
124
+ Explore the collection of models and interactive results on the Hugging Face Hub:
125
+ [AI Wizards @ CLEF 2025 - CheckThat! Lab - Task 1 Subjectivity](https://huggingface.co/collections/MatteoFasulo/clef-2025-checkthat-lab-task-1-subjectivity-6878f0199d302acdfe2ceddb)
126
+
127
+ ## Citation
128
+
129
+ If you find our work helpful or inspiring, please feel free to cite it:
130
+
131
+ ```bibtex
132
+ @misc{fasulo2025aiwizardscheckthat2025,
133
+ title={AI Wizards at CheckThat! 2025: Enhancing Transformer-Based Embeddings with Sentiment for Subjectivity Detection in News Articles},
134
+ author={Matteo Fasulo and Luca Babboni and Luca Tedeschini},
135
+ year={2025},
136
+ eprint={2507.11764},
137
+ archivePrefix={arXiv},
138
+ primaryClass={cs.CL},
139
+ url={https://arxiv.org/abs/2507.11764},
140
+ }
141
+ ```