Hugging Face's logo

thejosango
/
nuha-ajp-mlm

Fill-Mask
Transformers
PyTorch
Safetensors
Arabic
bert
masked-language-modeling
arabic
social-media
pilot
Model card Files
xet
 Community
4
yazansh commited on
Commit
375b07d
·
verified ·
1 Parent(s): ce20f49

mlm-6

Browse files
Files changed (9) hide show
  1. README.md +61 -24
  2. config.json +2 -2
  3. config.toml +11 -11
  4. pytorch_model.bin +2 -2
  5. special_tokens_map.json +3 -0
  6. tokenizer.json +0 -0
  7. tokenizer_config.json +5 -10
  8. training_args.bin +1 -1
  9. vocab.txt +0 -0
README.md CHANGED
@@ -1,5 +1,6 @@
1
  ---
2
- base_model: aubmindlab/bert-base-arabertv02-twitter
 
3
  tags:
4
  - generated_from_trainer
5
  datasets:
@@ -14,9 +15,9 @@ should probably proofread and complete it, then remove this comment. -->
14
 
15
  # nuha-mlm
16
 
17
- This model is a fine-tuned version of [aubmindlab/bert-base-arabertv02-twitter](https://huggingface.co/aubmindlab/bert-base-arabertv02-twitter) on the nuha-dataset dataset.
18
  It achieves the following results on the evaluation set:
19
- - Loss: 4.1000
20
 
21
  ## Model description
22
 
@@ -35,33 +36,69 @@ More information needed
35
  ### Training hyperparameters
36
 
37
  The following hyperparameters were used during training:
38
- - learning_rate: 5e-05
39
- - train_batch_size: 32
40
- - eval_batch_size: 32
41
  - seed: 42
42
- - gradient_accumulation_steps: 2
43
- - total_train_batch_size: 64
44
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
- - lr_scheduler_type: constant
46
- - lr_scheduler_warmup_steps: 1000.0
47
- - num_epochs: 3
48
  - label_smoothing_factor: 0.1
49
 
50
  ### Training results
51
 
52
- | Training Loss | Epoch | Step | Validation Loss |
53
- |:-------------:|:-----:|:----:|:---------------:|
54
- | 4.5289 | 0.25 | 500 | 4.3484 |
55
- | 4.4276 | 0.5 | 1000 | 4.2974 |
56
- | 4.402 | 0.75 | 1500 | 4.2336 |
57
- | 4.3494 | 1.0 | 2000 | 4.1649 |
58
- | 4.2526 | 1.26 | 2500 | 4.1969 |
59
- | 4.2335 | 1.51 | 3000 | 4.1577 |
60
- | 4.2401 | 1.76 | 3500 | 4.1136 |
61
- | 4.1889 | 2.01 | 4000 | 4.1469 |
62
- | 4.1366 | 2.26 | 4500 | 4.1297 |
63
- | 4.1435 | 2.51 | 5000 | 4.0907 |
64
- | 4.1207 | 2.76 | 5500 | 4.1000 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
65
 
66
 
67
  ### Framework versions
 
1
  ---
2
+ license: apache-2.0
3
+ base_model: bert-base-uncased
4
  tags:
5
  - generated_from_trainer
6
  datasets:
 
15
 
16
  # nuha-mlm
17
 
18
+ This model is a fine-tuned version of [bert-base-uncased](https://huggingface.co/bert-base-uncased) on the nuha-dataset dataset.
19
  It achieves the following results on the evaluation set:
20
+ - Loss: 5.7881
21
 
22
  ## Model description
23
 
 
36
  ### Training hyperparameters
37
 
38
  The following hyperparameters were used during training:
39
+ - learning_rate: 0.0001
40
+ - train_batch_size: 128
41
+ - eval_batch_size: 128
42
  - seed: 42
 
 
43
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
44
+ - lr_scheduler_type: linear
45
+ - lr_scheduler_warmup_steps: 5000.0
46
+ - num_epochs: 25
47
  - label_smoothing_factor: 0.1
48
 
49
  ### Training results
50
 
51
+ | Training Loss | Epoch | Step | Validation Loss |
52
+ |:-------------:|:-----:|:-----:|:---------------:|
53
+ | 9.2913 | 0.5 | 500 | 8.9187 |
54
+ | 8.5364 | 1.0 | 1000 | 8.4350 |
55
+ | 8.3161 | 1.51 | 1500 | 8.1825 |
56
+ | 8.1276 | 2.01 | 2000 | 7.9889 |
57
+ | 8.0073 | 2.51 | 2500 | 7.8551 |
58
+ | 7.8206 | 3.01 | 3000 | 7.7101 |
59
+ | 7.7224 | 3.51 | 3500 | 7.6287 |
60
+ | 7.6264 | 4.02 | 4000 | 7.4921 |
61
+ | 7.5062 | 4.52 | 4500 | 7.4265 |
62
+ | 7.408 | 5.02 | 5000 | 7.2955 |
63
+ | 7.2433 | 5.52 | 5500 | 7.1658 |
64
+ | 7.1062 | 6.02 | 6000 | 7.0132 |
65
+ | 6.9815 | 6.53 | 6500 | 6.9592 |
66
+ | 6.9258 | 7.03 | 7000 | 6.8659 |
67
+ | 6.801 | 7.53 | 7500 | 6.7687 |
68
+ | 6.7334 | 8.03 | 8000 | 6.7175 |
69
+ | 6.6422 | 8.53 | 8500 | 6.6583 |
70
+ | 6.5958 | 9.04 | 9000 | 6.5778 |
71
+ | 6.5044 | 9.54 | 9500 | 6.5266 |
72
+ | 6.4488 | 10.04 | 10000 | 6.4990 |
73
+ | 6.3713 | 10.54 | 10500 | 6.4321 |
74
+ | 6.3531 | 11.04 | 11000 | 6.4003 |
75
+ | 6.2718 | 11.55 | 11500 | 6.3708 |
76
+ | 6.2478 | 12.05 | 12000 | 6.2975 |
77
+ | 6.1886 | 12.55 | 12500 | 6.2802 |
78
+ | 6.1405 | 13.05 | 13000 | 6.2521 |
79
+ | 6.0967 | 13.55 | 13500 | 6.1969 |
80
+ | 6.0378 | 14.06 | 14000 | 6.1440 |
81
+ | 5.9893 | 14.56 | 14500 | 6.1456 |
82
+ | 5.9551 | 15.06 | 15000 | 6.1178 |
83
+ | 5.9215 | 15.56 | 15500 | 6.0791 |
84
+ | 5.8603 | 16.06 | 16000 | 6.0696 |
85
+ | 5.7924 | 16.57 | 16500 | 6.0288 |
86
+ | 5.7764 | 17.07 | 17000 | 6.0061 |
87
+ | 5.7545 | 17.57 | 17500 | 5.9600 |
88
+ | 5.7025 | 18.07 | 18000 | 5.9287 |
89
+ | 5.6703 | 18.57 | 18500 | 5.9549 |
90
+ | 5.6887 | 19.08 | 19000 | 5.8808 |
91
+ | 5.629 | 19.58 | 19500 | 5.8967 |
92
+ | 5.6139 | 20.08 | 20000 | 5.8629 |
93
+ | 5.5925 | 20.58 | 20500 | 5.8651 |
94
+ | 5.5506 | 21.08 | 21000 | 5.8567 |
95
+ | 5.5177 | 21.59 | 21500 | 5.8216 |
96
+ | 5.479 | 22.09 | 22000 | 5.8242 |
97
+ | 5.468 | 22.59 | 22500 | 5.8100 |
98
+ | 5.4426 | 23.09 | 23000 | 5.7828 |
99
+ | 5.4648 | 23.59 | 23500 | 5.7881 |
100
+ | 5.4199 | 24.1 | 24000 | 5.8162 |
101
+ | 5.3976 | 24.6 | 24500 | 5.7881 |
102
 
103
 
104
  ### Framework versions
config.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
- "_name_or_path": "aubmindlab/bert-base-arabertv02-twitter",
3
  "architectures": [
4
  "BertForMaskedLM"
5
  ],
@@ -22,5 +22,5 @@
22
  "transformers_version": "4.32.1",
23
  "type_vocab_size": 2,
24
  "use_cache": true,
25
- "vocab_size": 64000
26
  }
 
1
  {
2
+ "_name_or_path": "bert-base-uncased",
3
  "architectures": [
4
  "BertForMaskedLM"
5
  ],
 
22
  "transformers_version": "4.32.1",
23
  "type_vocab_size": 2,
24
  "use_cache": true,
25
+ "vocab_size": 17513
26
  }
config.toml CHANGED
@@ -1,5 +1,5 @@
1
  [experiment]
2
- name = "mlm-5"
3
  type = "mlm"
4
 
5
 
@@ -11,18 +11,18 @@ undersampling_strategy = false
11
 
12
 
13
  [model]
14
- pretrained_model_name_or_path = "aubmindlab/bert-base-arabertv02-twitter"
15
-
16
 
17
  [training]
18
- num_train_epochs = 3
19
- warmup_steps = 1e3
20
- lr_scheduler_type = "constant"
21
- learning_rate = 5e-5
22
- per_device_train_batch_size = 32
23
- per_device_eval_batch_size = 32
24
- gradient_accumulation_steps = 2
25
- weight_decay = 0.00
26
  label_smoothing_factor = 0.1
27
  weighted_loss = false
28
  early_stopping_patience = 5
 
1
  [experiment]
2
+ name = "mlm-6"
3
  type = "mlm"
4
 
5
 
 
11
 
12
 
13
  [model]
14
+ pretrained_model_name_or_path = "from-scratch"
15
+ num_hidden_layers = 8
16
 
17
  [training]
18
+ num_train_epochs = 25
19
+ warmup_steps = 5e3
20
+ lr_scheduler_type = "linear"
21
+ learning_rate = 1e-4
22
+ per_device_train_batch_size = 128
23
+ per_device_eval_batch_size = 128
24
+ gradient_accumulation_steps = 1
25
+ weight_decay = 0.01
26
  label_smoothing_factor = 0.1
27
  weighted_loss = false
28
  early_stopping_patience = 5
pytorch_model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:7aed42dcd14312adec1767d61596c1213bfdac1ca5e6620ff838b7c82551b6a2
3
- size 541104501
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f224c9401d608c9a9f895b65da3fdfc19e506c37b2bf03f924513528cda25ae7
3
+ size 398110453
special_tokens_map.json CHANGED
@@ -1,4 +1,7 @@
1
  {
 
 
 
2
  "cls_token": "[CLS]",
3
  "mask_token": "[MASK]",
4
  "pad_token": "[PAD]",
 
1
  {
2
+ "additional_special_tokens": [
3
+ "[URL]"
4
+ ],
5
  "cls_token": "[CLS]",
6
  "mask_token": "[MASK]",
7
  "pad_token": "[PAD]",
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
@@ -1,21 +1,16 @@
1
  {
 
 
 
2
  "clean_up_tokenization_spaces": true,
3
  "cls_token": "[CLS]",
4
- "do_basic_tokenize": true,
5
- "do_lower_case": false,
6
  "mask_token": "[MASK]",
7
- "max_len": 512,
8
  "model_max_length": 512,
9
- "never_split": [
10
- "[بريد]",
11
- "[مستخدم]",
12
- "[رابط]"
13
- ],
14
  "pad_token": "[PAD]",
15
  "sep_token": "[SEP]",
16
  "strip_accents": null,
17
  "tokenize_chinese_chars": true,
18
  "tokenizer_class": "BertTokenizer",
19
- "unk_token": "[UNK]",
20
- "use_fast": true
21
  }
 
1
  {
2
+ "additional_special_tokens": [
3
+ "[URL]"
4
+ ],
5
  "clean_up_tokenization_spaces": true,
6
  "cls_token": "[CLS]",
7
+ "do_lower_case": true,
 
8
  "mask_token": "[MASK]",
 
9
  "model_max_length": 512,
 
 
 
 
 
10
  "pad_token": "[PAD]",
11
  "sep_token": "[SEP]",
12
  "strip_accents": null,
13
  "tokenize_chinese_chars": true,
14
  "tokenizer_class": "BertTokenizer",
15
+ "unk_token": "[UNK]"
 
16
  }
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:591a687be7219249edf221ee703b4450d76d95a56bc77b37a6061088d416e8a0
3
  size 4091
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:29ad69535f054100cad43211110e83041f4e88fa0d78ec0f7024840e795e7944
3
  size 4091
vocab.txt CHANGED
The diff for this file is too large to render. See raw diff