Update README.md

README.md

@@ -10,18 +10,6 @@ datasets:
 metrics:
   - perplexity
 pipeline_tag: text-generation
-widget:
-  - text: "A capital do Brasil é"
-    example_title: Exemplo
-  - text: "O principal rio do Brasil é"
-    example_title: Exemplo
-inference:
-  parameters:
-    repetition_penalty: 1.2
-    temperature: 0.1
-    top_k: 50
-    top_p: 1.0
-    max_new_tokens: 150
 co2_eq_emissions:
   emissions: 124630
   source: CodeCarbon
@@ -29,7 +17,7 @@ co2_eq_emissions:
   geographical_location: Germany
   hardware_used: NVIDIA A40
 model-index:
-  - name: Gigaverbo-v2-ablation-Synth-1.5B
+  - name: GigaVerbo-v2-ablation-Synth-1.5B
     results:
       - task:
           type: text-generation
@@ -58,7 +46,7 @@ model-index:
             num_few_shot: 5
         metrics:
           - type: acc_norm
-            value: 43.9 
+            value: 43.9
             name: accuracy (normalized)
         source:
           url: https://github.com/Nkluge-correa/lm-evaluation-harness
@@ -74,7 +62,7 @@ model-index:
             num_few_shot: 5
         metrics:
           - type: acc
-            value: 56.1 z
+            value: 56.1
             name: accuracy
         source:
           url: https://github.com/Nkluge-correa/lm-evaluation-harness
@@ -117,7 +105,7 @@ model-index:
 
 ## Model Summary
 
-**[Gigaverbo-v2-ablation-Synth-1.5B](https://huggingface.co/Polygl0t/Gigaverbo-v2-ablation-Synth-1.5B)** is a decoder-transformer natively pretrained in Portuguese. This model is part of an ablation study to measure the impact of our educational data filtering/augmentation strategy on the dowstream performance of models trained with [Gigaverbo-v2](https://huggingface.co/datasets/Polygl0t/gigaverbo-v2) and [Gigaverbo-v2-synth](https://huggingface.co/datasets/Polygl0t/gigaverbo-v2-synth). Gigaverbo-v2-ablation-Synth-1.5B was trained with ~46 billion tokens, those being a repetition of the synthetic data from Gigaverbo-v2-synth. This model has 1.5 billion parameters and a context length of 4096 tokens.
+**[GigaVerbo-v2-ablation-Synth-1.5B](https://huggingface.co/Polygl0t/GigaVerbo-v2-ablation-Synth-1.5B)** is a decoder-transformer natively pretrained in Portuguese. This model is part of an ablation study to measure the impact of our educational data filtering/augmentation strategy on the downstream performance of models trained with [GigaVerbo-v2](https://huggingface.co/datasets/Polygl0t/gigaverbo-v2) and [GigaVerbo-v2-synth](https://huggingface.co/datasets/Polygl0t/gigaverbo-v2-synth). Gigaverbo-v2-ablation-Synth-1.5B was trained with ~46 billion tokens, those being a repetition of the synthetic data from Gigaverbo-v2-synth. This model has 1.5 billion parameters and a context length of 4096 tokens.
 
 ## Details
 
@@ -134,7 +122,7 @@ model-index:
 - **Emissions:** 124 KgCO2 (Germany)
 - **Total energy consumption:** 327 kWh
 
-This repository has the [source code](https://github.com/Nkluge-correa/polyglot) used to train this model. The full configuration used for training is available in the following config file:
+This repository has the [source code](https://github.com/Polygl0t/Polygl0t) used to train this model. The complete configuration used for training is available in the following config file:
 
 - Single stage (linear warmup with cosine decay): [training_config.yaml](training_config.yaml)
 
@@ -143,7 +131,7 @@ The main branch of this repository contains the final checkpoint saved at step 2
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-model_id = "Polygl0t/Gigaverbo-v2-ablation-Synth-1.5B"
+model_id = "Polygl0t/GigaVerbo-v2-ablation-Synth-1.5B"
 revision = "step-2000"  # Change this to the desired checkpoint branch
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(model_id, revision=revision)
@@ -153,49 +141,23 @@ Or, you can access all the revisions for the models via the following code snipp
 
 ```python
 from huggingface_hub import list_repo_refs
-out = list_repo_refs("Polygl0t/Gigaverbo-v2-ablation-Synth-1.5B")
+out = list_repo_refs("Polygl0t/GigaVerbo-v2-ablation-Synth-1.5B")
 branches = [b.name for b in out.branches]
 print(branches)
 ```
 
-<details>
-<summary><b>Learning Curves 📈</b></summary>
-
-![Learning Curves](./learning_curves.png)
-
-This plot illustrates the evolution of model performance (measured by loss and perplexity) as a function of training time, measured in tokens seen during training.
-
-</details>
-
 ## Intended Uses
 
-The primary intended use of this model is to serve as a baseline for evaluating the impact of data quality and filtering on Portuguese language model performance. Researchers and practitioners can utilize this model as a reference point for further ablation studies or to compare against other models trained with different data mixtures.
+The primary intended use of this model is to serve as a baseline for evaluating the impact of data quality and filtering on Portuguese language model performance. Researchers and practitioners can use this model as a reference point for further ablation studies or for comparison with other models trained on different data mixtures.
 
 ## Basic usage
 
-### Via HuggingFace Transformers
-
-Using the `pipeline`:
-
-```python
-from transformers import pipeline
-
-generator = pipeline("text-generation", model="Polygl0t/Gigaverbo-v2-ablation-Synth-1.5B")
-
-completions  = generator("A capital de Portugal é", num_return_sequences=2, max_new_tokens=150)
-
-for comp in completions:
-  print(f"🤖 {comp['generated_text']}")
-```
-
-Using the `AutoTokenizer` and `AutoModelForCausalLM`:
-
 ```python
 from transformers import GenerationConfig, TextGenerationPipeline, AutoTokenizer, AutoModelForCausalLM
 import torch
 
 # Specify the model and tokenizer
-model_id = "Polygl0t/Gigaverbo-v2-ablation-Synth-1.5B"
+model_id = "Polygl0t/GigaVerbo-v2-ablation-Synth-1.5B"
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(model_id)
 
@@ -222,132 +184,33 @@ completion = generator(prompt, generation_config=generation_config)
 print(completion[0]['generated_text'])
 ```
 
-### Via vLLM Inference
-
-```python
-from vllm import LLM, SamplingParams
-from transformers import AutoTokenizer
-import torch
-
-model_id = "Polygl0t/Gigaverbo-v2-ablation-Synth-1.5B"
-prompt = "A capital de Portugal é"
-
-tokenizer = AutoTokenizer.from_pretrained(
-    model_id,
-    use_fast=True,
-)
-
-model = LLM(
-    model=model_id,
-)
-
-sampling_params = SamplingParams(
-    max_tokens=150,
-    stop=[tokenizer.eos_token],
-    stop_token_ids=[tokenizer.eos_token_id],
-    n=2,
-    temperature=0.1,
-    repetition_penalty=1.2,
-    top_k=50,
-    top_p=1.0
-)
-
-outputs = model.generate([prompt], sampling_params)
-
-for output in outputs:
-    print(f"🤖 {output.outputs[0].text}")
-```
-
 ## Evaluations
 
-The table below compares our ablation models with checkpoints from the first [Tucano series](https://huggingface.co/TucanoBR). Given that Tucanos have also have (1) been trained on Portuguese data of a similar nature, and (2) provide several checkpoints at different training stages. To keep our comparison fair, we selected Tucano checkpoints that are closest in terms of training tokens seen (31B and 52B tokens) to our ablation models (46B tokens), and that are of similar size (1.1B and 2.4B parameters).
-
-|                                            | ARC Challenge |    Calame | Global PIQA | HellaSwag |  Lambada |
-| :----------------------------------------- | ------------: | --------: | ----------: | --------: | -------: |
-| Gigaverbo-v2-ablation-EDU+Synth-1.5B (46B) |     **0.344** | **0.579** |        0.75 |  **0.46** | **0.39** |
-| Gigaverbo-v2-ablation-NonEDU-1.5B (46B)    |         0.256 |     0.565 |        0.65 |     0.383 |    0.352 |
-| Gigaverbo-v2-ablation-EDU-1.5B (46B)       |         0.328 |     0.579 |    **0.82** |     0.449 |    0.377 |
-| Gigaverbo-v2-ablation-Synth-1.5B (46B)     |         0.326 |     0.561 |        0.72 |     0.439 |    0.339 |
-| Tucano-1b1 (52B)                           |         0.284 |     0.464 |        0.64 |     0.401 |    0.257 |
-| Tucano-2b4 (52B)                           |         0.274 |     0.456 |        0.71 |     0.412 |    0.248 |
-| Tucano-1b1 (31B)                           |         0.286 |     0.447 |        0.63 |      0.38 |    0.235 |
-| Tucano-2b4 (31B)                           |         0.275 |     0.462 |        0.61 |     0.394 |    0.274 |
-
-Below are detailed plots for each benchmark, as well as a consolidated comparison table including other relevant models trained on Portuguese data.
-
-- **Note:** We selected these specific benchmarks (ARC Challenge, Calame, Global PIQA, HellaSwag, and Lambada) because they provide a good signal, eve at early training stages, of how well the models are learning to reason, understand context, and generate coherent text in Portuguese. Unlike oter benchmarks (e.g., ENEM, OAB, MMLU) that require more extensive training to see meaningful differences, these benchmarks are sensitive enough to capture the impact of our educational data mixture even at the 1.5B parameter scale and with a relatively modest training budget (~46B tokens).
-
-<details>
-<summary><b>ARC Challenge ⭐</b></summary>
-
-![](./arc.png)
-
-Arc is perhaps where we see the most significant difference between data mixtures. The EDU+Synth model achieves a score (34%) that substantially outperforms the NonEDU mixture (25%), and also surpasses both the EDU-only (32%) and Synth-only (32%) mixtures. This suggests that the combination of high-quality educational web data and the synthetic portion of GigaVerbo-v2 provides complementary benefits that enhance the model's reasoning capabilities in QA tasks.
-
-</details>
-
-<details>
-<summary><b>Calame ⭐</b></summary>
-
-![](./calame.png)
-
-For Calame, much like Lambada, is where we see a less pronounced advantage coming from the EDU portion of GigaVerbo-v2. We hypothesize that this is due to the nature of these benchmarks (i.e., predict the last word of a passage), which may not benefit as much from educational content as other benchmarks focused on reasoning or commonsense understanding. Nevertheless, the EDU+Synth model still outperforms the NonEDU model (57.9% vs 56.5%), indicating that educational data does contribute positively even in this context.
-
-</details>
-
-<details>
-<summary><b>Global PIQA ⭐</b></summary>
-
-![](./global_piqa.png)
-
-On Global PIQA, we see that the adition of synthetic data to the educational portion of GigaVerbo-v2 yields a decrease in performance (75% vs 82%). However, both models still significantly outperform the NonEDU model (65%). Given that the synthetic data was not specifically tailored for physical commonsense reasoning tasks (or cultal knowledge tasks in general), this result is not entirely surprising. It does, however, highlight some shortcomings of our synthetic dataset that we aim to address in future iterations.
-
-</details>
-
-<details>
-<summary><b>HellaSwag ⭐</b></summary>
-
-![](./hellaswag.png)
-
-Lambada is the only benchmark where the NonEdua mixture outperforms the Synth-only mixture (35.2% vs 33.9%). However, the Edu+Synth mixture still achieves the best performance (39%), followed closely by the Edu-only mixture (37.7%). Again, we hypothesize that this is due to the nature of the benchmark, which focuses on predicting the last word of a passage, a task that may not benefit as much from the repetitive patterns found in our synthetic data.
-
-</details>
-
-<details>
-<summary><b>Lambada ⭐</b></summary>
-
-![](./lambada.png)
-
-Like in the case of Calame, the GigaVerbo-v2-Synth model shows a clear advantage over the Non-EDU model on Lambada (39% vs 35.2%), albeit by a smaller margin.
-
-</details>
-
-<details>
-<summary><b>More Comparisons 📊</b></summary>
-
-|                                            | ARC Challenge |    Calame | Global PIQA | HellaSwag |   Lambada |
-| :----------------------------------------- | ------------: | --------: | ----------: | --------: | --------: |
-| Gigaverbo-v2-ablation-EDU+Synth-1.5B (46B) |     **0.344** |     0.579 |        0.75 |      0.46 |      0.39 |
-| Gigaverbo-v2-ablation-NonEDU-1.5B (46B)    |         0.256 |     0.565 |        0.65 |     0.383 |     0.352 |
-| Gigaverbo-v2-ablation-EDU-1.5B (46B)       |         0.328 |     0.579 |    **0.82** |     0.449 |     0.377 |
-| Gigaverbo-v2-ablation-Synth-1.5B (46B)     |         0.326 |     0.561 |        0.72 |     0.439 |     0.339 |
-| Carvalho_pt-gl-1.3B (26B + 5B)             |          0.27 |     0.534 |        0.63 |     0.385 |     0.336 |
-| Curio-1.1b (1T + 150B)                     |         0.304 |     0.592 |        0.75 | **0.495** |     0.467 |
-| Curio-1.1b (1T + 100B)                     |         0.309 | **0.599** |        0.74 |     0.489 |     0.468 |
-| Curio-1.1b (1T + 50B)                      |         0.294 |     0.589 |        0.74 |      0.48 | **0.469** |
-| Curio-edu-1b1 (1T + ?)                     |         0.322 |     0.549 |        0.69 |     0.463 |     0.429 |
-| GlorIA-1.3B (35B)                          |         0.264 |     0.547 |        0.64 |     0.364 |     0.367 |
-| Llama-3.2-1B (9T)                          |         0.317 |       0.5 |        0.55 |     0.453 |     0.456 |
-| Tucano-1b1 (250B)                          |         0.301 |     0.489 |        0.68 |     0.441 |     0.284 |
-| Tucano-1b1 (31B)                           |         0.286 |     0.447 |        0.63 |      0.38 |     0.235 |
-| Tucano-1b1 (52B)                           |         0.284 |     0.464 |        0.64 |     0.401 |     0.257 |
-| Tucano-2b4 (31B)                           |         0.275 |     0.462 |        0.61 |     0.394 |     0.274 |
-| Tucano-2b4 (500B)                          |         0.304 |     0.503 |        0.73 |     0.488 |     0.324 |
-| Tucano-2b4 (52B)                           |         0.274 |     0.456 |        0.71 |     0.412 |     0.248 |
-
-Even with a much smaller compute budget (~46B tokens) compared to other models (e.g., Curio seen 1 trillion tokens of pretraining and 150 billion tokens of continual pretraining, Llama-3.2-1B seen 9 trillion tokens of pretraining), the models trained on the Educational (and synthetic) portions of GigaVerbo-v2 are able to achieve competitive performance across all evaluated benchmarks. Moreover, the results from our ablation study clearly indicate a signifcant improvment in terms of GigaVerbo-v1, given that the models trained on GigaVerbo-v2 consistently outperform the first iteration of the Tucano models across all benchmarks.
-
-</details>
+The table below compares our ablation models with checkpoints from the first [Tucano series](https://huggingface.co/TucanoBR). Tucano models are a natural point of comparison because they were trained on Portuguese data of a similar nature and provide multiple checkpoints across different stages of training. To ensure a fair comparison, we select Tucano checkpoints that are closest to our ablation models in terms of both the number of training tokens seen (31B and 52B vs. 46B) and model size (1.1B and 2.4B parameters). We also include additional models for which reliable information on training data volume and model size is available and whose sizes are comparable to our ablation models. Performance is summarized using the NPM (Normalized Performance Metric), which provides a balanced aggregate view across tasks by normalizing each task’s score relative to its random baseline, thereby accounting for differences in task difficulty.
+
+|                                | NPM    | ARC Challenge | Calame | Global PIQA | HellaSwag | Lambada |
+| ------------------------------ | ------ | ------------- | ------ | ----------- | --------- | ------- |
+| GigaVerbo-v2 (EDU)             | 39.306 | 0.328         | 0.579  | 0.82        | 0.449     | 0.377   |
+| Curio-1.1b (1T + 150B)         | 39.156 | 0.304         | 0.592  | 0.75        | 0.495     | 0.467   |
+| Curio-1.1b (1T + 100B)         | 38.88  | 0.309         | 0.599  | 0.74        | 0.489     | 0.468   |
+| Curio-1.1b (1T + 50B)          | 38.057 | 0.294         | 0.589  | 0.74        | 0.48      | 0.469   |
+| GigaVerbo-v2 (EDU+Synth)       | 37.49  | 0.344         | 0.579  | 0.75        | 0.46      | 0.39    |
+| Curio-edu-1b1 (1T + 20B)       | 34.774 | 0.322         | 0.549  | 0.69        | 0.463     | 0.429   |
+| GigaVerbo-v2 (Synth)           | 33.864 | 0.326         | 0.561  | 0.72        | 0.439     | 0.339   |
+| Tucano-2b4 (500B)              | 33.551 | 0.304         | 0.503  | 0.73        | 0.488     | 0.324   |
+| Tucano-1b1 (250B)              | 29.124 | 0.301         | 0.489  | 0.68        | 0.441     | 0.284   |
+| Llama-3.2-1B (9T)              | 28.315 | 0.317         | 0.5    | 0.55        | 0.453     | 0.456   |
+| GigaVerbo-v2 (NonEDU)          | 28.049 | 0.256         | 0.565  | 0.65        | 0.383     | 0.352   |
+| Tucano-2b4 (52B)               | 27.433 | 0.274         | 0.456  | 0.71        | 0.412     | 0.248   |
+| GlorIA-1.3B (35B)              | 27.274 | 0.264         | 0.547  | 0.64        | 0.364     | 0.367   |
+| Carvalho_pt-gl-1.3B (26B + 5B) | 26.746 | 0.27          | 0.534  | 0.63        | 0.385     | 0.336   |
+| Tucano-1b1 (52B)               | 24.927 | 0.284         | 0.464  | 0.64        | 0.401     | 0.257   |
+
+### ⭐ GigaVerbo-v2 Ablations: The Impact of 46B Tokens of Educational & Synthetic Data ⭐
+
+All individual benchmark scores and their evolution across training time can be found in the [.plots](https://huggingface.co/Polygl0t/GigaVerbo-v2-ablation-Synth-1.5B/tree/main/.plots) folder.
+
+![GigaVerbo-v2 Ablation: Impact of Educational & Synthetic Data (46B tokens)](./.plots/gigaverbo_v2_ablation_comparison.png)
 
 ## Cite as 🤗