---
license: apache-2.0
language:
  - pt
  - en
pipeline_tag: text-generation
tags:
  - qwen
  - lora
  - peft
  - legal
  - portuguese
  - instruct
  - brazilian-portuguese
  - lm-evaluation-harness
  - oab
  - enem
base_model: unsloth/Qwen3.5-9B-Base
widget:
  - text: "Explique o princípio da presunção de inocência no direito brasileiro."
    example_title: "Legal Reasoning"
---

# Athenas-Symbiote-9B

<div align="center">
  <img src="image.png" alt="Athenas-Symbiote Logo" width="300"/>
</div>

## Model Details

**Athenas-Symbiote-9B** is a large language model (LLM) fine-tuned for advanced reasoning, complex text interpretation, and legal/academic domain expertise, primarily optimized for Brazilian Portuguese (PT-BR).

Built upon the `Qwen3.5-9B` architecture, this model leverages Parameter-Efficient Fine-Tuning (PEFT) to adapt its robust multilingual baseline specifically to the nuances, terminology, and logical structures required by the Brazilian legal and educational systems.

📄 **Technical Report**: [Available on Zenodo](https://zenodo.org/records/21418168) — full methodology, evaluation protocol, and per-edition analysis.

### Model Description

- **Developed by:** Bruno Kodjaoglanian Cardoso Tulux (Independent AI Researcher)
- **Model Type:** Causal Language Model (Transformer)
- **Base Model:** `unsloth/Qwen3.5-9B-Base`
- **Parameter Count:** 9 Billion
- **Context Length:** 32,768 tokens
- **Language(s) (NLP):** Portuguese (Primary), English
- **License:** Apache 2.0
- **Finetuning Approach:** LoRA (Low-Rank Adaptation)

---

## Evaluation & Benchmarks

The model was comprehensively evaluated using the `lm-evaluation-harness` framework. Inference was conducted on high-performance infrastructure (1x NVIDIA H100 80GB) utilizing native `bfloat16` precision to ensure metric fidelity and prevent quantization degradation.

*[Insert your consolidated performance bar/radar chart here]*
### National Benchmarks (Brazil)
Evaluations targeting the Brazilian cultural, legal, and educational framework.

| Benchmark | Metric | Score (Zero/Few-Shot) |
| :--- | :--- | :---: |
| **OAB Exams** (Brazilian Bar Association) | Accuracy | **63.23%** (3-shot) |
| **ENEM Challenge** (National High School Exam) | Accuracy | **80.90%** (3-shot) |

> **Benchmark Datasets:**
> - OAB Exams: [`maritaca-ai/oab-bench`](https://huggingface.co/datasets/maritaca-ai/oab-bench)
> - ENEM Challenge: [`maritaca-ai/enem`](https://huggingface.co/datasets/maritaca-ai/enem)

#### ENEM Challenge — Detailed Breakdown by Exam

| Exam | Accuracy | Std. Error |
| :--- | :---: | :---: |
| 2009 | 84.35% | ±1.95% |
| 2010 | 83.76% | ±1.97% |
| 2011 | 85.47% | ±1.88% |
| 2012 | 83.62% | ±1.98% |
| 2013 | 79.63% | ±2.24% |
| 2014 | 81.65% | ±2.15% |
| 2015 | 81.51% | ±2.05% |
| 2016 | 77.69% | ±2.17% |
| 2016 (2ª Aplicação) | 80.49% | ±2.07% |
| 2017 | 82.76% | ±2.03% |
| 2022 | 71.43% | ±2.26% |
| 2023 | 80.00% | ±2.00% |
| **Overall** | **80.90%** | **±0.60%** |

#### OAB Exams — Detailed Breakdown by Exam

| Exam | Accuracy | Std. Error |
| :--- | :---: | :---: |
| 2010-01 | 50.59% | ±3.13% |
| 2010-02 | 68.00% | ±2.69% |
| 2011-03 | 56.57% | ±2.89% |
| 2011-04 | 56.25% | ±3.21% |
| 2011-05 | 65.00% | ±3.08% |
| 2012-06 | 65.00% | ±3.08% |
| 2012-06a | 72.50% | ±2.88% |
| 2012-07 | 57.50% | ±3.20% |
| 2012-08 | 65.00% | ±3.07% |
| 2012-09 | 54.55% | ±3.26% |
| 2013-10 | 63.75% | ±3.11% |
| 2013-11 | 58.75% | ±3.20% |
| 2013-12 | 71.25% | ±2.92% |
| 2014-13 | 53.75% | ±3.22% |
| 2014-14 | 75.00% | ±2.80% |
| 2014-15 | 71.79% | ±2.94% |
| 2015-16 | 58.75% | ±3.19% |
| 2015-17 | 71.79% | ±2.95% |
| 2015-18 | 62.50% | ±3.11% |
| 2016-19 | 62.82% | ±3.17% |
| 2016-20 | 61.25% | ±3.14% |
| 2016-20a | 61.25% | ±3.12% |
| 2016-21 | 58.75% | ±3.18% |
| 2017-22 | 72.50% | ±2.88% |
| 2017-23 | 65.00% | ±3.08% |
| 2017-24 | 68.75% | ±3.00% |
| 2018-25 | 60.00% | ±3.17% |
| **Overall** | **63.23%** | **±0.60%** |

### Global Benchmarks (Translated / Multilingual)
Frontier evaluations assessing general intelligence and mathematical reasoning.

| Benchmark | Description | Score |
| :--- | :--- | :---: |
| **MMLU** | Massive Multitask Language Understanding (57 subjects) | — |
| **GSM8K** | Grade School Math (Chain-of-Thought Reasoning) | — |
| **HellaSwag** | Commonsense NLI | — |
| **ARC Challenge** | Advanced Scientific Reasoning | — |

> **Note:** Global benchmark evaluations are pending. Scores will be updated upon completion.

---

## How to Get Started with the Model

Use the code below to get started with the model. As this is a base model fine-tuned with LoRA, there is no built-in chat template. The ChatML format is recommended for optimal inference with instruction-style prompts.

### Requirements
```bash
pip install transformers accelerate bitsandbytes

```

### Python Inference

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Kodjaoglanian/Athenas-Symbiote-9B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

messages = [
    {"role": "system", "content": "Você é a Athenas, uma assistente virtual altamente inteligente e precisa."},
    {"role": "user", "content": "Explique a diferença entre dolo e culpa no código penal brasileiro."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs, 
    max_new_tokens=512, 
    temperature=0.3,
    do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

```

---

## Training Details

### Training Data

The model was fine-tuned on the [`ClassiCC-Corpus/ClassiCC-PT`](https://huggingface.co/datasets/ClassiCC-Corpus/ClassiCC-PT) dataset, consumed in **streaming mode** with a strict quality filter (`edu_score > 0.8`). This heuristic ensured the model ingested only high-rigor educational, literary, and academic content. The training corpus includes:

* Brazilian jurisprudence, legislation, and legal proceedings.
* Academic literature and technical documentation in Portuguese.
* Chain-of-Thought (CoT) reasoning structures.

**Data Pipeline:** Streaming → `edu_score > 0.8` filter → EOS concatenation → Sequence packing (`packing=True`)

### Training Pipeline

```mermaid
graph TD
    A["Load Qwen3.5-9B-Base<br/>(4-bit, Unsloth)"] --> B["Configure PEFT<br/>LoRA r=32, α=16, rsLoRA"]
    B --> C["Dataset: ClassiCC-PT<br/>(streaming, edu_score > 0.8)"]
    C --> D["SFTTrainer<br/>1,200 steps, LR 5e-5<br/>adamw_8bit, packing=True"]
    D --> E{"Training OK?"}
    E -- No --> F["Auto-retry<br/>(5x, resume checkpoint)"]
    F --> D
    E -- Yes --> G["Push to HF Hub<br/>(Safetensors + Tokenizer)"]
    G --> H["Evaluation<br/>lm-eval-harness<br/>H100, BF16, 3-shot"]

    style A fill:#1B3A5C,color:#fff
    style B fill:#1B3A5C,color:#fff
    style C fill:#1B3A5C,color:#fff
    style D fill:#1B3A5C,color:#fff
    style F fill:#C5A059,color:#fff
    style G fill:#C5A059,color:#fff
    style H fill:#C5A059,color:#fff
```

**Training:** Kaggle T4 (16GB) | **Evaluation:** Lightning AI H100 (80GB)

### Training Procedure

The fine-tuning process prioritized structural knowledge retention and computational efficiency.

* **Hardware:** 1x NVIDIA T4 (16GB VRAM) via Kaggle
* **Quantization:** 4-bit (NF4 via bitsandbytes)
* **Precision:** FP16 (T4 does not support BF16)
* **Framework:** Unsloth + TRL + PEFT + bitsandbytes
* **Optimization:** LoRA adapters applied to all linear projections
* **LoRA Rank:** 32
* **LoRA Alpha:** 16
* **Scaling Strategy:** rsLoRA (`use_rslora=True`)
* **LoRA Dropout:** 0.0
* **Target Modules:** `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
* **Learning Rate:** 5e-5
* **LR Scheduler:** Linear
* **Warmup Steps:** 50
* **Training Steps:** 1,200 (Dataset via Streaming)
* **Batch Size:** 1 (Effective Batch Size: 8 via Gradient Accumulation)
* **Optimizer:** adamw_8bit
* **Weight Decay:** 0.01
* **Packing:** True
* **Gradient Checkpointing:** Unsloth
* **Seed:** 3407
* **Max Sequence Length (Training):** 512 tokens

### Evaluation Hardware

* **Infrastructure:** Lightning AI platform
* **GPU:** 1x NVIDIA H100 (80GB VRAM)
* **Precision:** bfloat16 (native, no quantization)
* **Framework:** lm-evaluation-harness (EleutherAI)
* **Few-Shot:** 3-shot
* **Bootstrap Iterations:** 100,000
* **Execution:** Sequential evaluation to isolate memory overhead.

---

## Intended Use & Limitations

### Intended Use

* **Primary Use Case:** Research and application in Natural Language Processing (NLP) specifically for the Portuguese language.
* **Secondary Use Case:** Automation of complex text analysis, summarization, and logical reasoning within legal and academic contexts (assisted by human review).

### Out-of-Scope Use

The model is not designed to operate autonomously in high-stakes environments. It should not be used for:

* Automated legal counsel or medical diagnosis without human-in-the-loop validation.
* Generation of deceptive, malicious, or highly biased content.

### Bias, Risks, and Limitations

Like all statistical language models, **Athenas-Symbiote-9B** is susceptible to generating hallucinations—plausible-sounding but factually incorrect information. Furthermore, the model may inherit and amplify biases present in its pre-training data and fine-tuning corpus. Users must exercise critical judgment and implement rigorous verification protocols before utilizing the model's outputs in production environments.

---

## Citation

If you use this model in your research, please cite it as follows:

```bibtex
@misc{athenas_symbiote_9b,
  author       = {Bruno Kodjaoglanian Cardoso Tulux},
  title        = {Athenas-Symbiote-9B: A Parameter-Efficient Fine-Tuned LLM for Brazilian Portuguese Legal and Academic Domains},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.21418168},
  url          = {https://zenodo.org/records/21418168},
  note         = {Model weights: https://huggingface.co/Kodjaoglanian/Athenas-Symbiote-9B}
}
```