---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- causal-lm
- base-model
- transformers
- safetensors
- veyra
- small-language-model
model-index:
- name: Veyra2-Blueberry-10M-Base
  results:
  - task:
      type: multiple-choice
      name: Multiple Choice
    dataset:
      name: SciQ
      type: sciq
    metrics:
    - name: Accuracy
      type: accuracy
      value: 68.1
    - name: Normalized Accuracy
      type: acc_norm
      value: 56.9
      source:
        name: Local lm-evaluation-harness
  - task:
      type: multiple-choice
      name: Multiple Choice
    dataset:
      name: PIQA
      type: piqa
    metrics:
    - name: Normalized Accuracy
      type: acc_norm
      value: 55.44
      source:
        name: Local lm-evaluation-harness
  - task:
      type: multiple-choice
      name: Multiple Choice
    dataset:
      name: ARC-Easy
      type: ai2_arc
      config: ARC-Easy
    metrics:
    - name: Normalized Accuracy
      type: acc_norm
      value: 35.44
      source:
        name: Local lm-evaluation-harness
  - task:
      type: multiple-choice
      name: Multiple Choice
    dataset:
      name: ARC-Challenge
      type: ai2_arc
      config: ARC-Challenge
    metrics:
    - name: Normalized Accuracy
      type: acc_norm
      value: 21.59
      source:
        name: Local lm-evaluation-harness
  - task:
      type: multiple-choice
      name: Multiple Choice
    dataset:
      name: HellaSwag
      type: hellaswag
    metrics:
    - name: Normalized Accuracy
      type: acc_norm
      value: 28.04
      source:
        name: Local lm-evaluation-harness
  - task:
      type: multiple-choice
      name: Multiple Choice
    dataset:
      name: Winogrande
      type: winogrande
    metrics:
    - name: Accuracy
      type: accuracy
      value: 50.12
      source:
        name: Local lm-evaluation-harness
  - task:
      type: multiple-choice
      name: Multiple Choice
    dataset:
      name: OpenBookQA
      type: openbookqa
    metrics:
    - name: Accuracy
      type: accuracy
      value: 15.0
    - name: Normalized Accuracy
      type: acc_norm
      value: 27.6
      source:
        name: Local lm-evaluation-harness
  - task:
      type: question-answering
      name: Question Answering
    dataset:
      name: BoolQ
      type: boolq
    metrics:
    - name: Accuracy
      type: accuracy
      value: 61.99
      source:
        name: Local lm-evaluation-harness
  - task:
      type: multiple-choice
      name: Multiple Choice
    dataset:
      name: ArithMark-3.0
      type: AxiomicLabs/Arithmark-3.0
      split: train
    metrics:
    - name: Accuracy
      type: accuracy
      value: 34.6
      source:
        name: Local evaluation
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceFW/finephrase
- mlfoundations/dclm-baseline-1.0
- HuggingFaceTB/finemath
---

![Veyra Banner](https://cdn-uploads.huggingface.co/production/uploads/6857f2cfae68b377f17aff8c/MfZTOclKIpOM1RbUlnfcr.jpeg)

# Veyra2-Blueberry-10M-Base

Veyra2-Blueberry-10M-Base is a 9.9M-parameter Llama-like causal language model trained from scratch on approximately 30B tokens. It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation.

## Model Details

| Property | Value |
| :--- | :--- |
| **Parameters** | 9,976,832 |
| **Architecture** | Qwen3ForCausalLM |
| **Layers** | 7 |
| **Hidden size** | 384 |
| **Attention heads** | 6 |
| **KV heads** | 2 |
| **Head dim** | 64 |
| **Intermediate size** | 700 |
| **Vocabulary size** | 8192 |
| **Context length used in training** | 2048 |
| **Activation** | SwiGLU / SiLU |
| **Normalization** | RMSNorm |
| **Attention** | GQA |
| **Positional encoding** | RoPE |
| **Weight tying** | Tied input embeddings and LM head |
| **Training tokens** | Approximately 30B |
| **Training precision** | bfloat16 |
| **Optimizer** | Muon |

## Tokenizer

Special tokens:

- `<|eos|>`: 0
- `<|bos|>`: 1
- `<|im_start|>`: 2
- `<|im_end|>`: 3
- `<|pad|>`: 4

## Training Data

The model was trained on a 30B-token pretraining mixture.

30,000,000,000 tokens 120 shards

Mixture:

    fineweb_edu: 40%
    finephrase: 20%
    dclm_baseline: 20%
    finemath_4plus: 20%

## Training Summary

- Final step: 14,305 
- Tokens seen: 30,000,000,000  
- Tokens per step: 2.097M
- Sequence length: 2048 
- Last train loss: 2.68 

## Usage

<pre><code>import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "veyra-ai/Veyra2-Blueberry-10M-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="cuda",
)

prompt = "In the 19th century"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=auto,
        temperature=0.6,
        top_p=0.9,
        repetition_penalty=1.1,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=False))
</code></pre>

## Notes on Generation

Veyra2-Blueberry-10M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant.
Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant.

## Intended Use

This model is intended for:

- small language model research
- continued pretraining
- benchmarking
- experimentation with compact causal LMs

## Limitations

- Not instruction tuned
- Not RLHF tuned
- Not safe for factual or high-stakes use without additional validation
- Can hallucinate names, citations, species, references, and technical claims
- Open-ended text may drift off-topic
- Context length during training was 2048 tokens

## Citation

If you use this model, please cite the model repository:

`veyra-ai/Veyra2-Blueberry-10M-Base`