README.md

metadata

language:
  - en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - causal-lm
  - base-model
  - transformers
  - safetensors
  - veyra
  - small-language-model
model-index:
  - name: Veyra2-Mango-15M-Base
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: SciCloze-900
          type: veyra-ai/SciCloze-900
        metrics:
          - name: Accuracy
            type: accuracy
            value: 36.78
            source:
              name: Local evaluation
              url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base
      - task:
          type: multiple-choice
          name: Multiple Choice
        dataset:
          name: SciQ
          type: sciq
        metrics:
          - name: Accuracy
            type: accuracy
            value: 65.4
          - name: Normalized Accuracy
            type: acc_norm
            value: 58.8
            source:
              name: Local lm-evaluation-harness
              url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base
      - task:
          type: multiple-choice
          name: Multiple Choice
        dataset:
          name: PIQA
          type: piqa
        metrics:
          - name: Normalized Accuracy
            type: acc_norm
            value: 58
            source:
              name: Local lm-evaluation-harness
              url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base
      - task:
          type: multiple-choice
          name: Multiple Choice
        dataset:
          name: ARC-Easy
          type: ai2_arc
          config: ARC-Easy
        metrics:
          - name: Normalized Accuracy
            type: acc_norm
            value: 37.16
            source:
              name: Local lm-evaluation-harness
              url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base
      - task:
          type: multiple-choice
          name: Multiple Choice
        dataset:
          name: ARC-Challenge
          type: ai2_arc
          config: ARC-Challenge
        metrics:
          - name: Normalized Accuracy
            type: acc_norm
            value: 22.87
            source:
              name: Local lm-evaluation-harness
              url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base
      - task:
          type: multiple-choice
          name: Multiple Choice
        dataset:
          name: HellaSwag
          type: hellaswag
        metrics:
          - name: Normalized Accuracy
            type: acc_norm
            value: 27.73
            source:
              name: Local lm-evaluation-harness
              url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base
      - task:
          type: multiple-choice
          name: Multiple Choice
        dataset:
          name: Winogrande
          type: winogrande
        metrics:
          - name: Accuracy
            type: accuracy
            value: 50.99
            source:
              name: Local lm-evaluation-harness
              url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base
      - task:
          type: multiple-choice
          name: Multiple Choice
        dataset:
          name: OpenBookQA
          type: openbookqa
        metrics:
          - name: Accuracy
            type: accuracy
            value: 14.2
          - name: Normalized Accuracy
            type: acc_norm
            value: 26.2
            source:
              name: Local lm-evaluation-harness
              url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base
      - task:
          type: question-answering
          name: Question Answering
        dataset:
          name: BoolQ
          type: boolq
        metrics:
          - name: Accuracy
            type: accuracy
            value: 54.68
            source:
              name: Local lm-evaluation-harness
              url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base
      - task:
          type: multiple-choice
          name: Multiple Choice
        dataset:
          name: ArithMark-2.0
          type: AxiomicLabs/ArithMark-2.0
          split: train
        metrics:
          - name: Accuracy
            type: accuracy
            value: 28.08
            source:
              name: Local evaluation
              url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base
      - task:
          type: multiple-choice
          name: Multiple Choice
        dataset:
          name: ArithMark-3.0
          type: AxiomicLabs/Arithmark-3.0
          split: train
        metrics:
          - name: Accuracy
            type: accuracy
            value: 34.9
            source:
              name: Local evaluation
              url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base

Veyra2-Mango-15M-Base

Veyra2-Mango-15M-Base is a 15.7M-parameter Llama-like causal language model trained from scratch on approximately 30B tokens. It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation.

Model Details

Property	Value
Parameters	15,735,168
Architecture	LlamaForCausalLM
Layers	8
Hidden size	384
Attention heads	6
KV heads	2
Head dim	64
Intermediate size	1024
Vocabulary size	8192
Context length used in training	2048
Activation	SwiGLU / SiLU
Normalization	RMSNorm
Attention	GQA
Positional encoding	RoPE
Weight tying	Tied input embeddings and LM head
Training tokens	Approximately 30B
Training precision	bfloat16
Optimizer	AdamW

Tokenizer

Special tokens:

• <|endoftext|>: 0
• <|im_start|>: 1
• <|im_end|>: 2
• <|pad|>: 3

Training Data

The model was trained on a 30B-token pretraining mixture.

Stage 1 18,000,000,000 tokens 180 shards

Mixture:

dclm_baseline: 50%
finephrase: 20%
cosmopedia_v2: 10%
finemath_4plus: 10%
ultrafineweb_multistyle: 5%
ultrafineweb_qa: 5%

Stage 1.5 4,000,000,000 tokens 40 shards

This stage linearly transitions from the Stage 1 mixture to the Stage 2 mixture.

Stage 2 8,000,000,000 tokens 80 shards

Mixture:

finephrase: 30%
dclm_baseline: 30%
cosmopedia_v2: 18%
finemath_4plus: 10%
ultrafineweb_multistyle: 5%
ultrafineweb_qa: 5%
ultrachat: 2%

Training Summary

• Final step: 14,306
• Tokens seen: 30,000,000,000
• Tokens per step: 2,097,022
• Sequence length: 2048
• Last train loss: 2.8336

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "veyra-ai/Veyra2-Mango-15M-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

prompt = "In the 19th century"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.6,
        top_p=0.9,
        repetition_penalty=1.1,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Notes on Generation

Veyra2-Mango-15M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant. Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant.

Intended Use

This model is intended for:

• small language model research
• continued pretraining
• benchmarking
• experimentation with compact causal LMs

Limitations

• Not instruction tuned
• Not RLHF tuned
• Not safe for factual or high-stakes use without additional validation
• Can hallucinate names, citations, species, references, and technical claims
• Open-ended text may drift off-topic
• Context length during training was 2048 tokens

Citation

If you use this model, please cite the model repository:

veyra-ai/Veyra2-Mango-15M-Base