--- language: - en license: apache-2.0 library_name: transformers pipeline_tag: text-generation tags: - causal-lm - base-model - transformers - safetensors - veyra - small-language-model model-index: - name: Veyra2-Mango-15M-Base results: - task: type: text-generation name: Text Generation dataset: name: SciCloze-900 type: veyra-ai/SciCloze-900 metrics: - name: Accuracy type: accuracy value: 36.78 source: name: Local evaluation url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base - task: type: multiple-choice name: Multiple Choice dataset: name: SciQ type: sciq metrics: - name: Accuracy type: accuracy value: 65.40 - name: Normalized Accuracy type: acc_norm value: 58.80 source: name: Local lm-evaluation-harness url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base - task: type: multiple-choice name: Multiple Choice dataset: name: PIQA type: piqa metrics: - name: Normalized Accuracy type: acc_norm value: 58.00 source: name: Local lm-evaluation-harness url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base - task: type: multiple-choice name: Multiple Choice dataset: name: ARC-Easy type: ai2_arc config: ARC-Easy metrics: - name: Normalized Accuracy type: acc_norm value: 37.16 source: name: Local lm-evaluation-harness url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base - task: type: multiple-choice name: Multiple Choice dataset: name: ARC-Challenge type: ai2_arc config: ARC-Challenge metrics: - name: Normalized Accuracy type: acc_norm value: 22.87 source: name: Local lm-evaluation-harness url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base - task: type: multiple-choice name: Multiple Choice dataset: name: HellaSwag type: hellaswag metrics: - name: Normalized Accuracy type: acc_norm value: 27.73 source: name: Local lm-evaluation-harness url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base - task: type: multiple-choice name: Multiple Choice dataset: name: Winogrande type: winogrande metrics: - name: Accuracy type: accuracy value: 50.99 source: name: Local lm-evaluation-harness url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base - task: type: multiple-choice name: Multiple Choice dataset: name: OpenBookQA type: openbookqa metrics: - name: Accuracy type: accuracy value: 14.20 - name: Normalized Accuracy type: acc_norm value: 26.20 source: name: Local lm-evaluation-harness url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base - task: type: question-answering name: Question Answering dataset: name: BoolQ type: boolq metrics: - name: Accuracy type: accuracy value: 54.68 source: name: Local lm-evaluation-harness url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base - task: type: multiple-choice name: Multiple Choice dataset: name: ArithMark-2.0 type: AxiomicLabs/ArithMark-2.0 split: train metrics: - name: Accuracy type: accuracy value: 28.08 source: name: Local evaluation url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base - task: type: multiple-choice name: Multiple Choice dataset: name: ArithMark-3.0 type: AxiomicLabs/Arithmark-3.0 split: train metrics: - name: Accuracy type: accuracy value: 34.90 source: name: Local evaluation url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base --- ![Veyra Banner](https://cdn-uploads.huggingface.co/production/uploads/6857f2cfae68b377f17aff8c/MfZTOclKIpOM1RbUlnfcr.jpeg) # Veyra2-Mango-15M-Base Veyra2-Mango-15M-Base is a 15.7M-parameter Llama-like causal language model trained from scratch on approximately 30B tokens. It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation. ## Model Details | Property | Value | | :--- | :--- | | **Parameters** | 15,735,168 | | **Architecture** | LlamaForCausalLM | | **Layers** | 8 | | **Hidden size** | 384 | | **Attention heads** | 6 | | **KV heads** | 2 | | **Head dim** | 64 | | **Intermediate size** | 1024 | | **Vocabulary size** | 8192 | | **Context length used in training** | 2048 | | **Activation** | SwiGLU / SiLU | | **Normalization** | RMSNorm | | **Attention** | GQA | | **Positional encoding** | RoPE | | **Weight tying** | Tied input embeddings and LM head | | **Training tokens** | Approximately 30B | | **Training precision** | bfloat16 | | **Optimizer** | AdamW | ## Tokenizer Special tokens: - `<|endoftext|>`: 0 - `<|im_start|>`: 1 - `<|im_end|>`: 2 - `<|pad|>`: 3 ## Training Data The model was trained on a 30B-token pretraining mixture. Stage 1 18,000,000,000 tokens 180 shards Mixture: dclm_baseline: 50% finephrase: 20% cosmopedia_v2: 10% finemath_4plus: 10% ultrafineweb_multistyle: 5% ultrafineweb_qa: 5% Stage 1.5 4,000,000,000 tokens 40 shards This stage linearly transitions from the Stage 1 mixture to the Stage 2 mixture. Stage 2 8,000,000,000 tokens 80 shards Mixture: finephrase: 30% dclm_baseline: 30% cosmopedia_v2: 18% finemath_4plus: 10% ultrafineweb_multistyle: 5% ultrafineweb_qa: 5% ultrachat: 2% ## Training Summary - Final step: 14,306 - Tokens seen: 30,000,000,000 - Tokens per step: 2,097,022 - Sequence length: 2048 - Last train loss: 2.8336 ## Usage 
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "veyra-ai/Veyra2-Mango-15M-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

prompt = "In the 19th century"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.6,
        top_p=0.9,
        repetition_penalty=1.1,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))
## Notes on Generation Veyra2-Mango-15M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant. Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant. ## Intended Use This model is intended for: - small language model research - continued pretraining - benchmarking - experimentation with compact causal LMs ## Limitations - Not instruction tuned - Not RLHF tuned - Not safe for factual or high-stakes use without additional validation - Can hallucinate names, citations, species, references, and technical claims - Open-ended text may drift off-topic - Context length during training was 2048 tokens ## Citation If you use this model, please cite the model repository: `veyra-ai/Veyra2-Mango-15M-Base`