| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - causal-lm |
| - base-model |
| - transformers |
| - safetensors |
| - veyra |
| - small-language-model |
| model-index: |
| - name: Veyra2-Mango-15M-Base |
| results: |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| name: SciCloze-900 |
| type: veyra-ai/SciCloze-900 |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 36.78 |
| source: |
| name: Local evaluation |
| url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base |
|
|
| - task: |
| type: multiple-choice |
| name: Multiple Choice |
| dataset: |
| name: SciQ |
| type: sciq |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 65.40 |
| - name: Normalized Accuracy |
| type: acc_norm |
| value: 58.80 |
| source: |
| name: Local lm-evaluation-harness |
| url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base |
|
|
| - task: |
| type: multiple-choice |
| name: Multiple Choice |
| dataset: |
| name: PIQA |
| type: piqa |
| metrics: |
| - name: Normalized Accuracy |
| type: acc_norm |
| value: 58.00 |
| source: |
| name: Local lm-evaluation-harness |
| url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base |
|
|
| - task: |
| type: multiple-choice |
| name: Multiple Choice |
| dataset: |
| name: ARC-Easy |
| type: ai2_arc |
| config: ARC-Easy |
| metrics: |
| - name: Normalized Accuracy |
| type: acc_norm |
| value: 37.16 |
| source: |
| name: Local lm-evaluation-harness |
| url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base |
|
|
| - task: |
| type: multiple-choice |
| name: Multiple Choice |
| dataset: |
| name: ARC-Challenge |
| type: ai2_arc |
| config: ARC-Challenge |
| metrics: |
| - name: Normalized Accuracy |
| type: acc_norm |
| value: 22.87 |
| source: |
| name: Local lm-evaluation-harness |
| url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base |
|
|
| - task: |
| type: multiple-choice |
| name: Multiple Choice |
| dataset: |
| name: HellaSwag |
| type: hellaswag |
| metrics: |
| - name: Normalized Accuracy |
| type: acc_norm |
| value: 27.73 |
| source: |
| name: Local lm-evaluation-harness |
| url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base |
|
|
| - task: |
| type: multiple-choice |
| name: Multiple Choice |
| dataset: |
| name: Winogrande |
| type: winogrande |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 50.99 |
| source: |
| name: Local lm-evaluation-harness |
| url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base |
|
|
| - task: |
| type: multiple-choice |
| name: Multiple Choice |
| dataset: |
| name: OpenBookQA |
| type: openbookqa |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 14.20 |
| - name: Normalized Accuracy |
| type: acc_norm |
| value: 26.20 |
| source: |
| name: Local lm-evaluation-harness |
| url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base |
|
|
| - task: |
| type: question-answering |
| name: Question Answering |
| dataset: |
| name: BoolQ |
| type: boolq |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 54.68 |
| source: |
| name: Local lm-evaluation-harness |
| url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base |
|
|
| - task: |
| type: multiple-choice |
| name: Multiple Choice |
| dataset: |
| name: ArithMark-2.0 |
| type: AxiomicLabs/ArithMark-2.0 |
| split: train |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 28.08 |
| source: |
| name: Local evaluation |
| url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base |
|
|
| - task: |
| type: multiple-choice |
| name: Multiple Choice |
| dataset: |
| name: ArithMark-3.0 |
| type: AxiomicLabs/Arithmark-3.0 |
| split: train |
| metrics: |
| - name: Accuracy |
| type: accuracy |
| value: 34.90 |
| source: |
| name: Local evaluation |
| url: https://huggingface.co/veyra-ai/Veyra2-Mango-15M-Base |
|
|
| --- |
| |
|  |
|
|
| # Veyra2-Mango-15M-Base |
|
|
| Veyra2-Mango-15M-Base is a 15.7M-parameter Llama-like causal language model trained from scratch on approximately 30B tokens. It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| | :--- | :--- | |
| | **Parameters** | 15,735,168 | |
| | **Architecture** | LlamaForCausalLM | |
| | **Layers** | 8 | |
| | **Hidden size** | 384 | |
| | **Attention heads** | 6 | |
| | **KV heads** | 2 | |
| | **Head dim** | 64 | |
| | **Intermediate size** | 1024 | |
| | **Vocabulary size** | 8192 | |
| | **Context length used in training** | 2048 | |
| | **Activation** | SwiGLU / SiLU | |
| | **Normalization** | RMSNorm | |
| | **Attention** | GQA | |
| | **Positional encoding** | RoPE | |
| | **Weight tying** | Tied input embeddings and LM head | |
| | **Training tokens** | Approximately 30B | |
| | **Training precision** | bfloat16 | |
| | **Optimizer** | AdamW | |
|
|
| ## Tokenizer |
|
|
| Special tokens: |
|
|
| - `<|endoftext|>`: 0 |
| - `<|im_start|>`: 1 |
| - `<|im_end|>`: 2 |
| - `<|pad|>`: 3 |
|
|
| ## Training Data |
|
|
| The model was trained on a 30B-token pretraining mixture. |
|
|
| Stage 1 18,000,000,000 tokens 180 shards |
|
|
| Mixture: |
|
|
| dclm_baseline: 50% |
| finephrase: 20% |
| cosmopedia_v2: 10% |
| finemath_4plus: 10% |
| ultrafineweb_multistyle: 5% |
| ultrafineweb_qa: 5% |
| |
| Stage 1.5 4,000,000,000 tokens 40 shards |
|
|
| This stage linearly transitions from the Stage 1 mixture to the Stage 2 mixture. |
|
|
| Stage 2 8,000,000,000 tokens 80 shards |
|
|
| Mixture: |
|
|
| finephrase: 30% |
| dclm_baseline: 30% |
| cosmopedia_v2: 18% |
| finemath_4plus: 10% |
| ultrafineweb_multistyle: 5% |
| ultrafineweb_qa: 5% |
| ultrachat: 2% |
| |
| ## Training Summary |
|
|
| - Final step: 14,306 |
| - Tokens seen: 30,000,000,000 |
| - Tokens per step: 2,097,022 |
| - Sequence length: 2048 |
| - Last train loss: 2.8336 |
|
|
| ## Usage |
|
|
| <pre><code>import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
| model_id = "veyra-ai/Veyra2-Mango-15M-Base" |
| |
| tokenizer = AutoTokenizer.from_pretrained( |
| model_id, |
| ) |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| torch_dtype=torch.float16, |
| device_map="auto", |
| ) |
| |
| prompt = "In the 19th century" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| with torch.no_grad(): |
| output = model.generate( |
| **inputs, |
| max_new_tokens=120, |
| do_sample=True, |
| temperature=0.6, |
| top_p=0.9, |
| repetition_penalty=1.1, |
| use_cache=True, |
| pad_token_id=tokenizer.pad_token_id, |
| eos_token_id=tokenizer.eos_token_id, |
| ) |
| |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| </code></pre> |
|
|
| ## Notes on Generation |
|
|
| Veyra2-Mango-15M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant. |
| Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant. |
|
|
| ## Intended Use |
|
|
| This model is intended for: |
|
|
| - small language model research |
| - continued pretraining |
| - benchmarking |
| - experimentation with compact causal LMs |
|
|
| ## Limitations |
|
|
| - Not instruction tuned |
| - Not RLHF tuned |
| - Not safe for factual or high-stakes use without additional validation |
| - Can hallucinate names, citations, species, references, and technical claims |
| - Open-ended text may drift off-topic |
| - Context length during training was 2048 tokens |
|
|
| ## Citation |
|
|
| If you use this model, please cite the model repository: |
|
|
| `veyra-ai/Veyra2-Mango-15M-Base` |
