---
datasets:
- HuggingFaceTB/smollm-corpus
language:
- en
pipeline_tag: text-generation
tags:
- SLM
- ROSA
license: mit
---

# SRLM-1M

A **S**mall **R**OSA based **L**anguage **M**odel, of 900k parameters.

## Training inforation

Trained on smollm-corpus fineweb-edu-dedup subset, over 16M tokens.

## Evaluations

- Wikitext v2 byte_perplexity: 4.4553
- Arc_easy acc_norm: 28.66%

## How to run it

To run this model, you have to:
1. Download the files
2. Compile rosa.pyx using `python setup.py build_ext --inplace`
3. Load the model with the correct configuration
4. Call the model