--- license: apache-2.0 pipeline_tag: text-generation tags: - llamafile - huggingfaceTB datasets: - HuggingFaceTB/smol-smoltalk language: - en base_model: - HuggingFaceTB/SmolLM2-360M-Instruct --- # SmolLM2 - llamafile The portable/executable AI chatbot for you ![image/png](https://i.ibb.co/C3wh7xvk/Smol-LM2-360-M-portable.png) ## Model Summary This is a portable version of SmolLM2-360M-Instruct with Mozilla/llamafile packing method. It executable, as it is. - Model creator: [Hugging Face TB](https://huggingface.co/HuggingFaceTB) - Original model: [HuggingFaceTB/SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct) Fabio Matricardi has packaged the LLaMA model into executable weights that we call [llamafiles](https://github.com/Mozilla-Ocho/llamafile). This gives you the easiest fastest way to use the model on Linux, MacOS, Windows, FreeBSD, OpenBSD 7.3, and NetBSD systems you control on both AMD64 and ARM64. *Software Last Updated: 2025-03-31* *Llamafile Version: 0.9.2* *The executable will start with a context window set to 8k tokens* ### About SmolLM2 model family SmolLM2 is a family of compact language models available in three size: 135M, 360M, and 1.7B parameters. They are capable of solving a wide range of tasks while being lightweight enough to run on-device. More details in our paper: https://arxiv.org/abs/2502.02737 SmolLM2 demonstrates significant advances over its predecessor SmolLM1, particularly in instruction following, knowledge, reasoning. The 360M model was trained on 4 trillion tokens using a diverse dataset combination: FineWeb-Edu, DCLM, The Stack, along with new filtered datasets we curated and will release soon. We developed the instruct version through supervised fine-tuning (SFT) using a combination of public datasets and our own curated datasets. We then applied Direct Preference Optimization (DPO) using [UltraFeedback](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized). The instruct model additionally supports tasks such as text rewriting, summarization and function calling (for the 1.7B) thanks to datasets developed by [Argilla](https://huggingface.co/argilla) such as [Synth-APIGen-v0.1](https://huggingface.co/datasets/argilla/Synth-APIGen-v0.1). You can find the SFT dataset here: https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk and finetuning code in the [alignement handbook](https://github.com/huggingface/alignment-handbook/tree/main/recipes/smollm2) For more details refer to: https://github.com/huggingface/smollm. You will find pre-training, post-training, evaluation and local inference code. --- ## Quickstart To get started, you need both the LLaMA weights, and the llamafile software. Both of them are included in a single file, which can be downloaded and run as follows: #### For Mac/Linux users ``` wget https://huggingface.co/FM-1976/SmolLM2-360M-it-llamafile/resolve/main/SmolLM2-360M-portable.llamafile chmod +x SmolLM2-360M-portable.llamafile ./SmolLM2-360M-portable.llamafile ``` #### For Windows user: simply rename the extension from `SmolLM2-360M-portable.llamafile` to `SmolLM2-360M-portable.exe` The default mode of operation for these llamafiles is our new command line chatbot interface. At the same time a Web interface is available at `http://127.0.0.1:8080/` and also exposed to your internal Network. An OpenAI compatible API endpoint server will be listening at `http://localhost:8080/v1` ## Usage You can use triple quotes to ask questions on multiple lines. You can pass commands like `/stats` and `/context` to see runtime status information. You can change the system prompt by passing the `-p "new system prompt"` flag. You can press CTRL-C to interrupt the model. Finally CTRL-D may be used to exit. If you prefer to use a web GUI, then a `--server` mode is provided, that will open a tab with a chatbot and completion interface in your browser. For additional help on how it may be used, pass the `--help` flag. The server also has an OpenAI API compatible completions endpoint that can be accessed via Python using the `openai` pip package. ``` When you launch the executable the oepnAI API server is started automatically ``` An advanced CLI mode is provided that's useful for shell scripting. You can use it by passing the `--cli` flag. For additional help on how it may be used, pass the `--help` flag. ### details of the executsble packaging I used the following `.args` file ```bash -m SmolLM2-360M-Instruct.Q8_0.gguf --host 0.0.0.0 -c 8192 ... ``` You need to download the official binaries from Mozilla/llamafile to do it yourself: ``` llamafile-0.9.2.exe zipalign-0.9.2.exe ``` Additionally you need the weights quiantized of the model. I used the [Bartowski repository](https://huggingface.co/bartowski/SmolLM2-360M-Instruct-GGUF) ones: - file name: SmolLM2-360M-Instruct-Q8_0.gguf - Link: [https://huggingface.co/bartowski/SmolLM2-360M-Instruct-GGUF/resolve/main/SmolLM2-360M-Instruct-Q8_0.gguf](https://huggingface.co/bartowski/SmolLM2-360M-Instruct-GGUF/resolve/main/SmolLM2-360M-Instruct-Q8_0.gguf?download=true) ### Prompt format ``` <|im_start|>system {system_prompt}<|im_end|> <|im_start|>user {prompt}<|im_end|> <|im_start|>assistant ``` ## Quickstart with python and openAI API endpoint Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents. ```python # Chat with an intelligent assistant in your terminal from openai import OpenAI import sys # Point to the local server client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed") history = [ {"role": "system", "content": "You are SMOLLM2-PORTABLE, an intelligent assistant. You always provide well-reasoned answers that are both correct and helpful. Always reply in the language of the instructions."}, {"role": "user", "content": "Hello, introduce yourself to someone opening this program for the first time. Be concise."}, ] print("\033[92;1m") while True: userinput = "" completion = client.chat.completions.create( model="local-model", # this field is currently unused messages=history, temperature=0.3, frequency_penalty = 1.4, max_tokens = 600, stream=True, ) new_message = {"role": "assistant", "content": ""} for chunk in completion: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) new_message["content"] += chunk.choices[0].delta.content history.append(new_message) print("\033[1;30m") #dark grey print("Enter your text (end input with Ctrl+D on Unix or Ctrl+Z on Windows) - type quit! to exit the chatroom:") print("\033[91;1m") #red lines = sys.stdin.readlines() for line in lines: userinput += line + "\n" if "quit!" in lines[0].lower(): print("\033[0mBYE BYE!") break history = [ {"role": "system", "content": "You are an intelligent assistant. You always provide well-reasoned answers that are both correct and helpful."}, ] history.append({"role": "user", "content": userinput}) print("\033[92;1m") ``` ## Context Window This model has a max context window size of 8k tokens. By default, a context window size of 8192 tokens is used. You can ask llamafile to use the maximum context size by passing the `-c 0` flag. That's big enough for a small book. If you want to be able to have a conversation with your book, you can use the `-f book.txt` flag. ## Evaluation In this section, we report the evaluation results of SmolLM2. All evaluations are zero-shot unless stated otherwise, and we use [lighteval](https://github.com/huggingface/lighteval) to run them. ## Base Pre-Trained Model | Metrics | SmolLM2-360M | Qwen2.5-0.5B | SmolLM-360M | |:-------------------|:------------:|:------------:|:------------:| | HellaSwag | **54.5** | 51.2 | 51.8 | | ARC (Average) | **53.0** | 45.4 | 50.1 | | PIQA | **71.7** | 69.9 | 71.6 | | MMLU (cloze) | **35.8** | 33.7 | 34.4 | | CommonsenseQA | **38.0** | 31.6 | 35.3 | | TriviaQA | **16.9** | 4.3 | 9.1 | | Winogrande | 52.5 | **54.1** | 52.8 | | OpenBookQA | **37.4** | **37.4** | 37.2 | | GSM8K (5-shot) | 3.2 | **33.4** | 1.6 | ## Instruction Model | Metric | SmolLM2-360M-Instruct | Qwen2.5-0.5B-Instruct | SmolLM-360M-Instruct | |:-----------------------------|:---------------------:|:---------------------:|:---------------------:| | IFEval (Average prompt/inst) | **41.0** | 31.6 | 19.8 | | MT-Bench | 3.66 | **4.16** | 3.37 | | HellaSwag | **52.1** | 48.0 | 47.9 | | ARC (Average) | **43.7** | 37.3 | 38.8 | | PIQA | **70.8** | 67.2 | 69.4 | | MMLU (cloze) | **32.8** | 31.7 | 30.6 | | BBH (3-shot) | 27.3 | **30.7** | 24.4 | | GSM8K (5-shot) | 7.43 | **26.8** | 1.36 | ## Limitations SmolLM2 models primarily understand and generate content in English. They can produce text on a variety of topics, but the generated content may not always be factually accurate, logically consistent, or free from biases present in the training data. These models should be used as assistive tools rather than definitive sources of information. Users should always verify important information and critically evaluate any generated content. ## Training ### Model - **Architecture:** Transformer decoder - **Pretraining tokens:** 4T - **Precision:** bfloat16 ### Hardware - **GPUs:** 64 H100 ### Software - **Training Framework:** [nanotron](https://github.com/huggingface/nanotron/tree/main) ## License [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) ## Citation ```bash @misc{allal2025smollm2smolgoesbig, title={SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model}, author={Loubna Ben Allal and Anton Lozhkov and Elie Bakouch and Gabriel Martín Blázquez and Guilherme Penedo and Lewis Tunstall and Andrés Marafioti and Hynek Kydlíček and Agustín Piqueres Lajarín and Vaibhav Srivastav and Joshua Lochner and Caleb Fahlgren and Xuan-Son Nguyen and Clémentine Fourrier and Ben Burtenshaw and Hugo Larcher and Haojun Zhao and Cyril Zakka and Mathieu Morlon and Colin Raffel and Leandro von Werra and Thomas Wolf}, year={2025}, eprint={2502.02737}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2502.02737}, } ```