Create README.md

README.md

@@ -0,0 +1,94 @@
+---
+base_model: HuggingFaceTB/SmolLM3-3B
+language:
+- en
+library_name: gguf
+tags:
+- quantization
+- llama.cpp
+- gguf
+- smollm
+- nim-kernels
+- 4-bit
+- 8-bit
+- consumer-hardware
+---
+
+# SmolLM3-3B — 4-bit GGUF with Custom Nim Kernels
+
+Quantized versions of [SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B) 
+running on consumer hardware under 6 GB VRAM, with custom matrix kernels written in Nim 
+for hot-path matrix-multiply operations.
+
+Full write-up: [Run LLMs on a 6 GB Laptop GPU & CPU — Medium](https://medium.com/@devbyankit/run-llms-on-a-6-gb-laptop-gpu-cpu-smollm-3-quantization-nim-kernels-6c9cbb233930)
+
+---
+
+## What's in this repo
+
+| File | Format | Size approx. | Use case |
+|------|--------|--------------|----------|
+| smolLM3-q4_k_m.gguf | Q4_K_M | ~1.9 GB | Best speed, GPU + CPU |
+| smolLM3-q8_0.gguf | Q8_0 | ~3.3 GB | Near FP16 quality, fits 6 GB |
+
+Memory footprint is reduced ~75% from the FP16 baseline.
+
+---
+
+## Performance (512-token prompt, averaged over 3 runs)
+
+| Configuration | Prompt Processing | Text Generation | VRAM / RAM |
+|--------------|-------------------|-----------------|------------|
+| 4-bit GPU | 11.25 tok/s | **14.60 tok/s** | ~5.5 GB VRAM |
+| 8-bit GPU | 2.57 tok/s | 12.65 tok/s | ~5.8 GB VRAM |
+| 4-bit CPU | 2.38 tok/s | 15.37 tok/s | ~3–4 GB RAM |
+| 8-bit CPU | 7.79 tok/s | 11.83 tok/s | ~3–4 GB RAM |
+
+Hardware used: NVIDIA GPU with 6 GB VRAM, Windows, CUDA 12.4.
+
+---
+
+## How to run
+
+```bash
+# 4-bit GPU
+llama-cli.exe -m smolLM3-q4_k_m.gguf -ngl 36 -n 256 --temp 0.7 \
+  --repeat-penalty 1.1 --color -sys "you are a helpful assistant"
+
+# 8-bit CPU
+llama-cli.exe -m smolLM3-q8_0.gguf -ngl 0 -n 256 --temp 0.7
+```
+
+Requires [llama.cpp](https://github.com/ggerganov/llama.cpp) 0.3.14+ built with CUDA 
+support for GPU runs.
+
+---
+
+## Custom Nim Kernels
+
+The repo ships two companion DLLs — `libsmolkernels_q4.dll` and `libsmolkernels_q8.dll` 
+— written in Nim, compiled with `-O3` and ORC memory management. They implement a 
+custom dequantization + matrix-multiply path (`mulQ4Mat` / `mulQ8Mat`) that is 
+dynamically loaded by a patched `main.cpp` at runtime based on the detected model type.
+llama-cli.exe detects "q4_k_m" in filename → loads libsmolkernels_q4.dll → mulQ4Mat()
+llama-cli.exe detects "q8_0" in filename   → loads libsmolkernels_q8.dll → mulQ8Mat()
+Full build instructions, kernel source code, and the main.cpp patch are in the 
+Medium article linked above.
+
+---
+
+## System requirements
+
+- NVIDIA GPU 4 GB+ VRAM (for GPU runs), or 8 GB+ RAM (for CPU runs)
+- CUDA Toolkit 12.0+
+- llama.cpp 0.3.x+
+- Windows (DLL-based kernel loader); Linux port straightforward with .so
+
+---
+
+## Related project
+
+This work is the research foundation for 
+[QBench CLI](https://github.com/AnkitTsj/qbench) — a C++ command-line tool that 
+automates LLM quantization workflows, model selection, and hardware compatibility 
+checks using the same quantization and kernel techniques developed here.