Instructions to use purplesquirrelnetworks/purple-squirrel-r1-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use purplesquirrelnetworks/purple-squirrel-r1-gguf with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="purplesquirrelnetworks/purple-squirrel-r1-gguf",
	filename="purple-squirrel-r1-Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use purplesquirrelnetworks/purple-squirrel-r1-gguf with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M

Use Docker

docker model run hf.co/purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M

LM Studio
Jan

vLLM

How to use purplesquirrelnetworks/purple-squirrel-r1-gguf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "purplesquirrelnetworks/purple-squirrel-r1-gguf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "purplesquirrelnetworks/purple-squirrel-r1-gguf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M

Ollama
How to use purplesquirrelnetworks/purple-squirrel-r1-gguf with Ollama:
```
ollama run hf.co/purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M
```

Unsloth Studio

How to use purplesquirrelnetworks/purple-squirrel-r1-gguf with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for purplesquirrelnetworks/purple-squirrel-r1-gguf to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for purplesquirrelnetworks/purple-squirrel-r1-gguf to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for purplesquirrelnetworks/purple-squirrel-r1-gguf to start chatting

Atomic Chat new
Docker Model Runner
How to use purplesquirrelnetworks/purple-squirrel-r1-gguf with Docker Model Runner:
```
docker model run hf.co/purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M
```

Lemonade

How to use purplesquirrelnetworks/purple-squirrel-r1-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull purplesquirrelnetworks/purple-squirrel-r1-gguf:Q4_K_M

Run and chat with the model

lemonade run user.purple-squirrel-r1-gguf-Q4_K_M

List all available models

lemonade list

Purple Squirrel R1 (GGUF)

GGUF quantized versions of Purple Squirrel R1 for local inference via llama.cpp, Ollama, or LM Studio.

Available Quantizations

File	Quant	Size	Quality	Speed	Use Case
`purple-squirrel-r1-f16.gguf`	F16	15 GB	Best	Slowest	Reference, re-quantization
`purple-squirrel-r1-Q8_0.gguf`	Q8_0	~8 GB	Excellent	Fast	High-quality local inference
`purple-squirrel-r1-Q5_K_M.gguf`	Q5_K_M	~5.5 GB	Great	Faster	Balanced quality/speed
`purple-squirrel-r1-Q4_K_M.gguf`	Q4_K_M	4.6 GB	Good	Fastest	Memory-constrained devices

Model Details

Property	Value
Base Model	DeepSeek-R1-Distill-Llama-8B
Parameters	8B
Architecture	Llama
Context Length	4096 tokens
Specialization	AIDP platform ops, video analysis, blockchain

Usage

Ollama (Recommended)

A ready-to-use Modelfile is included in this repo.

# Download the Modelfile and a GGUF
huggingface-cli download purplesquirrelnetworks/purple-squirrel-r1-gguf \
  Modelfile purple-squirrel-r1-Q5_K_M.gguf --local-dir .

# Create and run
ollama create purple-squirrel-r1 -f Modelfile
ollama run purple-squirrel-r1

To use a different quantization, edit the FROM line in the Modelfile.

llama.cpp

./llama-cli -m purple-squirrel-r1-Q4_K_M.gguf \
  -p "Explain how distributed GPU inference reduces costs" \
  -n 500 -c 4096

LM Studio

Download any GGUF file from this repo
Open LM Studio → Load Model → Select the file
Start chatting

Choosing a Quantization

16GB+ RAM: Use Q8_0 for best quality
8-16GB RAM: Use Q5_K_M for great balance
<8GB RAM: Use Q4_K_M for fastest inference
Re-quantizing: Start from F16

Related Resources

Resource	Link
Full Model (safetensors)	purple-squirrel-r1
Multichain Edition (MLX)	purple-squirrel-r1-multichain
LoRA Adapters	purple-squirrel-r1-multichain-lora
Research Paper	AIDP Neural Cloud
Research Paper	AIDP Video Forge
Coldstar Whitepaper	coldstar-whitepaper
Training Data	multichain-day-training
Full Collection	Purple Squirrel AI

Citation

@misc{purplesquirrel-r1-gguf-2025,
  title={Purple Squirrel R1 GGUF Quantizations},
  author={Karsten, Matthew},
  year={2025},
  publisher={Purple Squirrel Media},
  howpublished={\url{https://huggingface.co/purplesquirrelnetworks/purple-squirrel-r1-gguf}},
  note={GGUF quantized DeepSeek-R1-Distill-Llama-8B for local inference}
}

License

MIT

Built by Purple Squirrel Media | GitHub

Downloads last month: 44

GGUF

Model size

8B params

Architecture

llama

Hardware compatibility

4-bit

5-bit

8-bit

16-bit

Model tree for purplesquirrelnetworks/purple-squirrel-r1-gguf

Base model

deepseek-ai/DeepSeek-R1-Distill-Llama-8B

Quantized

(191)

this model

Space using purplesquirrelnetworks/purple-squirrel-r1-gguf 1

Collection including purplesquirrelnetworks/purple-squirrel-r1-gguf

Purple Squirrel AI — Models, Papers & Data

Collection

9 models, 3 papers, 3 datasets. Distributed AI, GPU video, multichain DeFi, Solana wallets. GGUF quants + LoRA + 1.3K training pairs. MIT. • 12 items • Updated Mar 2