Instructions to use Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M

Use Docker

docker model run hf.co/Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M

LM Studio
Jan

vLLM

How to use Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M

Ollama
How to use Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300 with Ollama:
```
ollama run hf.co/Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300 with Docker Model Runner:
```
docker model run hf.co/Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M
```

Lemonade

How to use Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Arm/deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300:Q4_K_M

Run and chat with the model

lemonade run user.deepseek-r1-distill-qwen-1-5b-q4-k-m-llamacpp-vivo-x300-Q4_K_M

List all available models

lemonade list

Atomic Chat

aorabdel commited on 13 days ago

Commit

e919adb

verified ·

1 Parent(s): fbdfa78

Sync model repo (text/metadata)

Browse files

Files changed (3) hide show

README.md +11 -9
benchmarks/deepseek-r1-distill-qwen-1-5b-llamacpp-vivo-x300-fp32.yaml +10 -9
benchmarks/deepseek-r1-distill-qwen-1-5b-llamacpp-vivo-x300-int4.yaml +10 -9

README.md CHANGED Viewed

@@ -33,7 +33,7 @@ The model is reasoning-distilled: the embedded chat template opens the assistant
 | Model format | GGUF `.gguf` (llama.cpp) |
 | Target device class | Mobile CPU |
 | Reference device | vivo X300 (C1-Ultra, C1-Premium, C1-Pro; Android 16 / OriginOS 6) |
-| Primary performance result | 41.21 tokens/sec decode, 2655.50 ms prefill |
 | Accuracy result | MATH-500 80.0% |
 | Size / memory result | 1065.56 MB model size, 3.19x smaller than the f16 baseline |
 | Precision | Mixed: 4-bit weights (Q4_K default tensor type and token embeddings), 6-bit output weight (Q6_K); activations dynamically quantized to 8-bit at runtime |
@@ -68,25 +68,27 @@ Performance was measured on the reference configuration below. Results are inten
 | Field | Value |
 |---|---|
 | Device / platform | vivo X300 |
-| CPU / accelerator | C1-Ultra, C1-Premium, C1-Pro (8 cores), cpu |
 | OS | android — Android 16 / OriginOS 6 |
 | Runtime | llama.cpp |
 | Backend / delegate | cpu, ggml-cpu-repack |
 | Batch size | 1 |
 | Precision | Mixed: 4-bit weights (Q4_K default tensor type and token embeddings), 6-bit output weight (Q6_K); activations dynamically quantized to 8-bit at runtime; imatrix-weighted k-quant, per-group, asymmetric |
-| Runs | 5 measured |
 **Performance results**
 | Metric | Original / baseline | Arm-optimized | Improvement |
-|---|---:|---:|---:|
-| Decode throughput | 10.01 tokens/sec | 41.21 tokens/sec | 4.12x |
-| Prefill time | 19284.20 ms | 2655.50 ms | 7.26x faster |
-| p50 latency | 32073.58 ms | 5761.18 ms | 5.57x faster |
 | Model size | 3395.48 MB | 1065.56 MB | 3.19x smaller |
-| Peak memory | 3529.36 MB | 2012.06 MB | 1.75x less |
-Latency here is the arithmetic consequence of the two measured llama-bench throughputs at the benched shape (512-token prompt, 128 generated tokens), not a directly timed wall-clock figure. Peak memory is Android USS (private clean plus private dirty), not peak RSS.
 ## Accuracy

 | Model format | GGUF `.gguf` (llama.cpp) |
 | Target device class | Mobile CPU |
 | Reference device | vivo X300 (C1-Ultra, C1-Premium, C1-Pro; Android 16 / OriginOS 6) |
+| Primary performance result | 34.67 tokens/sec decode, 761.30 ms prefill |
 | Accuracy result | MATH-500 80.0% |
 | Size / memory result | 1065.56 MB model size, 3.19x smaller than the f16 baseline |
 | Precision | Mixed: 4-bit weights (Q4_K default tensor type and token embeddings), 6-bit output weight (Q6_K); activations dynamically quantized to 8-bit at runtime |
 | Field | Value |
 |---|---|
 | Device / platform | vivo X300 |
+| CPU / accelerator | C1-Ultra, C1-Premium, C1-Pro, cpu |
 | OS | android — Android 16 / OriginOS 6 |
 | Runtime | llama.cpp |
 | Backend / delegate | cpu, ggml-cpu-repack |
 | Batch size | 1 |
 | Precision | Mixed: 4-bit weights (Q4_K default tensor type and token embeddings), 6-bit output weight (Q6_K); activations dynamically quantized to 8-bit at runtime; imatrix-weighted k-quant, per-group, asymmetric |
+| Runs | 5 warm-up + 20 measured |
+**Measurement conditions.** Each measured run consumes a 128-token prompt and generates 128 tokens, using 4 CPU threads, 5 warm-up runs and 20 measured runs. Each run starts only once Android reports thermal status 0 (NONE), after a 30 s settle. The device is set to fixed performance mode, which is the official recommendation.
 **Performance results**
 | Metric | Original / baseline | Arm-optimized | Improvement |
+| --- | ---: | ---: | ---: |
+| Decode throughput | 9.84 tokens/sec | 34.67 tokens/sec | 3.52x |
+| Prefill time | 5515.52 ms | 761.30 ms | 7.24x faster |
+| p50 latency | 18531.18 ms | 4458.02 ms | 4.16x faster |
 | Model size | 3395.48 MB | 1065.56 MB | 3.19x smaller |
+| Peak memory | 3469.12 MB | 1979.49 MB | 1.75x less |
+Latency here is derived from the measured llama-bench figures at the benched shape (128-token prompt, 128 generated tokens), not a directly timed wall-clock figure. Peak memory is Android USS (private clean plus private dirty), not peak RSS.
 ## Accuracy

benchmarks/deepseek-r1-distill-qwen-1-5b-llamacpp-vivo-x300-fp32.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 version: 1.2.0
 report_type: llm-generative
 profile: Baseline
-created_at: '2026-08-11T03:16:44Z'
 context:
   target_ref: vivo-x300
   runtime:
@@ -16,20 +16,21 @@ context:
       - GGML_CPU_KLEIDIAI=ON
   dataset:
     name: llama-bench synthetic tokens
-    sample_count: 5
-    slice: -p 512 -n 128 -r 5 -t 4
   benchmark:
     batch_size: 1
-    prompt_length_tokens: 512
     generation_length_tokens: 128
     deployment_mode: chat
-    num_runs: 5
 performance:
   end_to_end_latency_ms:
-    p50: 32073.58
-  prefill_encode_time_ms: 19284.2
-  tokens_per_second: 10.008307
-  peak_memory_mb: 3529.35546875
 accuracy:
   benchmark_name: MATH-500
   accuracy_pct: 82.8

 version: 1.2.0
 report_type: llm-generative
 profile: Baseline
+created_at: '2026-09-04T20:59:14Z'
 context:
   target_ref: vivo-x300
   runtime:
       - GGML_CPU_KLEIDIAI=ON
   dataset:
     name: llama-bench synthetic tokens
+    sample_count: 20
+    slice: -p 128 -n 128 -r 20 -t 4
   benchmark:
     batch_size: 1
+    prompt_length_tokens: 128
     generation_length_tokens: 128
     deployment_mode: chat
+    num_runs: 20
+    warmup_runs: 5
 performance:
   end_to_end_latency_ms:
+    p50: 18531.182
+  prefill_encode_time_ms: 5515.516
+  tokens_per_second: 9.836
+  peak_memory_mb: 3469.125
 accuracy:
   benchmark_name: MATH-500
   accuracy_pct: 82.8

benchmarks/deepseek-r1-distill-qwen-1-5b-llamacpp-vivo-x300-int4.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 version: 1.2.0
 report_type: llm-generative
 profile: Arm-Optimized
-created_at: '2026-08-11T03:16:44Z'
 context:
   target_ref: vivo-x300
   runtime:
@@ -16,20 +16,21 @@ context:
       - GGML_CPU_KLEIDIAI=ON
   dataset:
     name: llama-bench synthetic tokens
-    sample_count: 5
-    slice: -p 512 -n 128 -r 5 -t 4
   benchmark:
     batch_size: 1
-    prompt_length_tokens: 512
     generation_length_tokens: 128
     deployment_mode: chat
-    num_runs: 5
 performance:
   end_to_end_latency_ms:
-    p50: 5761.18
-  prefill_encode_time_ms: 2655.5
-  tokens_per_second: 41.214719
-  peak_memory_mb: 2012.0625
 accuracy:
   benchmark_name: MATH-500
   accuracy_pct: 80.0

 version: 1.2.0
 report_type: llm-generative
 profile: Arm-Optimized
+created_at: '2026-09-04T20:59:14Z'
 context:
   target_ref: vivo-x300
   runtime:
       - GGML_CPU_KLEIDIAI=ON
   dataset:
     name: llama-bench synthetic tokens
+    sample_count: 20
+    slice: -p 128 -n 128 -r 20 -t 4
   benchmark:
     batch_size: 1
+    prompt_length_tokens: 128
     generation_length_tokens: 128
     deployment_mode: chat
+    num_runs: 20
+    warmup_runs: 5
 performance:
   end_to_end_latency_ms:
+    p50: 4458.016
+  prefill_encode_time_ms: 761.304
+  tokens_per_second: 34.671
+  peak_memory_mb: 1979.492
 accuracy:
   benchmark_name: MATH-500
   accuracy_pct: 80.0