Instructions to use Mintche/Spark-X2.5-4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Mintche/Spark-X2.5-4B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Mintche/Spark-X2.5-4B-GGUF:Q6_K
# Run inference directly in the terminal:
llama cli -hf Mintche/Spark-X2.5-4B-GGUF:Q6_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Mintche/Spark-X2.5-4B-GGUF:Q6_K
# Run inference directly in the terminal:
llama cli -hf Mintche/Spark-X2.5-4B-GGUF:Q6_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Mintche/Spark-X2.5-4B-GGUF:Q6_K
# Run inference directly in the terminal:
./llama-cli -hf Mintche/Spark-X2.5-4B-GGUF:Q6_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Mintche/Spark-X2.5-4B-GGUF:Q6_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Mintche/Spark-X2.5-4B-GGUF:Q6_K

Use Docker

docker model run hf.co/Mintche/Spark-X2.5-4B-GGUF:Q6_K

LM Studio
Jan

vLLM

How to use Mintche/Spark-X2.5-4B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Mintche/Spark-X2.5-4B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Mintche/Spark-X2.5-4B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Mintche/Spark-X2.5-4B-GGUF:Q6_K

Ollama
How to use Mintche/Spark-X2.5-4B-GGUF with Ollama:
```
ollama run hf.co/Mintche/Spark-X2.5-4B-GGUF:Q6_K
```
Unsloth Desktop

How to use Mintche/Spark-X2.5-4B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Mintche/Spark-X2.5-4B-GGUF:Q6_K

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Mintche/Spark-X2.5-4B-GGUF:Q6_K"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Mintche/Spark-X2.5-4B-GGUF with Docker Model Runner:
```
docker model run hf.co/Mintche/Spark-X2.5-4B-GGUF:Q6_K
```

Lemonade

How to use Mintche/Spark-X2.5-4B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Mintche/Spark-X2.5-4B-GGUF:Q6_K

Run and chat with the model

lemonade run user.Spark-X2.5-4B-GGUF-Q6_K

List all available models

lemonade list

Hermes Agent

How to use Mintche/Spark-X2.5-4B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Mintche/Spark-X2.5-4B-GGUF:Q6_K

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Mintche/Spark-X2.5-4B-GGUF:Q6_K

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Mintche/Spark-X2.5-4B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Mintche/Spark-X2.5-4B-GGUF:Q6_K

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Mintche/Spark-X2.5-4B-GGUF:Q6_K" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Spark-X2.5-4B-GGUF / README.md

Mintche

Experimental v0.1: sanitized Q6_K, Q8_0 and reproducibility artifacts

d7e365f verified 3 days ago

preview code

Raw

History Blame Contribute Delete

10.7 kB

metadata

base_model:
  - XHToken/Spark-X2.5-4B
base_model_relation: quantized
library_name: gguf
pipeline_tag: text-generation
tags:
  - gguf
  - llama.cpp
  - imatrix
  - quantization
license: apache-2.0

Spark-X2.5-4B GGUF — experimental v0.1

GGUF quantizations of XHToken/Spark-X2.5-4B, pinned to source revision 1e4c2477f330d1f11304d0dc092a9dbf4075f470.

This is an experimental v0.1 release built with Spark2.5 support from llama.cpp commit fe158c6c4db8b0cb8d74c7cfe23401f7c21a45fe (build 10668). Use a llama.cpp build containing compatible spark2_5 support.

Files

File	Quantization	Size	SHA256	Recommendation
`Spark-X2.5-4B-Q6_K.gguf`	Q6_K with mixed 4K+16K iMatrix	3.147 GiB	`107F0E30E317B4FFFF3DD63CC30F1945C161E82B31EF7BC73187941D3EA71E42`	Default balance for an 8 GB GPU
`Spark-X2.5-4B-Q8_0.gguf`	Q8_0	4.075 GiB	`5C2C3C190E4337E1016B8593CA8E26E8B18C972200B107385D4EC61A25D9DEA2`	Highest fidelity of this release

The published Q6_K is the mixed-context iMatrix candidate. Compared with the 4K-only iMatrix candidate, it was directionally better on most 16K metrics, but the difference is small and the quick-run uncertainty overlaps. The clear result is that either iMatrix build substantially improves Q6_K over the no-iMatrix ablation.

Quick use

llama-cli.exe `
  -m .\Spark-X2.5-4B-Q6_K.gguf `
  -ngl 99 `
  -fa on `
  -c 16384

This llama.cpp build selects the embedded chat template without the obsolete -cnv flag. Reduce -ngl if another application is using VRAM. The 16K passkey checks below used -ngl 28 so the same compute path could also be used for the BF16 reference.

Fidelity results

All candidates were compared token-by-token against the same converted BF16 GGUF. Lower KLD/RMS is better; higher Same top p is better. “Same top p” is llama.cpp's name for top-1 agreement, not nucleus-sampling top-p.

Neutral held-out evaluation at 4K

Model	Mean KLD ↓	q99 KLD ↓	q99.9 KLD ↓	RMS Δp ↓	Same top p ↑	PPL(Q)
BF16 self-check	0.000000 ± 0.000000	0.000036	0.000049	0.001 ± 0.000%	100.000 ± 0.000%	6.910707 ± 0.268179
Q8_0	0.001354 ± 0.000096	0.011297	0.073939	1.275 ± 0.106%	98.192 ± 0.208%	6.906721 ± 0.267697
Q6_K, no iMatrix	0.013738 ± 0.001238	0.105705	1.387995	4.127 ± 0.338%	94.919 ± 0.343%	7.035107 ± 0.273836
Q6_K, 4K iMatrix	0.010021 ± 0.001285	0.064892	1.252379	3.950 ± 0.424%	95.896 ± 0.310%	6.977921 ± 0.271801
Q6_K, mixed 4K+16K iMatrix	0.008930 ± 0.000969	0.070500	0.778783	3.599 ± 0.382%	95.774 ± 0.314%	6.975759 ± 0.271574

Protocol: AtomicChat eval_neutral, 2 chunks, 4,096 scored positions, context 4,096, -ngl 28, batch 2,048, micro-batch 512, 8 threads and Flash Attention enabled. The BF16 KLD base is 1,073,283,076 bytes with SHA256 712B7248BCC72D0057EC56F61AF838D251A83EDA3F938DDD5E54FA03BFFCE322.

The same evaluation at 16K

Model	Mean KLD ↓	q99 KLD ↓	q99.9 KLD ↓	Max KLD ↓	RMS Δp ↓	Same top p ↑	PPL(Q)
BF16 self-check	0.000000 ± 0.000000	0.000035	0.000050	0.000063	0.001 ± 0.000%	100.000 ± 0.000%	8.565097 ± 0.173169
Q8_0	0.002709 ± 0.000307	0.015888	0.217027	3.244917	1.703 ± 0.152%	98.089 ± 0.107%	8.550264 ± 0.172665
Q6_K, no iMatrix	0.021932 ± 0.000852	0.178709	1.397083	5.835053	4.680 ± 0.164%	93.926 ± 0.187%	8.918584 ± 0.182141
Q6_K, 4K iMatrix	0.010021 ± 0.000345	0.074524	0.468998	2.875325	3.012 ± 0.120%	95.483 ± 0.162%	8.666384 ± 0.175889
Q6_K, mixed 4K+16K iMatrix	0.009891 ± 0.000325	0.074419	0.451430	2.363686	3.035 ± 0.124%	95.660 ± 0.159%	8.660181 ± 0.175757

Protocol: the same held-out corpus and compute path, 2 chunks, 16,384 scored positions, context 16,384. The BF16 KLD base is 4,294,705,156 bytes with SHA256 CB3BDBC2088CD3399ABCFD5D35B427B9F5C0463EFBF8FE4C9F1D68B28804AB52.

These are deliberately labelled quick2 measurements. They are useful for selecting a v0.1 candidate, not a substitute for a full benchmark suite.

iMatrix and calibration

Q6_K was quantized directly from the BF16 GGUF; it was never requantized from Q8_0.

BF16 reference SHA256: 8CECF405A41A4A10F833530910C2E13FDE9FB39C325C8AFC3C5D10E4181E1A14
iMatrix SHA256: 696688C6E4CB5AD6B1A2F4E3F460BBDCA50C2AC3CDBB7BC13C0455CA9F4BE489
Main calibration: 2,022,075 Spark tokens, context 4,096
Long-context calibration: 279,736 Spark tokens, context 16,384
Observations merged per tensor: 2,019,328 from the 4K pass and 278,528 from the 16K pass, approximately 88% / 12%
Batch size: 2,048; --parse-special enabled
Calibration corpus SHA256: DD4DADD436A3AB9329C2EE98000CC2FF6540E9C190400292A99EB736AA9E8796
Long-context corpus SHA256: 219DF033B7F9B30787C6DEC452FA16967A7C6FD4C0F2E42D84282E26C2AF940B
Corpus source: AtomicChat calib-corpora commit 814d662f6c94d207fc8f38545a1b4abea11484b5

Calibration and evaluation inputs are disjoint. The calibration manifests and the 3.4 MiB mixed-context iMatrix are included under artifacts/; the full text corpora are not redistributed in this model repository.

Reproduction of the selected quantization:

llama-quantize.exe `
  --imatrix .\artifacts\imatrix\Spark-X2.5-4B-imatrix-4k-16k.gguf `
  .\Spark-X2.5-4B-BF16.gguf `
  .\Spark-X2.5-4B-Q6_K-raw.gguf `
  Q6_K

python <LLAMA_CPP>\gguf-py\gguf\scripts\gguf_new_metadata.py `
  .\Spark-X2.5-4B-Q6_K-raw.gguf `
  .\Spark-X2.5-4B-Q6_K-reproduced.gguf `
  --remove-metadata quantize.imatrix.file `
  --force

The BF16 input is intentionally not included in this repository. Convert the pinned upstream revision with the same llama.cpp commit before reproducing the quantization.

The raw evaluated Q6_K container had SHA256 162985111D4DD15DA0FF2AD111965CABCF60690C40A52D53D2CB3640483DE51D. Before publication, the non-functional quantize.imatrix.file metadata key was removed because llama.cpp had stored its local absolute path. gguf_hash.py --no-layer gives the same tensor-data SHA256 before and after sanitization: f70482cf653bc90d935ab68ab83202d2a02c260699c5213d188d736c469bd1ae. The published container therefore has exactly the evaluated weights despite its different whole-file hash.

16K functional check

A deterministic synthetic passkey was inserted near 10%, 50% and 90% of a 14,997-token prompt. BF16, Q8_0, Q6_K without iMatrix, Q6_K with 4K iMatrix and Q6_K with mixed 4K+16K iMatrix all passed exact-match retrieval at all three depths: 15/15 runs.

This test is saturated and should only be read as a functional 16K smoke test. It does not distinguish the quantizations and does not establish general long-horizon reasoning quality. Prompts, manifests and result JSON files are included under artifacts/passkey_16k/.

RTX 4060 Laptop observations

llama-bench, CUDA, full GPU offload (-ngl 99), Flash Attention enabled:

File	pp512	tg128
Q8_0	4153.21 ± 224.67 tok/s	51.74 ± 0.25 tok/s
Q6_K, mixed iMatrix	3678.87 ± 101.74 tok/s	64.74 ± 0.20 tok/s

At context 16K, the reported VRAM figures were approximately 4.5 GB total for Q8_0 and 3.5 GB total for Q6_K. They include roughly 0.4–0.5 GB already allocated to the Windows/display system because this laptop setup has no iGPU handling the display. The model-run increments were therefore approximately 4.0–4.1 GB for Q8_0 and 3.0–3.1 GB for Q6_K. These were manual observations; treat VRAM and speed values as hardware- and runtime-dependent, not universal requirements.

Reproducibility artifacts

The artifacts/ directory contains:

parsed metrics in results.json;
source, model, iMatrix, evaluation-base and release hashes in audit-manifest.json and SHA256SUMS.txt;
raw KLD, quantization, iMatrix, tokenizer and speed logs;
the Q6_K public-container metadata sanitization and tensor-identity proof;
calibration manifests;
the exact benchmark and quantization scripts used locally (paths must be adapted before reuse);
the 16K passkey prompts and result JSON files.

The approximately 1 GiB and 4 GiB KLD base files are not uploaded in v0.1. Their SHA256 values, exact protocols and raw comparison logs are preserved so they can be regenerated and checked.

Next steps

The next experimental phase will explore mixed-context iMatrices more systematically. In particular:

vary the share of 4K and long-context calibration observations instead of testing only the current approximately 88% / 12% mix;
test additional long-context calibration lengths and task families, rather than relying on a single 16K corpus recipe;
evaluate each recipe at matching long contexts with the same BF16 reference and compute path;
retain a new recipe only when the improvement is repeatable across KLD, tail KLD and functional long-context checks.

The present 4K+16K result is therefore a useful first data point, not a claim that this mixture is optimal.

Validation status and limitations

HF/llama.cpp tokenizer parity passed 10/10 fixed cases, including multilingual text and special-token/chat-template cases.
GGUF architecture, tokenizer metadata and embedded chat template checks passed.
A direct Transformers-vs-GGUF logit parity comparison was not completed for v0.1. Consequently, KLD results establish fidelity to the converted BF16 GGUF; they do not independently prove that the experimental conversion is numerically identical to the upstream Transformers implementation.
Q8_0 has strong local provenance: the quantization command was recovered and metadata match the current BF16. However, the BF16 hash was not recorded at the exact time of the original Q8_0 run, so this is not a complete cryptographic chain of custody.
Spark2.5 support was experimental at the pinned llama.cpp commit. Compatibility with older releases is not expected.
No benchmark here validates the upstream maximum-context claim. Only 4K distributional checks and a 16K distributional/functional check were run.

See the upstream model card for architecture, intended use and model-level limitations. The upstream Apache-2.0 license is included as LICENSE.