Instructions to use prism-ml/Ternary-Bonsai-2-27B-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use prism-ml/Ternary-Bonsai-2-27B-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf prism-ml/Ternary-Bonsai-2-27B-gguf:F16
# Run inference directly in the terminal:
llama cli -hf prism-ml/Ternary-Bonsai-2-27B-gguf:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf prism-ml/Ternary-Bonsai-2-27B-gguf:F16
# Run inference directly in the terminal:
llama cli -hf prism-ml/Ternary-Bonsai-2-27B-gguf:F16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf prism-ml/Ternary-Bonsai-2-27B-gguf:F16
# Run inference directly in the terminal:
./llama-cli -hf prism-ml/Ternary-Bonsai-2-27B-gguf:F16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf prism-ml/Ternary-Bonsai-2-27B-gguf:F16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf prism-ml/Ternary-Bonsai-2-27B-gguf:F16

Use Docker

docker model run hf.co/prism-ml/Ternary-Bonsai-2-27B-gguf:F16

LM Studio
Jan

vLLM

How to use prism-ml/Ternary-Bonsai-2-27B-gguf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "prism-ml/Ternary-Bonsai-2-27B-gguf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prism-ml/Ternary-Bonsai-2-27B-gguf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/prism-ml/Ternary-Bonsai-2-27B-gguf:F16

Ollama
How to use prism-ml/Ternary-Bonsai-2-27B-gguf with Ollama:
```
ollama run hf.co/prism-ml/Ternary-Bonsai-2-27B-gguf:F16
```
Unsloth Desktop

How to use prism-ml/Ternary-Bonsai-2-27B-gguf with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf prism-ml/Ternary-Bonsai-2-27B-gguf:F16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "prism-ml/Ternary-Bonsai-2-27B-gguf:F16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use prism-ml/Ternary-Bonsai-2-27B-gguf with Docker Model Runner:
```
docker model run hf.co/prism-ml/Ternary-Bonsai-2-27B-gguf:F16
```

Lemonade

How to use prism-ml/Ternary-Bonsai-2-27B-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull prism-ml/Ternary-Bonsai-2-27B-gguf:F16

Run and chat with the model

lemonade run user.Ternary-Bonsai-2-27B-gguf-F16

List all available models

lemonade list

Hermes Agent

How to use prism-ml/Ternary-Bonsai-2-27B-gguf with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf prism-ml/Ternary-Bonsai-2-27B-gguf:F16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default prism-ml/Ternary-Bonsai-2-27B-gguf:F16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use prism-ml/Ternary-Bonsai-2-27B-gguf with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf prism-ml/Ternary-Bonsai-2-27B-gguf:F16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "prism-ml/Ternary-Bonsai-2-27B-gguf:F16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

在4060笔记本，8G下，优化到45Token/S之后，再提10%到50Token/S，但这还不是上限...理论上限可能高达65Token/S

#46

by SuperLogic - opened 14 days ago

Discussion

SuperLogic

14 days ago

先说重点，实测DFlash模式理论可行，Mtp只能接受2个草稿，改用DFlash做到3个草稿的话，理论上还有30%+的提速空间。期待大家来共同完成这件事情。
8GB 笔记本显卡跑 27B：45 → 50 t/s,无损,全程实操如下（关闭思考模式的速度，开思考会降一些）
平台:RTX 4060 Laptop(8GB)· 模型 Qwen3.8-27B-Ternary-PTQ1_0-MTP(三值量化 1.75 bpw,5.8 GiB)· 解码吞吐 45 t/s → 50 t/s(约 +11%),贪心 temp=0 逐 token 校验零差异、无损。
一、技术线路(怎么从 0 到 45,再到 50)
三值专用解码内核(pr-ptq1-mmv):Planar-Transposed 激活布局 + 定制 2D mat-vec,把 K=5120 投影的 lane 利用率从 31% 拉满到 128 线程,解决主干解码慢的根问题。
MTP 投机解码(--spec-type draft-mtp --spec-draft-n-max 2):用模型自带 nextn 头一次猜 2 个 token、主干批量验证,摊薄访存受限的逐 token 开销。
BATCH_INVARIANT 无损开关:保证"单发解码"与"投机 batch 内验证"的 logits 逐位一致 → MTP 严格无损。
显存工程:-ctk q4_0 -ctv q4_0 --kv-mean-center 让 24K 上下文和 5.8GB 权重在 8GB 内共存。
fork 独有缓存子系统:q8_cache(约 60% 量化发射重复,直接消重)、tq_rot_cache、算子融合、CUDA Graph + PDL 重叠启动。
→ 以上构成 45 t/s 的生产版(v1)。
最后一步 = 叠 #220 GDN gather:每层解码省掉一趟 GET_ROWS state 拷贝,kernel 直读 cache 行 → 50 t/s。
二、获得的代码(上游 PR)
仓库:github.com/PrismML-Eng/llama.cpp
#220(作者 professorpalmer)= GATED_DELTA_NET recurrent-state gather 折叠,本次提速主力。
#218(sudoingX)= GGML_CUDA_BATCH_INVARIANT 小批量 mat-vec(建 mmvq-ptq1_0.cuh)。
#217(sudoingX)= qwen35 MTP graph 的 Hadamard inverse。
#215 = PTQ1_0 SoA warp-per-row GEMV —— 与 #218 抢同一条 GEMV 路,已弃。
#221 = #218+#215+MTP 集成检查点;#210 = dflash Hadamard 借用修复。
拉取方式:git fetch origin refs/pull/220/head:pr220(其余同)。
三、分支与位置
fork 基座 tag:prism-b10709-9a9394a8;#220 与我们 fork 的 merge-base = 9a9394a8(真公共祖先)。
本地隔离树:
forks_speedmerge\repo(git 仓库,分支 full = base+#218+#217+#220);
forks\llamAmpere_gdn(v1 源码 + 手缝 #220),产物二进制:forks\llamAmpere_gdn\build-gdn\bin\llama-server.exe(这就是 50 t/s 那颗)。
启动脚本:llmserver3.bat(与生产 45 逐参数一致,唯一 delta 是 BIN + #220)。
说明:改动目前在本地隔离树,尚未 push 到公开分支;要发全球我可以建分支推上去(给个 go)。
四、怎么合并(方法,比结果更值得抄)
不是 git apply 硬贴(我们 fork 已删 raw-gates、launcher 改成显式级联、还多一个自研 ILP kernel,直接贴会全 reject)。做法:
clone 上游 → fetch PR refs → git merge-base 确认公共祖先;
git merge-file -p --diff3(只读)量化冲突:结果 common.cuh=0 / gdn.cu=2 / gdn.cuh=0 / ggml-cuda.cu=1 → 整次叠加只有 3 处要手解,全在 GDN,一丁点不碰 PTQ1_0 mat-vec;
0 冲突文件取三方合并版;3 处冲突把 gather 特性(而非整块 diff) 缝进 fork 自己的 kernel/launcher/图执行器;
关键暗坑:fork 独有的 gated_delta_net_cuda_ilp kernel 上游根本不知道,s_ids 必须一致地缝进主 + ILP 两个 kernel,否则解码走 ILP 路会读被跳过的空壳 → 静默算错。这是"能编、能跑、却悄悄出错"的雷。
五、复现入口
set GGML_CUDA_BATCH_INVARIANT=1
set GGML_CUDA_GRAPH_OPT=1 & set GGML_CUDA_PDL=1 & set LLAMA_SPEC_PQ=0
llama-server -m Qwen3.8-27B-Ternary-PTQ1_0-MTP.gguf ^
-fa on -ctk q4_0 -ctv q4_0 --kv-mean-center kv-bias.gguf ^
-c 24576 -b 4096 -ub 1024 -ngl 99 ^
--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0

BillFan666

12 days ago

用这个模型蒸馏一个MTP 接受度应该会提高速度就会快

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment