jcbtc/Ternary-Bonsai-27B-AMD-Strix-Halo

Ternary Bonsai 27B on AMD Strix Halo

Ciru's optimized DSpark more than doubles generation throughput

Three days of full-stack optimization turned Prism ML's DSpark path into a fast, validated AMD runtime. On Ryzen AI Max+ 395 / Radeon 8060S, Ciru's packaged N4 lane delivered 2.06x the weighted generation throughput of unmodified Prism DSpark on the same ROCm stack, with the same 39,693 output tokens and every output hash matched.

Locked nine-request DSpark A/B	Unmodified Prism DSpark	Ciru optimized DSpark	Improvement
Weighted generation throughput	15.864 tok/s	32.758 tok/s	+106.50% / 2.06x
Weighted prompt processing	170.830 tok/s	237.166 tok/s	+38.83%
Summed client time	2,519.32 s	1,224.27 s	51.40% lower
Completion tokens	39,693	39,693	exact match

That is the port-optimization comparison: Ciru's AMD DSpark implementation versus the unmodified Prism DSpark path as it ran on the same Strix Halo stack. The separate product comparison is also strong: the release serving policy cuts end-to-end latency versus official Prism AR by 45.51% at 256 tokens, 31.72% at 2K, and 21.41% at 4K.

This release brings Prism ML's official Ternary-Bonsai-27B-Q2_0.gguf and its paired DSpark drafter to AMD Ryzen AI Max+ 395 / Radeon 8060S systems. It packages AMD-specific kernel, graph, recurrent-state, speculative-decoding, long-context, and release engineering into a validated llama.cpp runtime for gfx1151.

It runs Prism's official Bonsai target and DSpark sidecar byte-for-byte. Download both model files from Prism ML, then run them with the Ciru AMD package.

The result is one validated binary closure with two explicit serving lanes:

• Bonsai AR is the stable default for general and long-context service. It keeps the full 262,144-token model context and is faster than the official control at the long end of the release sweep.
• Bonsai DSpark is the speculative lane, validated through 16,384-token rendered prompts. It delivers the largest end-to-end gains on short and medium prompts while retaining a full 262,144-token server allocation.

End-to-end latency against official AR

On AMD Ryzen AI Max+ 395 / Radeon 8060S, the release policy reduced total latency against an unmodified official Prism AR control at every measured prompt size:

Rendered prompt	Release lane	Prompt processing	Generation	Total latency	Change vs official AR
256 tokens	DSpark N4	257.62 tok/s	46.02 tok/s	12.12 s	45.51% lower
2,048 tokens	DSpark N4	293.28 tok/s	44.41 tok/s	18.51 s	31.72% lower
4,096 tokens	DSpark N4	306.46 tok/s	41.10 tok/s	25.83 s	21.41% lower
16,384 tokens	optimized AR	349.19 tok/s	21.90 tok/s	70.31 s	2.66% lower

Across one request at each size, DSpark used 130.87 seconds versus 154.45 seconds for official AR, a 15.27% reduction in aggregate wall time. At 16K, the release routes to its optimized AR lane; forced DSpark remained within 3.02% of official AR while accepting 80.91% of proposed tokens.

Every row used exactly 512 generated tokens, temperature zero, thinking off, one slot, Flash Attention, a cold prompt cache, and a 262,144-token server context. Each release output was byte-identical to the corresponding official control output.

Why this release matters

Bonsai is unusual: it retains 27B-class reasoning while storing its ternary language weights in 2-bit slots. Prism's official model card reports a roughly 7.2 GB target, 1.71 true bits per weight, 262K context, and 95% of the FP16 benchmark average. It also ships a target-specific DSpark drafter for lossless speculative decoding.

The missing piece for Strix Halo was a complete AMD serving path. The public Bonsai performance work centered on CUDA and Metal; simply compiling the same code for ROCm left recurrent state, low-bit kernels, capture, and verification far from the hardware's potential. Ciru optimized the entire path and packaged the result as a reproducible release rather than a one-off research checkout.

This creates practical value in three places:

1. The ordinary model is faster. Target-only Bonsai generation improved independently of speculative decoding.
2. DSpark becomes useful on AMD. The drafter, target capture, verification, and low-bit kernels were optimized as one pipeline.
3. The result is deployable. Full context, source provenance, launchers, binary closure, hash validation, safety checks, and measured workload routing are packaged together.

What Ciru delivered in three days

The release contains 30 source patches and 48 accepted engineering improvements across recurrent state, low-bit execution, speculative decoding, full-context operation, safety, testing, and packaging. Ciru took Bonsai far beyond basic ROCm compatibility: we profiled the complete serving stack, removed the actual bottlenecks, proved exact output, qualified model quality, and shipped a self-validating runtime closure.

Ciru's work spans the AMD integration, direct recurrent-state path, native gfx1151 Q2 and graph work, DSpark pipeline, full-context repair, production policy, validation, packaging, and benchmark evidence that make the complete stack a working release.

1. Gated Delta Net and recurrent-state execution

Qwen3.6's hybrid architecture spends most layers in linear attention, making recurrent-state handling central to real performance. The AMD runtime now:

• keeps the single-slot Gated Delta Net state directly on device;
• removes unnecessary state gathers, aliases, and copy-backs;
• validates identity recurrent-state mappings before using the fast path;
• pins the optimization to the safe plain single-slot serving geometry;
• covers the exact alias shape used by the official 27B model;
• keeps convolution-state concatenation contiguous; and
• uses a channels-major SSM convolution path that avoids an entire transpose.

These are model-preserving execution changes: they reduce movement and launch overhead without changing weights, positions, cache semantics, or model math.

2. Quantized activation and normalization reuse

The target and verifier repeatedly need the same normalized activations in Q8 form. The release avoids producing and moving those intermediates more than once:

• RMSNorm can produce reusable Q8 activations directly;
• residual addition and Q8 normalization are composed;
• multi-row verification reuses the same normalized Q8 data;
• Q8 buffer lifetimes follow the verification layout safely; and
• the Delta Net Q/K L2 normalizations are paired into one dispatch.

This work benefits both normal autoregressive decode and speculative target verification.

3. Native Q2 work for gfx1151

Bonsai's value comes from consuming the compact ternary representation directly. The AMD path therefore received low-bit kernels shaped around the actual 27B workload:

• specialized Q2 transpose handling for Strix Halo;
• transposed Q2 MMQ codes for more efficient access;
• batched Q2 MMVQ column reductions;
• paired output-row execution for N5 verification shapes;
• broader two-row MMVQ width coverage; and
• graph-safe QKV scheduling so concurrent work cannot reuse temporary storage incorrectly.

The final target-only composition improved TG128 from 23.8574 to 26.0979 tok/s, or 9.39%, with exact output. TG1 GPU dispatches fell from 2,156 to 1,660, a 23.0% reduction.

4. A real AMD DSpark execution path

DSpark spans an entire pipeline: target hidden-state capture, sidecar ingest, draft attention, confidence/Markov work, target verification, acceptance, and cache rollback. The release improves each stage:

• enables the Q4_1 Markov graph on the AMD backend;
• exposes the correct post-normalization confidence logits;
• restricts draft attention queries to rows that can produce draft tokens;
• preserves the complete Q path needed for correct attention;
• removes duplicate host staging of captured target features;
• compacts the trailing attention mask;
• gathers embeddings only for draft-token rows;
• reuses staged graphs when tensor shapes repeat;
• applies the multi-row Q8 reuse path during verification; and
• shares the same canonical gfx1151 specialization across the pipeline.

Against unmodified Prism DSpark running on the same AMD stack, the locked nine-request natural-completion A/B improved weighted generation throughput from 15.864 to 32.758 tok/s: +106.50%, or 2.06x. It preserved all 39,693 completion tokens and every output hash while cutting summed client time by 51.40%. A separate accepted six-request implementation screen still showed +87.07%, with all six output hashes matched.

5. Long-context capture without target-sized draft allocation

The original full-context path expanded temporary sidecar work toward the 262K target allocation. Ciru separated persistent context capacity from temporary draft compute:

• the server still allocates the full 262,144-token target context;
• DSpark capture ingest is bounded to an 8,192-row working batch;
• large initial captures are ingested in ordered chunks;
• persistent sidecar state and absolute target positions remain coherent;
• the real N4 proposal block is attached only to the final chunk; and
• invalid cache transitions fail closed instead of continuing with corrupted bookkeeping.

This removes the allocation failure that made a low-bit target plus sidecar look unexpectedly memory-hungry at large configured contexts.

6. Training-span-aware DSpark history

The shipped drafter was trained for a 4,096-token span. Ciru's recent-history policy keeps its direct attention history aligned with that span while leaving the target model fully contextual:

• target history remains complete;
• target positions remain absolute;
• target verification remains authoritative;
• only the sidecar's direct proposal history is bounded to the most recent 4,092 committed rows plus the current four-token proposal;
• short prompts are an identity path; and
• stale, overlapping, or inconsistent transitions park the speculative state rather than risk a wrong continuation.

This is what restores useful DSpark acceptance at 16K: 390 of 482 proposed tokens were accepted, or 80.91%, while target output remained exact.

7. Request-safe draft control

The server can lower effective draft depth per request without exceeding the startup allocation or changing the checkpoint's trained four-token block. The request value is bounded, re-evaluated against remaining context and output budget each round, and covered by focused parser and boundary tests. The production DSpark profile remains N4; this control exists for safe workload policy and diagnostics.

8. Production packaging instead of a research checkout

The release is a self-validating deployment package:

• one binary and shared-library closure for both lanes;
• full 262,144-token context by default;
• strict target and drafter SHA-256 checks;
• dynamic-library closure validation before model launch;
• loopback binding and API-key requirement by default;
• explicit AR and DSpark launchers with no silent lane switching;
• startup, health, properties, response, draft-counter, clean-unload, and GPU cleanup checks;
• complete source patch series and model/runtime provenance; and
• machine-readable performance and quality evidence.

Serving lanes

Launcher	Intended use	Prompt evidence	Context allocation	Slots
run-bonsai.sh / run-ar.sh	Stable default, long documents, agents, general serving	release AR lane, including 16K	262,144	configurable
run-dspark-short.sh	Lowest latency for short/medium prompts; opt-in speculative serving through 16K	256, 2K, 4K, 16K	262,144	1

The short launcher names the measured accelerated lane through 16K prompts; the server still allocates the full 262K context.

Files to download

Model weights come directly from Prism ML; the runtime archive contains the Ciru AMD software and deployment closure.

File	Source	SHA-256
Ternary-Bonsai-27B-Q2_0.gguf	Prism ML model repo	868c11714cf8fe47f5ec9eeb2be0ab1a337112886f92ee0ede6b855c4fa31757
Ternary-Bonsai-27B-dspark-Q4_1.gguf	Prism ML model repo	c4810091d244eddc61a0cc4966e584b0959f141e3c66c0d371a6652d9f647da9
bonsai-amd-strix-halo-v1.0.0-rc2-20260719.tar.zst	this repo	published beside the final release card

Quickstart

1. Download the official target and sidecar

mkdir -p models/Ternary-Bonsai-27B-gguf

hf download prism-ml/Ternary-Bonsai-27B-gguf \
  Ternary-Bonsai-27B-Q2_0.gguf \
  Ternary-Bonsai-27B-dspark-Q4_1.gguf \
  --local-dir models/Ternary-Bonsai-27B-gguf

2. Download and unpack the AMD runtime

hf download jcbtc/Ternary-Bonsai-27B-AMD-Strix-Halo \
  bonsai-amd-strix-halo-v1.0.0-rc2-20260719.tar.zst \
  --local-dir .

tar --zstd -xf bonsai-amd-strix-halo-v1.0.0-rc2-20260719.tar.zst
cd bonsai-amd-strix-halo-v1.0.0-rc2-20260719

3. Configure paths and validate

cp profile.env.example profile.env

# Edit BONSAI_MODEL_DIR and BONSAI_THEROCK in profile.env, then:
. ./profile.env
./run-bonsai.sh --validate
./run-dspark-short.sh --validate

The package expects a gfx1151 TheRock/ROCm runtime. It validates model hashes, the executable, and its shared-library closure before serving.

The prebuilt closure uses the standard Linux x86-64 ELF interpreter, runs directly on the host, and passed dependency and runtime testing in a clean Ubuntu 24.04 container.

4. Start the stable full-context service

. ./profile.env
export BONSAI_API_KEY='use-a-long-random-local-key'
./run-bonsai.sh

The default service binds to 127.0.0.1:18278 and exposes the llama.cpp OpenAI-compatible API.

5. Start the DSpark service

. ./profile.env
export BONSAI_API_KEY='use-a-long-random-local-key'
./run-dspark-short.sh

The DSpark service binds to 127.0.0.1:18277, uses the paired Q4_1 sidecar, fixed N4 drafting, Flash Attention, one slot, bounded capture ingest, and the recent-4092 history policy.

API example

curl http://127.0.0.1:18278/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $BONSAI_API_KEY" \
  -d '{
    "model": "Ternary-Bonsai-27B-Q2_0.gguf",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain ternary neural networks simply."}
    ],
    "temperature": 0.7,
    "top_p": 0.95,
    "top_k": 20,
    "max_tokens": 1024
  }'

Those generation settings follow Prism's upstream recommendations. The release benchmarks used temperature zero and thinking off only to create deterministic, exactly comparable performance and quality evidence.

Faster without giving up model quality

DSpark accelerates candidate proposal while the official Bonsai target remains the final authority for every accepted token. The recent-history optimization keeps full target history and target verification intact.

The optimized DSpark release tied the official AR control on HumanEval and Tool-Eval and improved HermesAgent-20 by 6 points. Release qualification uses complete thinking-off HumanEval/EvalPlus, HermesAgent-20, and all 69 Tool-Eval scenarios with the same generous output allowance used for the official control.

Quality gate	Official AR baseline	RC2 AMD release	Outcome
HumanEval	145/164 (88.41%)	145/164 (88.41%)	score tie
HumanEval+	141/164 (85.98%)	140/164 (85.37%)	one task lower
HermesAgent-20	72/100	78/100	+6 points
Tool-Eval	83/100	83/100	score tie

Tool-Eval completed all 69 scenarios with 52 pass, 11 partial, and 6 fail, identical to the official baseline distribution. HermesAgent-20 improved most strongly in skills/procedural memory and finished above both the thinking-off official control and the earlier 76-point thinking-on result.

All 164 HumanEval generations stopped naturally with zero request retries, generation-harness failures, token-cap hits, or non-stop finish reasons. One HumanEval+ test execution reached EvalPlus's isolated test timeout and was counted as a non-pass, separate from model inference. The release allowed up to 131,072 generated tokens and 7,200 seconds per request or scenario. Machine-readable results are in evidence/quality.json.

Prism's 15-benchmark thinking-mode evaluation describes the underlying model; these release scores validate quality across the optimized AMD runtime.

Context and memory

• Both launchers allocate the full 262,144-token model context by default.
• The packaged services started, reported that allocation, served a response, and unloaded cleanly.
• The performance sweep measured rendered prompts through 16K; a separate live gate validated the full 262K server allocation and serving path.
• On this unified-memory platform, the relevant model footprint is process-attributable VRAM+GTT: target and optional drafter weights, KV/cache state, graph workspace, and runtime allocations. System-wide totals include unrelated processes and are neither model-specific nor comparable across users, so this card does not report them.
• Bounded draft compute avoids scaling temporary sidecar workspaces to the full target context.
• Exact memory behavior remains dependent on context occupancy, KV types, ROCm runtime, and other processes sharing unified memory.

Compatibility

This package is built for:

• AMD Ryzen AI Max+ 395 / Radeon 8060S;
• gfx1151;
• TheRock ROCm 7.15 runtime layout used by the release; and
• Prism's exact Q2_0 target and Q4_1 DSpark files listed above.

This prebuilt closure is tuned and validated specifically for gfx1151. Other AMD architectures require their own build and validation pass. Prism's official runtimes cover CUDA, Metal, CPU, MLX, and Apple deployment.

Reproducibility and provenance

The archive includes:

• BUILD.md and build-from-source.sh for clean public reconstruction;
• complete source patches from the Prism lineage to the release composition;
• SOURCE.md with source and build identities;
• RELEASE.json and model-manifest.json;
• accepted and rejected performance evidence;
• focused test results and runtime sanity evidence; and
• artifact-wide SHA-256 checksums.

The packaged llama-server SHA-256 is:

0d09159021a361492acf22501809bf4ff17574716c15d52e2d7652b48aedc8ae

Benchmark scope

The +106.50% result compares optimized and unmodified DSpark on the same Strix Halo ROCm stack. The up to 45.51% lower latency curve separately compares the release serving policy with official Prism AR. Both use the official unchanged target and sidecar files. Performance was measured through 16K prompts; full 262K allocation and serving were validated separately. The stable AR lane remains available for unrestricted workloads.

Credits

• Prism ML for Ternary Bonsai 27B, the ternary conversion, Q2_0 format, DSpark sidecar, reference integration, model evaluation, and Apache-2.0 release.
• Qwen for the Qwen3.6-27B hybrid-attention base architecture.
• Gaurav Garg / NVIDIA for Gated Delta Net state-copy folding, merged in llama.cpp PR #23940.
• Piotr Wilkin for RMSNorm-produced Q8 reuse, residual-plus-RMSNorm fusion, and paired Delta Net L2 normalization, tracked in llama.cpp PR #25793.
• Robert Esclapez Garcia / AMD for contiguous Delta Net concatenation and channels-major SSM convolution, merged in ROCm/llama.cpp PR #52.
• llama.cpp, ggml, TheRock, and ROCm contributors for the serving and AMD compute foundations.
• Ciru Inference Lab for integrating and adapting the combined runtime, direct single-slot recurrent-state work and guards, gfx1151 Q2 and graph optimization, AMD DSpark pipeline work, context-capacity repair, release policy, validation, packaging, and AMD benchmark evidence.

Further reading

• Official Ternary Bonsai 27B GGUF card
• Prism ML Bonsai demo and whitepaper
• Prism ML llama.cpp fork
• RELEASE-NOTES.md in this repository for the complete accepted improvement inventory
• BUILD.md in this repository for the clean-source gfx1151 build recipe