--- license: apache-2.0 pipeline_tag: text-generation base_model: Qwen/Qwen3.8-27B inference: false tags: - speculative-decoding - dspark - dflash - draft-model - specforge - sglang - memra - qwen3 - agentic - conversational model-index: - name: Qwen3.8-27B-DSpark-Agentic results: - task: type: text-generation name: Speculative decoding acceptance, DSpark block size 7 dataset: name: held-out real agentic session prompts (128 turns, two length buckets) type: agentic-sessions-heldout metrics: - name: mean acceptance length, short chat turns (n=64) type: acceptance-length value: 2.88 - name: mean acceptance length, long agentic turns (n=64) type: acceptance-length value: 2.92 - task: type: text-generation name: Speculative decoding acceptance, DSpark block size 7 dataset: name: GSM8K type: openai/gsm8k metrics: - name: mean acceptance length (128 prompts, concurrency 8) type: acceptance-length value: 4.61 - name: aggregate decode throughput, 1x RTX PRO 6000 Blackwell 96GB type: throughput value: 693 --- # Qwen3.8-27B-DSpark-Agentic A **DSpark speculative-decoding drafter** for [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B), trained on **real agentic coding/assistant sessions**. DSpark drafts a block of tokens conditioned on auxiliary features tapped from the target model's layers, with a Markov head for intra-block token dependency and a confidence head that predicts per-position acceptance. Trained with [SpecForge](https://github.com/sgl-project/SpecForge); serves with [SGLang](https://github.com/sgl-project/sglang). A drafter never changes model output: every drafted token is verified by the target model. It only changes decode speed, through the acceptance rate on your traffic. This one is aimed at agentic/assistant workloads. ## Model specifications | | | |---|---| | Target model | [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) (evals below ran FP8 and NVFP4 builds of this trunk) | | Draft parameters | 1,359,284,737 (1.36B), BF16, 62 tensors in one safetensors file | | Architecture | 5 full-attention transformer layers, hidden 5,120, GQA 40 query / 8 KV heads, head_dim 128, MLP 10,240 | | Target feature taps | layers 4, 16, 28, 40, 52 (of 64) | | Heads | vanilla Markov head, rank 256; confidence head with Markov features | | DSpark block size | 7 draft tokens (verify width 8 with the target bonus token) | | Positional encoding | YaRN (factor 32, original 8,192), max position 262,144 | | Vocabulary | 248,320 (uses the target model's tokenizer; no separate tokenizer shipped) | `model.safetensors` sha256: `3c4e7d51f6a7492fd3c289e392424ab0fba31dc7b4ab4896da10a5eeb21fcd6f` ## Training Trained **from scratch** (random initialization — no third-party model weights) with the SpecForge harness. Labels are the target model's own temperature-0 regenerations over a prompt mix of real agentic coding/assistant sessions and a public chat prompt blend, covering both thinking and non-thinking modes, training on the last turn of each multi-turn session. Roughly 1,000 cumulative optimizer steps on B200 / RTX PRO 6000-class hardware; this repo is the final checkpoint of that run. ## Evaluation results All numbers below are absolute measurements of **this checkpoint**. ### Acceptance length — SGLang, FP8 target, sampled thinking-mode decoding Setup: SGLang; target `Qwen/Qwen3.8-27B-FP8` on a single RTX PRO 6000 Blackwell (96 GB); this drafter unquantized BF16; DSpark block size 7; temperature 0.6, top-k 20, top-p 0.95; thinking enabled; `max_new_tokens=2048`. Acceptance length = mean accepted tokens per verification step including the target bonus token (per-request `completion_tokens / spec_verify_ct`, unweighted mean). | Workload | Prompts | Acceptance length | |---|---:|---:| | Real agentic sessions — short chat turns | 64 | **2.88** (per-request 2.09–5.94) | | Real agentic sessions — long agentic turns | 64 | **2.92** (per-request 2.03–5.45) | | GSM8K | 128, concurrency 8 | **4.61** | The GSM8K cell decoded 60,365 output tokens in 87.1 s — **693 tok/s aggregate** at concurrency 8 on the single card. The two session buckets produced 81,199 and 115,263 output tokens respectively. The session prompts are held-out real agentic/assistant turns, not used in training. ### Acceptance — greedy argmax, NVFP4 GGUF trunk (memra engine) Setup: [memra](https://github.com/avifenesh/memra) engine, greedy decoding, single stream, RTX PRO 6000 Blackwell (96 GB); trunk = [NVFP4 GGUF build of the same target](https://huggingface.co/Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF); drafter BF16, drafting through the trunk's own embeddings. The speculative stream is gated **byte-identical** to plain greedy decode on every prompt. Tokens per round = 1 + accepted/rounds. | Workload | Generation length | Tokens per round | tok/s with drafter | |---|---:|---:|---:| | Agentic-session prompts (12) | 128 | 1.43 (range 1.16–1.60) | 62.5–62.8 | | Agentic-session prompts (12) | 768 | ~1.50 | 65.7 | | Math-class prompts (4) | 256 | 1.69 (max 1.88) | 75.7 | | Math-class prompts (4) | 768 | 1.5–1.8 | 75.0 | Acceptance is protocol-dependent: sampled thinking-mode decoding on an FP8 trunk (first table) and greedy argmax on an NVFP4 trunk at short generation lengths (second table) are different observables. Both are reported as measured; expect your own numbers to depend on trunk quantization, sampling settings, and traffic shape. ## Serving SGLang (use a version with DSPARK support): ```bash sglang serve \ --trust-remote-code \ --model-path Qwen/Qwen3.8-27B-FP8 \ --tp-size 1 \ --speculative-algorithm DSPARK \ --speculative-draft-model-path Avifenesh/Qwen3.8-27B-DSpark-Agentic \ --speculative-dspark-block-size 7 \ --speculative-draft-model-quantization unquant \ --mamba-scheduler-strategy extra_buffer ``` [memra](https://github.com/avifenesh/memra) (Rust + CUDA engine for RTX Blackwell) has the DSpark route wired and exactness-gated — spec output byte-identical to plain decode — landing in an upcoming release behind an env-gated flag. The team behind this drafter serves Qwen3.8-27B in production at [inference.tiyuvta.ai](https://inference.tiyuvta.ai). ## License Apache-2.0. The drafter was trained from scratch; no third-party model weights were used for initialization. Training labels were generated by [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) (Apache-2.0); training used the open-source SpecForge harness. ## Citations - DSpark (speculative decoding method): [arXiv:2607.05147](https://arxiv.org/abs/2607.05147) - DFlash (block drafting lineage): [z-lab/dflash](https://github.com/z-lab/dflash), [arXiv:2602.06036](https://arxiv.org/abs/2602.06036) - SpecForge (training harness): [sgl-project/SpecForge](https://github.com/sgl-project/SpecForge)