Long-Context Retrieval: Single-Needle Recall Holds to 1M Tokens (3.85x YaRN)

Model: PiehSoft/Qwen3.6-40B-Deckard-MTP (the MTP-grafted variant)
Base merge: DavidAU/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-NEO-CODE-Di-IMatrix-MAX-GGUF
Hardware: Single NVIDIA RTX PRO 6000 Blackwell (96GB)
Runtime: llama.cpp (build 9352 / b4c0549a4), native sm_120a CUDA kernels

TL;DR

This model retrieves a single planted fact ("needle") cleanly from contexts up to
1,010,000 tokens using YaRN extension (3.85x over the 262,144 native window), at
100% pass rate across every needle depth tested, on a single workstation GPU.

There is a llama.cpp server bug that caps context to the training length regardless of
YaRN settings; the workaround is documented below.

This is a single-needle retrieval result. It demonstrates recall, not multi-hop
reasoning at length. See the Scope section before drawing broader conclusions.

Results

Needle-in-a-haystack (NIAH): a coherent filler text is built to a target token length,
an out-of-distribution fact is planted at a given depth, and the model is asked to recall
it. Pass = exact match of the planted value. Token-accurate haystack sizing via the real
Qwen tokenizer.

Config: Q6_K, q8 keys / q4 values KV cache, ubatch 2048, FlashAttention on.

length     0%    25%   50%   75%   100%   extension factor
131072    PASS  PASS  PASS  PASS  PASS    native
262144    PASS  PASS  PASS  PASS  PASS    native
393216    PASS   -    PASS   -    PASS    1.5x YaRN
524288     -     -    PASS   -     -      2x   YaRN
1010000   PASS  PASS  PASS  PASS  PASS    3.85x YaRN

(- = not run, not a failure. The 393K/512K rows were probes; native and 1M are full
depth sweeps.)

Native 262K is perfect across all depths. Extension via YaRN holds clean with no
lost-in-the-middle degradation at any tested length, including the full 1M sweep.

VRAM

Single instance at 1M context loads at ~90GB with q8 keys / q4 values, leaving
headroom on the 96GB card. (An earlier 73GB reading that suggested tighter limits turned
out to be two model instances running simultaneously; single-instance footprint is lower
than that scare implied, but the q4 value cache is doing real work to keep 1M under the
ceiling. Full q8 KV at 1M would not fit.)

Prefill speed at length

Prefill (prompt processing) decays with context as expected from the attention term, but
gracefully:

context     prefill tok/s (approx, cold)
~12K        ~2590
~32K        ~2190
~131K       ~1410
~262K       ~1090
~512K       ~700
~1M         ~390

A full 1M prefill takes roughly 34 minutes on this hardware. This is a one-time entry
cost to reach high context, not a per-token tax once the context is loaded.

The llama.cpp context-cap workaround

llama.cpp's server caps n_ctx to the model's declared training context regardless of
your YaRN flags. Symptom in the startup log:

n_ctx_seq (524288) > n_ctx_train (262144) -- possible training context overflow
the slot context (524288) exceeds the training context of the model (262144) - capping
new slot, n_ctx = 262144

This is a known issue (ggml-org/llama.cpp#22140).
Bare --rope-scaling yarn flags do not defeat it, and this GGUF ships with no
rope.scaling metadata baked in (architecture prefix is qwen35), so the CLI flags alone
get clamped.

The fix is to override the declared training context so the cap check passes:

--override-kv qwen35.context_length=int:1010000 \
--rope-scaling yarn \
--rope-scale 3.85 \
--yarn-orig-ctx 262144 \
-c 1010000 \
--cache-type-k q8_0 --cache-type-v q4_0 \
--ctx-checkpoints 0

Notes:

• --override-kv qwen35.context_length=int:1010000 raises the declared training context.
  Set it to your maximum target once; control the actual context with -c.
• --rope-scale is target / native. For 512K that's 2.0; for 1M, 1010000 / 262144 = 3.85.
• --ctx-checkpoints 0 is for clean one-shot benchmarking only. For interactive use,
  leave checkpoints on so multi-turn sessions reuse cache.
• A correct launch shows new slot, n_ctx = 1010000 and a "full capacity will not be
  utilized" note instead of the "capping" line.

A second gotcha for anyone scripting NIAH against the server: a 1M prefill exceeds the
OpenAI client's default ~600s read timeout, which silently cancels the request mid-prefill.
Set an explicit infinite read timeout on the client (httpx.Timeout(read=None)).

Scope and honest caveats

This result is deliberately narrow. What it shows and does not show:

• Single-needle, not multi-hop. This proves the model can retrieve one planted fact
  at extreme length. It does not show it can reason across multiple facts at length,
  which is what real long-context work (e.g. codebase reasoning) requires. Multi-needle
  and multi-hop at extension are the next tests.
• trials=1 per cell. 5/5 PASS across depths is a strong signal, but it is "did not
  fail once," not a statistical reliability claim. Paper-grade would want trials=3-5.
• q8 keys / q4 values, not full q8 KV. Results are valid for that cache config, which
  is how you'd realistically run 1M on this card anyway.

Why it extends this well (hypothesis)

Qwen3.6 is a hybrid: gated linear-attention / SSM layers with full attention only every
4th layer. YaRN only acts on the RoPE'd full-attention layers, so there are far fewer
layers to degrade under positional extension than in a pure-attention model. The same
property keeps the KV cache cheap as context grows (most layers don't carry a per-token
KV cost), which is why 1M fits in ~90GB. The architecture appears to be doing the heavy
lifting on both the retrieval-stability and the memory-footprint fronts.

Reproduce

The NIAH harness used to produce this grid is in the repo under
benchmarks/niah_smoke.py. It builds
token-accurate haystacks, plants randomized out-of-distribution needles, and scores exact
recall across a length x depth grid. See the script header for usage.

Spec-decode tuning for this model (combined ngram + MTP, and why n_max 2 beats n_max 3
on this graft) is written up in
benchmarks/SPEC_DECODE_TUNING.md,
with its harness at benchmarks/decode_speed.py.