# Workload Scope and Real-World Relevance

This note clarifies what the benchmark actually measured. The headline result is a runtime comparison for short-form local inference on Qwen3.6 MTP GGUF models. It should not be read as a full agent benchmark, a full repository benchmark, or a cross-hardware performance claim.

## Two Measurement Modes

This repository includes two different measurement modes that answer different questions.

|mode|question answered|sample shape|how to interpret it|
|---|---|---|---|
|Repeated short-task benchmark|Which runtime is faster on the controlled compact coding/technical prompt suite?|21 prompt cases x 10 repeats per completed configuration, for 210 timed requests per completed config|Primary benchmark. Used for headline means, standard errors, paired confidence intervals, and actionability claims.|
|Exploratory cumulative long-output curve|How does throughput behave as output grows across thousands of generated tokens?|8 configurations, sampled at 1k-token output milestones from 1k to 8k, with n=1 to n=3 per milestone|Diagnostic charting only. Useful for intuition, but excluded from headline statistical claims.|

The headline "about 399 generated tokens per request" number refers only to the repeated short-task benchmark. It does not describe the cumulative long-output chart, where a single generation is tracked through 1,000-token output increments.

## Request Size

The curated headline rows used a fixed prompt set with 21 prompt cases and 10 repeats per completed configuration. Every completed configuration therefore has 210 timed request instances.

|measure|value|
|---|---:|
|Completed configurations|12|
|Prompt cases per completed configuration|21|
|Repeats per prompt|10|
|Timed requests per completed configuration|210|
|Total completed timed requests|2,520|
|Total measured prompt tokens|70,440|
|Total measured generated tokens|1,005,361|
|Average measured prompt tokens per request|28.0|
|Average measured generated tokens per request|399.0|
|Per-config average generated-token range|364.6 to 429.2|

The measured prompt-token count is the server timing field captured from the completed rows. It is very small because the headline suite was designed to compare decode behavior under controlled prompts, not to stress long-context ingestion. The context window was configured at 64k, but the primary rows do not fill that context.

## Instances and Repeats

There are three different meanings of "instance" that matter here:

|instance type|what was used|
|---|---|
|Hardware instance|One local Windows CUDA workstation|
|Runtime/model configuration instances|12 completed configurations, plus 6 unsupported or skipped configurations recorded in the exclusions table|
|Request instances|210 completed timed requests per completed configuration, 2,520 completed timed requests total|

The paired statistical comparisons use 210 matched rows per comparison, keyed by prompt, repeat, and seed. That gives good within-machine evidence for the tested prompt suite, but it does not replace replication on other graphics processors, central processing units, memory speeds, drivers, or llama.cpp builds.

## What Kind of Code and Tasks Were Tested

The prompt suite is intentionally biased toward coding-assistant and technical-agent use. It includes:

- Python: memoization, bug fixing, data aggregation, and API implementation planning.
- C++: template utility review and performance-oriented reasoning.
- TypeScript: typed paginated API design.
- Rust: lifetime explanation for a TypeScript-oriented developer.
- SQL and backend systems: Postgres indexing, FastAPI latency review, and API contract design.
- CUDA and inference operations: out-of-memory debugging and long-context slowdown analysis.
- Security and data handling: file-path review and JSON transformation.
- Control prompts: math, translation, factual question answering, explanation, summarization, and short creative generation.

The test is therefore most relevant to local coding-assistant turns where the user asks a compact technical question and the model generates a few hundred tokens. It is less directly applicable to workloads dominated by very long prompts, large pasted repositories, retrieval-augmented generation with thousands of context tokens, multi-turn tool use, or 10,000-token generation sessions.

## Why This Still Matters

Even though the prompt inputs are short, the comparison is useful because every completed runtime sees the same prompt, seed, sampling settings, model family, context size, MoE offload setting, and MTP draft count. That makes the observed wall-throughput differences a meaningful runtime comparison for the tested local decode path.

The result should be framed narrowly:

- Strong claim: on this workstation, for this 21-prompt coding/technical suite, QuinsZouls llama-next produced materially higher wall tokens per second than clean MTP llama.cpp and the completed Atomic TurboQuant rows.
- Moderate claim: the code-level evidence suggests the speedup is driven more by Quins' Qwen MTP execution path than by TurboQuant key-value compression alone.
- Unsupported claim: these numbers should not be used as a general guarantee for all long-context, multi-turn, vision, or tool-using agent workloads.

## Long-Output Rows

The repository includes exploratory long-output decode-curve charts, but they are not part of the headline statistical claim. Those rows were intended to approximate a 10k-token-in, 10k-token-out setting, but the synthetic prompt tokenized to roughly 22.9k tokens in one runtime and several streams stopped early.

The retained long-output summary has 45 milestone rows across 8 configurations. It samples cumulative and interval throughput at output-token marks from 1,000 to 8,000 generated tokens. Sample counts vary by milestone and configuration from n=1 to n=3, so these charts are best used to inspect the curve shape, not to make publication-grade claims about runtime ranking.