tepirale/Ornith-Agents-A1-3.6-35B-A3B-MTP-GGUF

Ornith-Agents-A1-3.6-35B-A3B-GGUF

Multi-quant GGUF with a grafted MTP block for native speculative decoding in llama.cpp.

This repository contains GGUF quantizations of tepirale/Ornith-Agents-A1-3.6-35B-A3B-dare_ties, a dare_ties merge on top of Qwen3.5-35B-A3B (MoE), with a Multi-Token Prediction (MTP) block grafted in from the official Qwen3.6-35B-A3B GGUF. This lets the model use its own MTP block as a draft model for speculative decoding, with no need for a separate external draft model.

💡 TL;DR: download a quant + the MTP block, launch llama-server with --spec-type draft-mtp, and get native speculative decoding with ~45% draft acceptance and up to ~260 tok/s measured on an H100 (25 GB VRAM in use). GPU NVIDIA RTX 6000 Ada Generation 96GB V-RAM

---

Table of contents

• Available quants
• MTP graft — technical details
• Usage (llama.cpp)
• basic code and Performance benchmark
• Base model and lineage
• Limitations and notes

---

Available quants

Quant	Size	Draft head (MTP) precision	Recommended use
Q4_K_M	20.5 GiB	Q8_0	General use / limited VRAM
Q5_K_M	23.9 GiB	Q8_0	Quality/size balance
Q6_K	27.4 GiB	Q8_0	High fidelity
Q8_0	35.2 GiB	Q8_0	Maximum fidelity
f16	69.4 GiB	F16	Reference / debugging

Each quant ships with its own grafted MTP block (blk.40). Also distributed separately:

• Ornith-Agents-A1-3.6-35B-A3B-dare_ties-mtp-sidecar.gguf — the raw MTP block, extracted from the official Qwen3.6-35B-A3B GGUF, for anyone who wants to graft it onto other quants or custom builds.

---

MTP graft — technical details

• The MTP block is grafted after quantization, not before — so every final quant keeps full-precision draft heads regardless of the main model's quant level.
• Composition of the grafted block:
  • General weights: Q8_0
  • Routing gates: BF16
  • Norms: F32
  • No degradation of the draft heads across any of the published quants.
• Resulting graph metadata: block_count=41, nextn_predict_layers=1 (the MTP block is added as layer 41, on top of the base model's 40 layers).
• All quants are generated directly from F16, avoiding double quantization (F16 → Q4/Q5/Q6/Q8 in a single pass).

---

Usage (llama.cpp)

1. Build llama.cpp with CUDA support

apt-get update && apt-get install -y build-essential cmake git libcurl4-openssl-dev
pip install -U -q openai
git clone https://github.com/ggml-org/llama.cpp
pip install -U --force-reinstall "huggingface_hub[hf_transfer]"

cd llama.cpp
cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=90 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc) \
  --target llama-server llama-cli llama-quantize llama-gguf-split

Adjust CMAKE_CUDA_ARCHITECTURES for your GPU (90 = Hopper/H100; use 89 for Ada/L40S, 86 for Ampere consumer, etc.).

2. Download the quant you want + the MTP sidecar

hf download tepirale/Ornith-Agents-A1-3.6-35B-A3B-MTP-GGUF \
  --include "Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q4_K_M.gguf" \
  --include "Ornith-Agents-A1-3.6-35B-A3B-dare_ties-mtp-sidecar.gguf" \
  --include "mmproj-F32.gguf" \
  --local-dir ./models/ornith-a1

3. Launch the server with MTP speculative decoding

Tested on a GPU with 95.6 GB VRAM (actual model usage: ~25 GB):

# configuration with 250 tok/s acceptable and depending on (spec-draft-n-max)
./build/bin/llama-server \
  -m ./models/ornith-a1/Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q4_K_M.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --spec-draft-n-min 1 \
  --host 0.0.0.0 --port 8888 \
  --n-gpu-layers 999 \
  --ctx-size 131072 \
  --flash-attn on \
  --cont-batching \
  --parallel 4 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --metrics

Tested on a GPU with 95.6 GB VRAM (actual model usage: ~43 GB):

# configuration with 190 tok/s but gives better results (more VRAM)
./build/bin/llama-server \
  -m ./models/ornith-a1/Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q8_0.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-n-min 1 \
  --spec-draft-p-min 0.92 \
  --host 0.0.0.0 --port 8888 \
  --n-gpu-layers 999 \
  --ctx-size 254000  \
  --flash-attn on \
  --cont-batching \
  --parallel 4 \
  --batch-size 4096 \
  --ubatch-size 1024 \
  --reasoning-preserve \
  --metrics

# MTP DISABLED | 150-200 tok/s 
./build/bin/llama-server \
  -m ./models/ornith-a1/Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q8_0.gguf \
  --host 0.0.0.0 --port 8888 \
  --n-gpu-layers 999 \
  --ctx-size 254000  \
  --flash-attn on \
  --cont-batching \
  --parallel 4 \
  --batch-size 8192 \
  --ubatch-size 1024 \
  --reasoning-preserve \
  --temp 0.7 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --metrics

# mtp act - inference image act

# mmproj-F32.gguf -> https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder-MTP-GGUF/tree/main

./build/bin/llama-server \
  -m ./models/ornith-a1/Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q4_K_M.gguf \
  --mmproj /models/ornith-a1/mmproj-F32.gguf \
  --image-min-tokens 1024 \
  --chat-template-file /content/chat_template.jinja \
  --reasoning-preserve \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-n-min 1 \
  --host 0.0.0.0 --port 8080 \
  --n-gpu-layers 999 \
  --ctx-size 5000 \
  --flash-attn on \
  --cont-batching \
  --parallel 4 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --metrics

Notes on key flags:

• --spec-type draft-mtp: enables the internal MTP block as the draft model (no need for -md with an external model) | Enabling MTP uses ~2GB..
• --spec-draft-n-min/max: range of draft tokens per speculative step.
• --cache-type-k/v q8_0: quantizes the KV cache to reduce VRAM usage with long context (131k tokens).

4. Test client (OpenAI-compatible)

The server exposes an OpenAI-compatible API at http://localhost:8888/v1, so any client based on openai / AsyncOpenAI works out of the box. See the benchmarking section below for a full example with tok/s metrics and MTP draft acceptance rate.

---

basic code and Performance benchmark

import asyncio
import time
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="http://localhost:8888/v1",
    api_key="sk-no-key-required"
)

MODEL = "ornith-a1"

# --- Prompts escalados por tamaño de max_tokens ---
# Cada nivel pide más subtemas/profundidad para forzar que el modelo
# necesite ese espacio y no termine antes de tiempo (finish_reason='stop' prematuro).

def build_prompt(max_tokens: int) -> str:
    # Escala el número de subtemas según el presupuesto de tokens.
    # ~1 subtema profundo consume aprox 250-350 tokens en español con razonamiento incluido.
    n_subtemas = max(2, round(max_tokens / 300))

    subtemas_pool = [
        "la formulación matemática de DARE (Drop And REscale) y su relación con task_arithmetic",
        "cómo TIES resuelve conflictos de signo entre parámetros de distintos modelos fuente",
        "ventajas de dare_ties frente a un merge lineal simple (linear merge)",
        "desventajas y casos donde dare_ties degrada el rendimiento del modelo resultante",
        "el rol del parámetro de densidad (density) en la retención de pesos",
        "cómo afecta el número de modelos fuente combinados a la estabilidad del merge",
        "comparación entre dare_ties y model soups en términos de generalización",
        "el impacto de mergear modelos con arquitecturas MoE como Qwen3.6-A3B",
        "estrategias de evaluación post-merge (benchmarks recomendados y por qué)",
        "buenas prácticas al elegir los pesos de mezcla (weight) por modelo fuente",
        "cómo dare_ties interactúa con fine-tuning posterior (SFT/GRPO) del modelo mergeado",
        "riesgos de catastrophic forgetting al combinar modelos con dominios muy distintos",
    ]

    subtemas = subtemas_pool[:n_subtemas] if n_subtemas <= len(subtemas_pool) else (
        subtemas_pool * (n_subtemas // len(subtemas_pool) + 1)
    )[:n_subtemas]

    lista = "\n".join(f"{i+1}. {t}" for i, t in enumerate(subtemas))

    return (
        f"Escribe una explicación técnica exhaustiva sobre model merging con la técnica dare_ties, "
        f"cubriendo en detalle y con profundidad cada uno de los siguientes {n_subtemas} puntos "
        f"(desarrolla cada uno en varios párrafos, con ejemplos y justificación técnica):\n\n{lista}\n\n"
        f"No resumas ni omitas puntos. Desarrolla cada sección completamente."
    )


async def run_test(max_tokens: int) -> dict:
    prompt = build_prompt(max_tokens)

    start = time.perf_counter()
    response = await client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": "Eres un experto en machine learning que explica con profundidad técnica."},
            {"role": "user", "content": prompt}
        ],
        temperature=0.7,
        max_tokens=max_tokens,
        extra_body={"chat_template_kwargs": {"enable_thinking": True}}
    )
    elapsed = time.perf_counter() - start

    usage = response.usage
    raw = response.model_dump()
    timings = raw.get("timings", {})

    return {
        "max_tokens": max_tokens,
        "prompt_tokens": usage.prompt_tokens,
        "completion_tokens": usage.completion_tokens,
        "elapsed_s": round(elapsed, 2),
        "tok_s_medido": round(usage.completion_tokens / elapsed, 2) if elapsed > 0 else 0,
        "tok_s_nativo": timings.get("predicted_per_second"),
        "prompt_tok_s": timings.get("prompt_per_second"),
        "draft_n": timings.get("draft_n"),
        "draft_accepted": timings.get("draft_n_accepted"),
        "finish_reason": response.choices[0].finish_reason,
    }


async def main():
    max_tokens_list = [512 * i for i in range(1, 10)]  # 512 .. 4608
    results = []

    for mt in max_tokens_list:
        print(f"Probando max_tokens={mt}...")
        r = await run_test(mt)
        results.append(r)

    # --- Tabla comparativa ---
    header = (f"{'max_tok':>8} | {'compl_tok':>9} | {'time(s)':>8} | "
              f"{'tok/s medido':>13} | {'tok/s nativo':>13} | {'draft acc%':>10} | {'finish':>8}")
    print("\n" + header)
    print("-" * len(header))
    for r in results:
        if r["draft_n"] and r["draft_accepted"] is not None:
            acc_pct = f"{100 * r['draft_accepted'] / r['draft_n']:.1f}%"
        else:
            acc_pct = "-"
        print(f"{r['max_tokens']:>8} | {r['completion_tokens']:>9} | {r['elapsed_s']:>8} | "
              f"{r['tok_s_medido']:>13} | {str(r['tok_s_nativo']):>13} | {acc_pct:>10} | {r['finish_reason']:>8}")

    return results


if __name__ == "__main__":
    # asyncio.run(main())
    await main()

Measured with an async script that scales prompt/response length and records finish_reason, measured vs. native tok/s (llama.cpp timings), and MTP draft acceptance rate.

max_tokens	tokens generated	time (s)	tok/s measured	tok/s native	draft acceptance	finish
512	512	2.33	219.95	263.31	48.2%	length
1024	1024	4.04	253.50	260.19	46.9%	length
1536	1536	6.33	242.67	247.87	43.5%	length
2048	2048	8.47	241.72	245.74	43.0%	length
2560	2560	10.10	253.35	257.73	45.9%	length
3072	3072	12.16	252.57	256.01	44.7%	length
3584	3584	14.62	245.08	248.14	42.3%	length
4096	4096	15.60	262.53	266.24	47.9%	length
4608	4608	18.00	256.04	259.80	46.2%	length

Quick read: the MTP draft acceptance rate stays stable in the 42–48% range regardless of generation length, and measured throughput converges to ~245–260 tok/s, very close to llama.cpp's reported native throughput — indicating minimal overhead from the measurement pipeline.

Quant used in the benchmark: Q4_K_M, GPU with 95.6 GB VRAM (~25 GB used by the model), --parallel 4, context 131072.

VLLM: 130 tok/s measured

---

Base model and lineage

• Merged model: tepirale/Ornith-Agents-A1-3.6-35B-A3B-dare_ties
• Merge technique: dare_ties (mergekit) on top of Qwen3.6-35B-A3B
• MTP block source: official Qwen3.6-35B-A3B GGUF
• Architecture: Mixture-of-Experts (A3B), 40 base layers + 1 grafted MTP layer (41 total)

---

spec-draft-n-max

• spec-draft-n-max 4 (250 tokens/s) | Draft 70% (50/50 Sometimes it responds well, sometimes it doesn't)
• spec-draft-n-max 3 (270 tokens/s) | Draft 80% (gives better answers)
• spec-draft-n-max 2 (277 tokens/s) | Draft 86% (I don't think it's a good idea to use it because it can get stuck in a loop of reasoning.)

example code us Q8_0 / GPU G4 use 40GB VRAM (total gpu 95.5GB VRAM Google Colab)

from openai import OpenAI

def chat(
    prompt,
    system="Eres un asistente útil.",
    enable_thinking=True,
    stream=True,
    max_tokens=300,
    temperature=0.7,
    model="ornith-a1",
    verbose=True,
    show_metrics=False,
):
    """
    Wrapper unificado para llama-server (OpenAI-compatible) que maneja:
    - stream=True/False
    - enable_thinking=True/False
    - separación de reasoning_content vs content en ambos modos
    - métricas de tokens/velocidad (show_metrics=True)
    """
    extra_body = {
        "chat_template_kwargs": {"enable_thinking": enable_thinking}
    }

    create_kwargs = dict(
        model=model,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": prompt},
        ],
        temperature=temperature,
        max_tokens=max_tokens,
        stream=stream,
        extra_body=extra_body,
    )

    if stream:
        # Pide que el usage venga en el último chunk (soportado por llama-server reciente)
        create_kwargs["stream_options"] = {"include_usage": True}

    response = client.chat.completions.create(**create_kwargs)

    reasoning_buf, content_buf = "", ""
    usage = None
    timings = None

    if not stream:
        msg = response.choices[0].message
        reasoning_buf = getattr(msg, "reasoning_content", None) or ""
        content_buf = msg.content or ""
        finish_reason = response.choices[0].finish_reason
        usage = response.usage
        timings = getattr(response, "timings", None)

        if verbose:
            if reasoning_buf:
                print("🧠 Razonamiento:\n" + reasoning_buf)
                print("\n" + "-" * 50 + "\n")
            if content_buf:
                print("💬 Respuesta:\n" + content_buf)
            if not content_buf and reasoning_buf:
                print(f"⚠️  finish_reason='{finish_reason}' — se acabaron los tokens pensando. Sube max_tokens.")

    else:
        finish_reason = None
        printed_separator = False

        for chunk in response:
            if not chunk.choices:
                # chunk final solo con usage (cuando include_usage=True)
                if getattr(chunk, "usage", None):
                    usage = chunk.usage
                timings = getattr(chunk, "timings", None) or timings
                continue

            choice = chunk.choices[0]
            delta = choice.delta
            finish_reason = choice.finish_reason or finish_reason

            r = getattr(delta, "reasoning_content", None)
            c = delta.content

            if r:
                reasoning_buf += r
                if verbose:
                    print(r, end="", flush=True)
            if c:
                if verbose and reasoning_buf and not printed_separator:
                    print("\n" + "-" * 50 + "\n")
                    printed_separator = True
                content_buf += c
                if verbose:
                    print(c, end="", flush=True)

            if getattr(chunk, "usage", None):
                usage = chunk.usage
            timings = getattr(chunk, "timings", None) or timings

        if verbose:
            print()
            if not content_buf and reasoning_buf:
                print(f"⚠️  finish_reason='{finish_reason}' — se acabaron los tokens pensando. Sube max_tokens.")

    metrics = _build_metrics(reasoning_buf, content_buf, usage, timings)

    if show_metrics:
        _print_metrics(metrics)

    return {
        "reasoning": reasoning_buf,
        "content": content_buf,
        "finish_reason": finish_reason,
        "usage": usage,
        "timings": timings,
        "metrics": metrics,
        "raw": response if not stream else None,
    }


def _build_metrics(reasoning_buf, content_buf, usage, timings):
    """Calcula métricas de tokens y velocidad. La división razonamiento/respuesta
    es una ESTIMACIÓN proporcional por caracteres, ya que la API no separa
    completion_tokens_details por ahora (viene como None)."""
    metrics = {
        "prompt_tokens": None,
        "completion_tokens": None,
        "cached_tokens": None,
        "reasoning_tokens_est": None,
        "response_tokens_est": None,
        "prompt_tokens_per_sec": None,
        "gen_tokens_per_sec": None,
        "draft_n": None,
        "draft_n_accepted": None,
        "draft_accept_rate": None,
    }

    if usage:
        metrics["prompt_tokens"] = usage.prompt_tokens
        metrics["completion_tokens"] = usage.completion_tokens
        cached = getattr(usage.prompt_tokens_details, "cached_tokens", None) if usage.prompt_tokens_details else None
        metrics["cached_tokens"] = cached

        total_chars = len(reasoning_buf) + len(content_buf)
        if total_chars > 0 and usage.completion_tokens:
            reasoning_ratio = len(reasoning_buf) / total_chars
            metrics["reasoning_tokens_est"] = round(usage.completion_tokens * reasoning_ratio)
            metrics["response_tokens_est"] = usage.completion_tokens - metrics["reasoning_tokens_est"]

    if timings:
        metrics["prompt_tokens_per_sec"] = timings.get("prompt_per_second")
        metrics["gen_tokens_per_sec"] = timings.get("predicted_per_second")
        metrics["draft_n"] = timings.get("draft_n")
        metrics["draft_n_accepted"] = timings.get("draft_n_accepted")
        if timings.get("draft_n"):
            metrics["draft_accept_rate"] = round(
                100 * timings.get("draft_n_accepted", 0) / timings["draft_n"], 1
            )

    return metrics


def _print_metrics(m):
    print("\n" + "=" * 50)
    print("📊 MÉTRICAS")
    print("=" * 50)
    print(f"  Tokens prompt (entrada):     {m['prompt_tokens']}")
    print(f"  Tokens prompt cacheados:     {m['cached_tokens']}")
    print(f"  Tokens completion (total):   {m['completion_tokens']}")
    print(f"    ├─ Razonamiento (est.):    {m['reasoning_tokens_est']}")
    print(f"    └─ Respuesta (est.):       {m['response_tokens_est']}")
    print(f"  Velocidad prompt:            {m['prompt_tokens_per_sec']:.1f} tok/s" if m['prompt_tokens_per_sec'] else "  Velocidad prompt:            N/A")
    print(f"  Velocidad generación:        {m['gen_tokens_per_sec']:.1f} tok/s" if m['gen_tokens_per_sec'] else "  Velocidad generación:        N/A")
    if m['draft_n'] is not None:
        print(f"  Draft tokens propuestos:     {m['draft_n']}")
        print(f"  Draft tokens aceptados:      {m['draft_n_accepted']}  ({m['draft_accept_rate']}% acierto MTP)")
    print("=" * 50)

q = """
Genera una conversación larga y realista entre un usuario y un asistente de IA, en español.

TEMA: [describe aquí el tema/dominio de la conversación, ej: soporte técnico, tutoría de matemáticas, asistente de cocina, etc.]

FORMATO DE SALIDA (obligatorio):
- Responde ÚNICAMENTE con un array JSON válido. Nada de texto antes o después, nada de explicaciones, nada de markdown, nada de ```json.
- El array debe tener exactamente 500 elementos.
- Cada elemento es un objeto con esta estructura exacta:
  {
    "index": <entero, empieza en 0 y sube de 1 en 1>,
    "role": "user" | "assistant",
    "content": "<texto en español>"
  }
- Los roles deben alternar estrictamente: index par = "user", index impar = "assistant" (o el patrón que definas).
- La conversación debe tener coherencia temática de principio a fin: el asistente debe recordar y referenciar cosas mencionadas antes por el usuario, y el usuario debe hacer preguntas de seguimiento naturales.
- Varía la longitud y el estilo de los mensajes (algunos cortos, algunos más elaborados) para que se sienta natural, no repetitivo.
- No repitas la misma pregunta o respuesta con palabras distintas más de una vez.

REGLAS DE VALIDEZ JSON (muy importante):
- Usa comillas dobles para todas las claves y strings.
- Escapa correctamente comillas internas, saltos de línea (\\n) y backslashes dentro de "content".
- No dejes comas finales (trailing commas) después del último elemento de un objeto o array.
- No incluyas comentarios dentro del JSON.
- Verifica mentalmente que el JSON cierre correctamente todos los corchetes y llaves antes de terminar tu respuesta.

Genera los 500 elementos completos, sin cortar la respuesta a la mitad.
"""

result = chat(
    q,
    enable_thinking=True,
    stream=True,
    max_tokens=110000,
    show_metrics=True,
)

R.
...

==================================================
📊 MÉTRICAS
==================================================
  Tokens prompt (entrada):     145
  Tokens prompt cacheados:     12
  Tokens completion (total):   4454
    ├─ Razonamiento (est.):    3242
    └─ Respuesta (est.):       1212
  Velocidad prompt:            2048.6 tok/s
  Velocidad generación:        273.8 tok/s
  Draft tokens propuestos:     5172
  Draft tokens aceptados:      3163  (61.2% acierto MTP)
==================================================
'''

# ---

q= """
escribe una conversacion tipo usuario y asistente en formato lista de json sin errores y buen formato, la lista debe de contener 500 elementos y solo responde en formato de lista de json.

[
  {
    "index": 0,
    "role": "user",
    "content": "Esto es una pregunta o consulta de un usuario"
  },
  {
    "index": 1,
    "role": "assistant",
    "content": "respuesta que da el assitant al usuario"
  },
  ...
]
"""
result = chat(
    q,
    enable_thinking=True,
    stream=True,
    max_tokens=110000,
    show_metrics=True,
)

R.
...

==================================================
📊 MÉTRICAS
==================================================
  Tokens prompt (entrada):     147
  Tokens prompt cacheados:     12
  Tokens completion (total):   26007
    ├─ Razonamiento (est.):    7839
    └─ Respuesta (est.):       18168
  Velocidad prompt:            2055.0 tok/s
  Velocidad generación:        292.4 tok/s
  Draft tokens propuestos:     27080
  Draft tokens aceptados:      19238  (71.0% acierto MTP)
==================================================

Limitations and notes

• The MTP block was grafted, not retrained, on top of the merged weights; the acceptance rate (~42–48%) reflects that partial misalignment between the draft block and the main model after the merge.
• Quants are generated directly from F16 with no double quantization, but haven't been exhaustively evaluated on standard benchmarks (MMLU, HumanEval, etc.) — that evaluation applies to the unquantized base model.
• --spec-type draft-mtp requires an llama.cpp build with MTP support; verify your build includes this flag before reporting issues.
• If you graft the MTP sidecar onto a different quant.

---

License

Apache 2.0 (inherited from the base model). Check the individual licenses of the merge's source models (Ornith-Agents-A1-3.6-35B-A3B-dare_ties) for commercial use.