AlexWortega

/

SIQ-1-35B

like 95

firxed

the upload didnt seems to be done ?

the mtp file is partial? could you upload a complete version ?

fixed

0.15.043.289 E llama_model_load: error loading model: error loading model hyperparameters: key not found in model: qwen35moe.context_length
0.15.043.290 E llama_model_load_from_file_impl: failed to load model

Can you provide version

Can't replicate

models\SIQ-1-35B-AlexWortega\mtp-SIQ-1-35B.f16.gguf'
0.14.706.003 I srv operator(): operator(): cleaning up before exit...
0.14.706.793 E srv llama_server: exiting due to model loading error
\llama-b9741-bin-win-cuda-13.3-x64>

\llama-b9741-bin-win-cuda-13.3-x64>

i will try cuda 12

SIQ-1-35B-AlexWortega\mtp-SIQ-1-35B.f16.gguf'
0.32.893.395 E llama_model_load: error loading model: error loading model hyperparameters: key not found in model: qwen35moe.context_length
0.32.893.400 E llama_model_load_from_file_impl: failed to load model
0.32.893.402 E srv load_model: failed to load draft model, '..\models\SIQ-1-35B-AlexWortega\mtp-SIQ-1-35B.f16.gguf'
0.32.893.410 I srv operator(): operator(): cleaning up before exit...
0.32.894.244 E srv llama_server: exiting due to model loading error
J:\llama-b9775-bin-win-cuda-12.4-x64

the latest mtp-SIQ-1-35B.f16.gguf works, thanks!

How to merge mtp-SIQ-1-35B.f16.gguf (HuggingFace) with a standard Q8/Q4 GGUF trunk.

Output: A combined GGUF with 41 blocks, MTP speculative decoding enabled.

Prerequisites

• Standard llama.cpp build (no ROCmFP4 needed)
• SIQ-1-35B trunk GGUF (Q8_0, Q4_0, Q4_K_M, or any quantization)
• The HF MTP companion: mtp-SIQ-1-35B.f16.gguf
• Python 3 with numpy and the gguf package

pip install gguf numpy

Step 1: Download the MTP companion

cd /tmp
curl -LO https://huggingface.co/AlexWortega/SIQ-1-35B/resolve/main/gguf/mtp-SIQ-1-35B.f16.gguf

Verify: 19 tensors, 1.6 GB.

python3 -c "
from gguf.gguf_reader import GGUFReader
r = GGUFReader('/tmp/mtp-SIQ-1-35B.f16.gguf')
print(f'Tensors: {len(r.tensors)}, Size: {sum(len(bytes(t.data)) for t in r.tensors)/1e9:.2f} GB')
"

Step 2: Fix the norm weight bug

The HF file has a −1.0 offset on all 7 norm tensors compared to correct Qwen3.6 weights.
This corrupts RMS norm scaling and causes near-zero MTP acceptance. Must fix.

# fix_norms.py
import numpy as np
from gguf.gguf_reader import GGUFReader
from gguf.gguf_writer import GGUFWriter
from gguf.constants import GGMLQuantizationType

MTP_IN   = '/tmp/mtp-SIQ-1-35B.f16.gguf'
MTP_FIXED = '/tmp/mtp-SIQ-1-35B-fixed.f16.gguf'

src = GGUFReader(MTP_IN)

tensors = []
for t in src.tensors:
    ne   = [int(s) for s in t.shape]
    data = bytes(t.data)
    if 'norm' in t.name:
        arr  = np.frombuffer(data, dtype=np.float16).astype(np.float32)
        arr += 1.0  # undo the −1.0 corruption
        data = arr.astype(np.float16).tobytes()
        print(f'  Fixed: {t.name}')
    tensors.append((t.name, ne, t.tensor_type, data))

# Copy metadata from a reference model (or hardcode SIQ defaults)
w = GGUFWriter(MTP_FIXED, "qwen35moe")
w.add_name("SIQ-1-35B MTP companion (fixed)")
w.add_block_count(41)
w.add_context_length(262144)
w.add_embedding_length(2048)
w.add_head_count(16); w.add_head_count_kv(2)
w.add_key_length(256); w.add_value_length(256)
w.add_layer_norm_rms_eps(1e-6)
w.add_expert_count(256); w.add_expert_used_count(8)
w.add_expert_feed_forward_length(512); w.add_expert_shared_feed_forward_length(512)
w.add_vocab_size(248320)
w.add_full_attention_interval(4)
w.add_nextn_predict_layers(1)
w.add_file_type(1)
w.add_rope_dimension_count(64)
w.add_rope_dimension_sections([11, 11, 10, 0])
w.add_rope_freq_base(10000000.0)
w.add_ssm_conv_kernel(4)
w.add_ssm_state_size(128)
w.add_ssm_group_count(16)
w.add_ssm_time_step_rank(32)
w.add_ssm_inner_size(4096)
w.add_quantization_version(2)
w.add_type("model"); w.add_size_label("35B")
w.add_tokenizer_model("gpt2"); w.add_tokenizer_pre("default")
w.add_bos_token_id(248044); w.add_eos_token_id(248046); w.add_pad_token_id(248044)

for name, ne, qt, data in tensors:
    w.add_tensor_info(name, list(reversed(ne)), np.float16, len(data), GGMLQuantizationType.F16)

w.write_header_to_file()
w.write_kv_data_to_file()
w.write_ti_data_to_file()
for _, _, _, data in tensors:
    w.write_tensor_data(np.frombuffer(data, dtype=np.uint8))
w.close()
print(f'Saved: {MTP_FIXED}')

Step 3: Merge trunk + MTP into a combined GGUF

Set TRUNK to your existing SIQ GGUF (Q8_0, Q4_0, Q4_K_M, etc.).

# merge.py
import os, numpy as np
from gguf.gguf_reader import GGUFReader
from gguf.gguf_writer import GGUFWriter
from gguf.constants import GGML_QUANT_SIZES, GGMLQuantizationType

# --- Config: change these paths ---
TRUNK   = os.path.expanduser('~/model/llm/SIQ-1-35B/SIQ-1-35B-Q8_0.gguf')
MTP_SRC = '/tmp/mtp-SIQ-1-35B-fixed.f16.gguf'
OUT     = 'SIQ-1-35B-mtp-combined.gguf'

trunk = GGUFReader(TRUNK)
mtp   = GGUFReader(MTP_SRC)

# Collect all trunk metadata to copy forward
def get_u32(r, k):
    f = r.fields.get(k)
    return int([p for p in reversed(f.parts) if hasattr(p,'item')][0].item()) if f else None

def get_f32(r, k):
    f = r.fields.get(k)
    return float([p for p in reversed(f.parts) if hasattr(p,'item')][0].item()) if f else None

def extract_strings(r, key):
    """Extract tokenizer token/merge lists from GGUF metadata."""
    f = r.fields.get(key)
    if not f: return []
    res = []
    for p in f.parts:
        if hasattr(p, 'dtype') and p.dtype == np.uint8:
            try:
                s = bytes(p).decode('utf-8')
                if len(s) > 0 and s != key: res.append(s)
            except: pass
    return res

tokens = extract_strings(trunk, 'tokenizer.ggml.tokens')
merges = extract_strings(trunk, 'tokenizer.ggml.merges')

# Read file_type from trunk
file_type = get_u32(trunk, 'general.file_type') or 1

# Build combined GGUF
w = GGUFWriter(OUT, "qwen35moe")

# Copy all metadata from trunk, override block_count → 41
for key, field in trunk.fields.items():
    if key.startswith('GGUF.'): continue
    parts = field.parts
    types = field.types
    last  = parts[-1]
    t     = types[-1] if len(types) == 1 else types[0]

    # Override block_count and add nextn
    if key == 'qwen35moe.block_count':
        w.add_uint32(key, 41)
        w.add_uint32('qwen35moe.nextn_predict_layers', 1)
        continue
    if key == 'qwen35moe.nextn_predict_layers':
        continue  # we set it above

    # Simple scalar types
    if hasattr(last, 'item') and hasattr(last, 'dtype'):
        val = last.item()
        if last.dtype == np.float32:
            w.add_float32(key, float(val))
        elif last.dtype in (np.int32, np.uint32):
            w.add_uint32(key, int(val))
        elif last.dtype == np.uint64:
            w.add_uint64(key, int(val))
        elif last.dtype == np.float64:
            w.add_float64(key, float(val))
        else:
            pass  # skip unknown scalar types
        continue

    # String type
    if t.value == 8:  # STRING
        for p in parts:
            if hasattr(p, 'dtype') and p.dtype == np.uint8 and len(p) > 2:
                val = bytes(p).decode('utf-8')
                if val != key:
                    w.add_string(key, val)
                    break
        continue

    # Array types
    if t.value == 9:  # ARRAY
        etype = types[1].value if len(types) > 1 else None
        if etype == 8:  # STRING array
            arr = extract_strings(trunk, key)
            if key.endswith('.tokens'): w.add_token_list(arr)
            elif key.endswith('.merges'): w.add_token_merges(arr)
            else:
                for s in arr: w.add_string(key, s)
            continue
        if etype in (5, 4):  # INT32/UINT32 array
            for p in parts:
                if hasattr(p, 'dtype') and p.dtype in (np.int32, np.uint32) and len(p) > 1:
                    vals = [int(x) for x in p.tolist()]
                    for v in vals: w.add_int32(key, v)
                    break
            continue

# Collect all tensors
all_tensors = []
for t in trunk.tensors:
    all_tensors.append((t.name, [int(s) for s in t.shape], t.tensor_type, bytes(t.data)))
for t in mtp.tensors:
    all_tensors.append((t.name, [int(s) for s in t.shape], t.tensor_type, bytes(t.data)))

for name, ggml_ne, qt, data in all_tensors:
    nb = len(data)
    if qt == GGMLQuantizationType.F32:
        w.add_tensor_info(name, list(reversed(ggml_ne)), np.float32, nb, None)
    elif qt == GGMLQuantizationType.F16:
        w.add_tensor_info(name, list(reversed(ggml_ne)), np.float16, nb, None)
    else:
        bs, ts = GGML_QUANT_SIZES[qt]
        pt = list(reversed(ggml_ne))
        pt[-1] = pt[-1] // bs * ts
        w.add_tensor_info(name, pt, np.uint8, nb, qt)

w.write_header_to_file()
w.write_kv_data_to_file()
w.write_ti_data_to_file()
for _, _, _, d in all_tensors:
    w.write_tensor_data(np.frombuffer(d, dtype=np.uint8))
w.close()

print(f'Done: {os.path.getsize(OUT)/1024**3:.2f} GB ({len(all_tensors)} tensors)')

Step 4: Verify

python3 -c "
from gguf.gguf_reader import GGUFReader
r = GGUFReader('SIQ-1-35B-mtp-combined.gguf')
print(f'Tensors: {len(r.tensors)}')  # trunk tensors + 19

blocks = sorted(set(int(t.name.split('.')[1]) for t in r.tensors if t.name.startswith('blk.')))
print(f'Blocks: {min(blocks)}..{max(blocks)}  ({len(blocks)} blocks)')  # 0..40

# Verify block_count and nextn
for k in ['qwen35moe.block_count', 'qwen35moe.nextn_predict_layers']:
    f = r.fields.get(k)
    for p in reversed(f.parts):
        if hasattr(p, 'item'): print(f'{k}: {p.item()}'); break
"

Step 5: Run with MTP

llama-cli \
  -m SIQ-1-35B-mtp-combined.gguf \
  -ngl 99 -c 4096 -fa on \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  --jinja -f prompt.txt -n 128

Critical Pitfalls

SIQ-1-35B Metadata Reference

These values are hardcoded in the fix script (Step 2). If your trunk uses different values,
read them from the trunk GGUF and use those instead.

block_count              = 41       (40 trunk + 1 MTP)
context_length           = 262144
embedding_length         = 2048
attention.head_count     = 16
attention.head_count_kv  = 2
attention.key_length     = 256
attention.value_length   = 256
expert_count             = 256
expert_used_count        = 8
expert_feed_forward_length = 512
vocab_size               = 248320
full_attention_interval  = 4
rope.dimension_count     = 64
rope.dimension_sections  = [11, 11, 10, 0]
rope.freq_base           = 10000000.0
ssm.conv_kernel          = 4
ssm.state_size           = 128
ssm.group_count          = 16
ssm.time_step_rank       = 32
ssm.inner_size           = 4096
tokenizer.ggml.model     = gpt2
BOS = 248044, EOS = 248046, PAD = 248044

thx for answer, will care about it in future!