Duplicate from YTan2000/Qwen3.6-35B-A3B-TQ3_4S

Co-authored-by: Y Tan <YTan2000@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+thumbnail.png filter=lfs diff=lfs merge=lfs -text
+mmproj-BF16.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3.6-35B-A3B-TQ3_4S.gguf filter=lfs diff=lfs merge=lfs -text

Qwen3.6-35B-A3B-TQ3_4S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6af1c2df5cdeac6975079a6e129bc2043f7bbc0f0ac72125a7572016b452216e
+size 13298875360

README.md

@@ -0,0 +1,128 @@
+---
+license: apache-2.0
+base_model:
+  - Qwen/Qwen3.6-35B-A3B
+language:
+  - en
+tags:
+  - GGUF
+  - llama.cpp
+  - qwen3.6
+  - qwen
+  - quantization
+  - turboquant
+  - tq3_4s
+  - multimodal
+  - Mixture of Experts
+  - conversational
+pipeline_tag: image-text-to-text
+---
+
+![thumbnail](thumbnail.png)
+
+# Qwen3.6-35B-A3B-TQ3_4S
+
+GGUF quantization of [`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) using **TQ3_4S** with mixed-precision MoE compression — 2-bit experts, 4-bit attention.
+
+## Files
+
+| File | Description |
+|------|-------------|
+| `Qwen3.6-35B-A3B-TQ3_4S.gguf` | Main model (12.4 GiB, 3.07 BPW) |
+| `mmproj-BF16.gguf` | Multimodal projector (BF16) |
+
+## Quantization
+
+MoE experts tolerate aggressive compression because only 8/256 are active per token. This quantization exploits that asymmetry:
+
+| Component | Quant | Rationale |
+|-----------|-------|-----------|
+| Expert MLP gate/up | Q2_K | 98% of params, MoE-tolerant |
+| Expert MLP down | Q3_K | Write-back sensitivity |
+| Attention Q/K/V/O | TQ3_4S | WHT-protected |
+| Embeddings + output | Q6_K | Quality anchor |
+
+## Runtime Requirement
+
+This model requires the public TurboQuant runtime fork:
+* https://github.com/turbo-tan/llama.cpp-tq3
+
+## Recommended Settings (16GB VRAM)
+
+```bash
+./build/bin/llama-server \
+  -m Qwen3.6-35B-A3B-TQ3_4S.gguf \
+  -ngl 99 -c 4096 -np 1 \
+  -ctk q4_0 -ctv tq3_0 -fa on \
+  --jinja \
+  --reasoning off --reasoning-budget 0 --reasoning-format deepseek
+```
+
+With vision:
+
+```bash
+./build/bin/llama-server \
+  -m Qwen3.6-35B-A3B-TQ3_4S.gguf \
+  --mmproj mmproj-BF16.gguf \
+  -ngl 99 -c 4096 -np 1 \
+  -ctk q4_0 -ctv tq3_0 -fa on \
+  --jinja --no-mmproj-offload \
+  --reasoning off --reasoning-budget 0 --reasoning-format deepseek
+```
+
+## Performance (RTX 5060 Ti 16GB)
+
+| Metric | Value |
+|--------|------:|
+| PP512 | 1832 tok/s |
+| TG128 | 107 tok/s |
+| Size | 12.4 GiB |
+| BPW | 3.07 |
+| ngl | 99 (full GPU) |
+
+Fits entirely in 16GB VRAM — no CPU offload needed.
+
+## Quality
+
+10/10 correct on standard QA benchmark (capital of France, 2+2, Python reverse string, gravity, WW2, primes, boiling point, Shakespeare, Jupiter, hello→Hola).
+
+## Base Model
+
+* [`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)
+* Source: [`unsloth/Qwen3.6-35B-A3B-GGUF`](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF) (Q8_0)
+
+## License
+
+Apache 2.0 — same as the base model.
+
+## Tool Call Validation
+
+Tested with `--jinja` on both `--reasoning off` and `--reasoning on --reasoning-budget 2048`:
+
+| Test | reasoning off | reasoning on |
+|------|:---:|:---:|
+| Basic tool call trigger | ✅ | ✅ |
+| Tool response → final answer (no loop) | ✅ | ✅ |
+| Correct tool selection from multiple | ✅ | ✅ |
+| No tool call for simple questions | ✅ | ✅ |
+| Multi-step tool use | ✅ | ✅ |
+| Nested quote escaping retry (no loop) | ✅ | ✅ |
+| **Total** | **10/10** | **10/10** |
+
+### Recommended settings for tool-use / agentic workflows
+
+```bash
+--jinja --reasoning off --reasoning-budget 0 --reasoning-format deepseek
+```
+
+Avoid `--presence-penalty` above 0.5 for tool-use — high values diversify reasoning tokens but don't improve structured JSON output, and can cause repeated near-identical tool calls in agent loops.
+
+If using `--reasoning on`, ensure your agent framework detects consecutive identical tool calls and breaks after 2-3 retries.
+
+### Run tests yourself
+
+```bash
+chmod +x test_tool_calls.sh
+./test_tool_calls.sh 8085
+```
+

mmproj-BF16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:356dfaa3111376a4f7165e32e8749713378d1700b37cf52e0c50d9f23322334d
+size 902822624

test_tool_calls.sh

@@ -0,0 +1,133 @@
+#!/bin/bash
+# Comprehensive tool call test for Qwen3.6-35B-A3B
+# Usage: ./test_tool_calls.sh [port] 
+# Default port: 8085
+
+PORT="${1:-8085}"
+BASE="http://localhost:$PORT/v1/chat/completions"
+PASS=0
+FAIL=0
+
+check() {
+    local name="$1" expected="$2" actual="$3"
+    if echo "$actual" | grep -q "$expected"; then
+        echo "  ✅ $name"
+        ((PASS++))
+    else
+        echo "  ❌ $name (expected '$expected', got: $actual)"
+        ((FAIL++))
+    fi
+}
+
+echo "=== Tool Call Tests ==="
+echo "Server: http://localhost:$PORT"
+curl -s "http://localhost:$PORT/health" | grep -q ok || { echo "❌ Server not running"; exit 1; }
+echo ""
+
+# Test 1: Basic tool call
+echo "Test 1: Basic tool call trigger"
+R=$(curl -s --max-time 60 "$BASE" -H "Content-Type: application/json" -d '{
+  "messages": [{"role": "user", "content": "What is the weather in London?"}],
+  "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}}],
+  "max_tokens": 256, "temperature": 0
+}')
+TCALLS=$(echo "$R" | python3 -c "import sys,json;d=json.load(sys.stdin);tc=d['choices'][0]['message'].get('tool_calls');print(tc[0]['function']['name'] if tc else 'none')" 2>/dev/null)
+FINISH=$(echo "$R" | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['choices'][0]['finish_reason'])" 2>/dev/null)
+check "triggers tool call" "get_weather" "$TCALLS"
+check "finish_reason=tool_calls" "tool_calls" "$FINISH"
+
+# Test 2: Tool response → final answer (no loop)
+echo ""
+echo "Test 2: Tool response produces final answer (no loop)"
+R=$(curl -s --max-time 60 "$BASE" -H "Content-Type: application/json" -d '{
+  "messages": [
+    {"role": "user", "content": "What is the weather in London?"},
+    {"role": "assistant", "content": "", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"location\":\"London\"}"}}]},
+    {"role": "tool", "tool_call_id": "call_1", "content": "{\"temperature\": 15, \"condition\": \"cloudy\"}"}
+  ],
+  "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}}],
+  "max_tokens": 256, "temperature": 0
+}')
+CONTENT=$(echo "$R" | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['choices'][0]['message'].get('content',''))" 2>/dev/null)
+TCALLS=$(echo "$R" | python3 -c "import sys,json;d=json.load(sys.stdin);tc=d['choices'][0]['message'].get('tool_calls');print('has_calls' if tc else 'no_calls')" 2>/dev/null)
+FINISH=$(echo "$R" | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['choices'][0]['finish_reason'])" 2>/dev/null)
+check "has content" "London\|15\|cloudy" "$CONTENT"
+check "no further tool calls" "no_calls" "$TCALLS"
+check "finish_reason=stop" "stop" "$FINISH"
+
+# Test 3: Correct tool selection
+echo ""
+echo "Test 3: Selects correct tool from multiple"
+R=$(curl -s --max-time 60 "$BASE" -H "Content-Type: application/json" -d '{
+  "messages": [{"role": "user", "content": "Search for latest AI news"}],
+  "tools": [
+    {"type": "function", "function": {"name": "get_weather", "description": "Get weather for a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}},
+    {"type": "function", "function": {"name": "search_web", "description": "Search the web", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}
+  ],
+  "max_tokens": 256, "temperature": 0
+}')
+TCALLS=$(echo "$R" | python3 -c "import sys,json;d=json.load(sys.stdin);tc=d['choices'][0]['message'].get('tool_calls');print(tc[0]['function']['name'] if tc else 'none')" 2>/dev/null)
+check "picks search_web not get_weather" "search_web" "$TCALLS"
+
+# Test 4: No tool call when not needed
+echo ""
+echo "Test 4: No tool call for simple question"
+R=$(curl -s --max-time 60 "$BASE" -H "Content-Type: application/json" -d '{
+  "messages": [{"role": "user", "content": "What is 2+2?"}],
+  "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}}],
+  "max_tokens": 256, "temperature": 0
+}')
+CONTENT=$(echo "$R" | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['choices'][0]['message'].get('content',''))" 2>/dev/null)
+TCALLS=$(echo "$R" | python3 -c "import sys,json;d=json.load(sys.stdin);tc=d['choices'][0]['message'].get('tool_calls');print('has_calls' if tc else 'no_calls')" 2>/dev/null)
+check "answers directly with 4" "4" "$CONTENT"
+check "no tool call" "no_calls" "$TCALLS"
+
+# Test 5: Multi-step tool use
+echo ""
+echo "Test 5: Multi-step (2 cities)"
+R=$(curl -s --max-time 60 "$BASE" -H "Content-Type: application/json" -d '{
+  "messages": [{"role": "user", "content": "Compare weather in London and Paris"}],
+  "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get weather for a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}}],
+  "max_tokens": 256, "temperature": 0
+}')
+TCALLS=$(echo "$R" | python3 -c "import sys,json;d=json.load(sys.stdin);tc=d['choices'][0]['message'].get('tool_calls',[]);print(len(tc))" 2>/dev/null)
+check "calls tool (1 or 2 calls)" "[12]" "$TCALLS"
+
+# Summary
+echo ""
+echo "=== Results: $PASS passed, $FAIL failed ==="
+
+# Test 6: Nested quote escaping (stress test)
+echo ""
+echo "Test 6: Nested bash quote escaping (3 rounds)"
+TOOLS_T='[{"type":"function","function":{"name":"terminal","description":"Execute a bash command","parameters":{"type":"object","properties":{"command":{"type":"string"}},"required":["command"]}}}]'
+
+R1=$(curl -s --max-time 120 "$BASE" -H "Content-Type: application/json" -d "{
+  \"messages\":[{\"role\":\"user\",\"content\":\"Run: bash ~/script/proxy.sh \\\"web read --url \\\\\\\"https://example.com/\\\\\\\"\\\"\"}],
+  \"tools\":$TOOLS_T, \"max_tokens\":512, \"temperature\":1.0, \"top_p\":0.95, \"top_k\":20, \"presence_penalty\":1.5
+}")
+CMD1=$(echo "$R1" | python3 -c "import sys,json;d=json.load(sys.stdin);tc=d['choices'][0]['message'].get('tool_calls',[]);print(tc[0]['function']['arguments'] if tc else 'no_call')" 2>/dev/null)
+
+R2=$(curl -s --max-time 120 "$BASE" -H "Content-Type: application/json" -d "{
+  \"messages\":[
+    {\"role\":\"user\",\"content\":\"Run: bash ~/script/proxy.sh \\\"web read --url \\\\\\\"https://example.com/\\\\\\\"\\\"\"},
+    {\"role\":\"assistant\",\"content\":\"\",\"tool_calls\":[{\"id\":\"c1\",\"type\":\"function\",\"function\":{\"name\":\"terminal\",\"arguments\":$CMD1}}]},
+    {\"role\":\"tool\",\"tool_call_id\":\"c1\",\"content\":\"{\\\"output\\\":\\\"bash: unexpected EOF\\\\nSTATUS:FAILURE\\\",\\\"exit_code\\\":1}\"}
+  ],
+  \"tools\":$TOOLS_T, \"max_tokens\":512, \"temperature\":1.0, \"top_p\":0.95, \"top_k\":20, \"presence_penalty\":1.5
+}")
+CMD2=$(echo "$R2" | python3 -c "import sys,json;d=json.load(sys.stdin);tc=d['choices'][0]['message'].get('tool_calls',[]);print(tc[0]['function']['arguments'] if tc else 'gave_up')" 2>/dev/null)
+
+if [ "$CMD1" = "$CMD2" ]; then
+    echo "  ⚠️  identical commands (potential loop)"
+    ((FAIL++))
+else
+    echo "  ✅ commands differ across retries (no loop)"
+    ((PASS++))
+fi
+echo "    R1: $CMD1"
+echo "    R2: $CMD2"
+
+# Summary
+echo ""
+echo "=== Results: $PASS passed, $FAIL failed ==="