AGENTIC_VALIDATION/README.md

Qwen3.8-27B IQ1_M agentic and tool-call validation

This directory preserves both the original failed Ollama result and the repaired controller-plus-model result. The evidence is deliberately additive: the old 0/6 report remains relevant to raw Ollama compatibility, while the new validation shows a reproducible way to obtain native structured tool_calls from the same model file.

Result

Path	Result	Meaning
Ollama 0.32.14 OpenAI-compatible tool interface	0/6	The intended calls appeared as ordinary assistant text and tool_calls remained null.
Direct llama.cpp with one selected tool and tool_choice="required"	1/1	Native structured tool_calls serialization works.
Direct llama.cpp with three available tools and tool_choice="required"	0/1	Raw IQ1_M multi-tool selection repeated argument text and reached the token limit without a tool call.
Adapter with three available tools and tool_choice="auto"	6/6	Weather, calculator and note requests returned the correct function and exact JSON arguments.
Adapter with OpenAI named tool_choice object	1/1	The object form was normalized and returned weather({"city":"Berlin"}) as a native tool call.

The repaired path therefore passes the tested explicit tool-use cases. It does not turn the raw quant into an unrestricted semantic router: the adapter contributes deterministic tool selection, then asks the model runtime to serialize the chosen call.

Root cause

The investigation separated three issues that the original 0/6 score had bundled together:

1. The tested llama.cpp build expects tool_choice as a string. It rejects the OpenAI named-object form when sent directly.
2. The IQ1_M quant can emit a proper structured call when one selected tool is presented.
3. Raw selection across several tools is unreliable at this quantization level and runtime configuration.

ADAPTER.py repairs the integration boundary rather than changing or requantizing the model.

Adapter behaviour

The adapter is a small standard-library OpenAI-compatible proxy. It:

• defaults to loopback-only binding;
• accepts the OpenAI named tool_choice object and normalizes it;
• scores explicit user wording against tool names, descriptions and parameter names;
• presents only the selected tool to llama.cpp with tool_choice="required";
• preserves llama.cpp's native tool_calls response;
• can retry once when raw model content identifies exactly one available function;
• contains no credentials, remote-control features or external network dependencies.

For ambiguous requests where no tool clearly wins, the adapter does not invent a selection. A stronger semantic router can replace the scoring function without changing the downstream serialization path.

Reproduce

Requirements:

• the repository's Qwen3.8-27B-IQ1_M.gguf file;
• a llama.cpp llama-server build compatible with Qwen3.8 and Jinja tool templates;
• Python 3.10 or newer;
• curl for the launcher health checks.

Start the local runtime and adapter:

chmod +x AGENTIC_VALIDATION/RUN_AGENTIC_ADAPTER.sh
LLAMA_SERVER=/path/to/llama-server \
  AGENTIC_VALIDATION/RUN_AGENTIC_ADAPTER.sh

The default OpenAI-compatible endpoint is:

http://127.0.0.1:18540/v1/chat/completions

Send the included request:

curl -sS http://127.0.0.1:18540/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --data-binary @AGENTIC_VALIDATION/REQUEST_EXAMPLE.json

The response should have finish_reason: "tool_calls" and a weather call containing {"city":"Tokyo"}.

Validation configuration

• Model: Qwen3.8-27B-IQ1_M.gguf
• Model SHA-256: 131cdf5c1c4b547081543382b00434e9ebf3f8eb369ef3714550086074f80bdf
• Runtime: llama.cpp b9591-62061f910
• Chat template: Jinja, peg-native
• Reasoning: disabled, budget 0
• Context: 1,024 tokens
• Temperature: 0
• Adapter bind: loopback only
• Validation host: CPU-only GETH

The CPU-only timing was slow: the six cases took 1,019.264 seconds in total, with individual cases between 150.487 and 194.198 seconds. Those timings demonstrate correctness under constrained hardware, not production throughput. The earlier RTX 3090 ordinary-generation result remains the relevant acceleration measurement.

Files

• ADAPTER.py: OpenAI-compatible tool-selection and normalization proxy.
• RUN_AGENTIC_ADAPTER.sh: one-command local launcher for llama.cpp plus the adapter.
• REQUEST_EXAMPLE.json: two-tool example request.
• SUMMARY.json: concise machine-readable result and scope.
• ADAPTER_VALIDATION.json: full six-case results, arguments, timings and usage.
• NAMED_TOOL_CHOICE_VALIDATION.json: live OpenAI named-object compatibility result.
• RAW_RUNTIME_FINDINGS.json: direct runtime diagnosis and preserved Ollama result.
• CHECKSUMS.sha256: hashes for every file in this directory.

Interpretation and limits

The useful conclusion is narrower than “the 1-bit model is fully agentic” and much stronger than “the quant cannot use tools.” The exact model can serialize valid structured calls, but its raw multi-tool routing is brittle. A lightweight controller that chooses one explicit tool produces reliable structured output in the tested cases.

The tests cover explicit weather, arithmetic and note requests. They do not establish performance on ambiguous instructions, large overlapping tool catalogues, adversarial schemas, multi-call planning or long tool-result continuations.

Publication hygiene

This directory is sanitized for public release. It contains no API keys, access tokens, passwords, private keys, authorization headers, private network addresses, environment dumps or user-specific data.