Honest thoughts, this is my new daily driver.

#9
by Mk2Oracle - opened

I usually don't leave these posts but given how fast-paced LLM developments are and the people also wanting a reason to try something new and most important that *IT WORKS I found myself pleasantly surprised using this model.

Background: I'm just yet another software developer. I own and heavily use the OpenAI Codex $200 subscription. I own a small server with RTX 3090 while trying to see if vulkan/ROCM will ever catch up in performance but I don't know much about that area. Why try local LLMs? Well... as a linux user, open source supporter, etc, etc. Why not? But mostly the fact that these models can continuously work on the background while you sleep researching or coding or troubleshooting is something that really interests me as a developer. I become more of a product owner/creator than just laying out all the tedious architecture/etc. Most of my life has been about creating solutions and trying to breakthrough issues. LLMs feels like an extension, like waking up realizing you have been missing an arm and now it's there and fully functional.

So yes, I do have a wide-range of usage for local LLMs and even use openrouter.ai to try out the new ones. I only dabble a bit.

Main usage so far: Web development, mainly creating wordpress plugins and tightening security.

Observations: This model is more concise and instruction tuned than the base model of Qwen 27B. It feels firmly-rooted in following instructions. It even knows when to stop itself and say, "oh, I'm looping, let me break out and see this differently"

Obviously, being 27B it's not going to be frontier-level. But, guess what? What most of us want from a model is generally to be instruction-tuned, and that it knows how to troubleshoot things. It needs concise, multi-step, pivotal troubleshooting. This model, for whatever reason, delivers that.

I gave it a couple of instructions, create wordpress plugin that supports custom CPTs where you can manage it with metaboxes for the really custom-driven one. And create widgets in the block editor I can pick from the CPT entries. More over, create template-parts that allows me to further customize the CPT entries in a flexibly and re-usable way. It should support desktop and mobile views.

And it delivered. Also, amusingly the model doesn't like PHP which left me chuckling because it was the language I started with back with PHP3.

Main CLI tool: Hermes TUI (previous opencode, openclaw, but those things are awful I'm sorry).

Anyhow, thought I would share even if it doesn't mean much.

Llama.cpp setup

Models:
ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf
mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf

-ngl auto
-c 114688
-ctk q8_0
-ctv q8_0
-n 8192
-np 1
--reasoning on
--reasoning-preserve
--reasoning-budget 2048
--reasoning-format deepseek
--jinja
--chat-template-kwargs '{"preserve_thinking":true}'
--temp 0.6
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 1.0
--repeat-penalty 1.0
--image-min-tokens 1024
--image-max-tokens 4096
--host 0.0.0.0
--port 8000
--alias assistant
--timeout 7200
--sse-ping-interval 15
--sleep-idle-seconds -1
--no-webui

BottleCapAI org

Thanks for sharing your experience and setup. Quantized kvcache ctk q_8 and ctv q_8
are super important options for long context reasoning and save memory.

I have an industrial passive cooling computer with AMD 7840HS + 780M igpu + 64GB RAM + cachyos + llama-server with Vulkan + Hermes Agent.

This model seems pretty nice. I tested it with a task in chat (create a one pager breakout game) and comparing to the original 27b, with mtp=4 I got a very decent speed on PP and decode.
I run the same task with two mtp settings (2 and 4).
--spec-type draft-mtp
--spec-draft-n-max 4 --> and 2 in an other test
--spec-draft-p-min 0.85
--kv-unified
--cache-type-k q8_0
--cache-type-v q8_0

On the X axis you can see the generated tokens, while the Y axis is the tg/s.
One line is one continuous n_decode run.
In overall, it beat unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL in speed and quality.
I also tried unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q4_K_XL . This one was painfully slow.

tg_plot

My limited experience is, this qwen3.6 27b just beats everything in my setup.
Of course, I will need to check the same performance parameters during marketing research tasks and other kind of projects. For coding, it seems awesome.

I have an industrial passive cooling computer with AMD 7840HS + 780M igpu + 64GB RAM + cachyos + llama-server with Vulkan + Hermes Agent.

This model seems pretty nice. I tested it with a task in chat (create a one pager breakout game) and comparing to the original 27b, with mtp=4 I got a very decent speed on PP and decode.
I run the same task with two mtp settings (2 and 4).
--spec-type draft-mtp
--spec-draft-n-max 4 --> and 2 in an other test
--spec-draft-p-min 0.85
--kv-unified
--cache-type-k q8_0
--cache-type-v q8_0

On the X axis you can see the generated tokens, while the Y axis is the tg/s.
One line is one continuous n_decode run.
In overall, it beat unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL in speed and quality.
I also tried unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q4_K_XL . This one was painfully slow.

tg_plot

My limited experience is, this qwen3.6 27b just beats everything in my setup.
Of course, I will need to check the same performance parameters during marketing research tasks and other kind of projects. For coding, it seems awesome.

I don't know much about the state of AMD GPUs but try turning on flash attention it can honestly make a huge difference. I basically have it on on my RTX 3090 because devstral would stall a lot.

I don't know much about the state of AMD GPUs but try turning on flash attention it can honestly make a huge difference. I basically have it on on my RTX 3090 because devstral would stall a lot.

Thank you for your suggestion, in fact, the integrated GPU like 780M suffers from low memory bandwidth (dual channel, DDR5, 5600MHz -> 70-90GB/sec maximum), so, these numbers are awesome :)
Normally a non-mtp dense version (qwen3.6 27b - q8-q4) could achieve 1-2 tg/s.

This is my actual parameters, any hints are welcome (please note, igpu is very different than a dedicated GPU):
example: --chat-template-kwargs '{"enable_thinking":false}' -> deprecated, instead will be used: --reasoning on / --reasoning off

      llama-server --port ${PORT} --host 0.0.0.0
      -hf bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M
      --temp 0.6
      --top-p 0.95
      --top-k 20
      --min-p 0.00
      --repeat-penalty 1.05
      --presence-penalty 1.0
      -np 1
      --ctx-size 262144
      --flash-attn on
      --n-gpu-layers 99
      -b 1024
      -ub 256
      --spec-type draft-mtp
      --spec-draft-n-max 4
      --spec-draft-p-min 0.85
      --chat-template-kwargs '{"enable_thinking":false}'
      --reasoning-format deepseek
      --kv-unified
      --cache-type-k q8_0
      --cache-type-v q8_0
      --cache-reuse 256
      --cache-ram 16384
      --ctx-checkpoints 16
      --checkpoint-min-step 1024
      --jinja
      --no-mmproj
      --no-ui
      -t 6
      --no-warmup
      --timeout 0
      --log-file /home/pongraczi/llama-thinkingcap-qwen3.6-27-q4-lv5-1.log
      -lv 5

As I hit this bug: https://github.com/NousResearch/hermes-agent/issues/20866 , I still running in debug mode, I need to find a solution to avoid this annoying http 400 error, which blocks the "flow" :)

As I hit this bug: https://github.com/NousResearch/hermes-agent/issues/20866 , I still running in debug mode, I need to find a solution to avoid this annoying http 400 error, which blocks the "flow" :)

Not exactly this bug, but very similar, at this moment in Kanban task.

41.09.953.109 D Parsed message: {"role":"assistant","content":"","tool_calls":[{"type":"function","function":{"name":"write_file","arguments":"{"path":"/home/pongraczi/Projektek/WMS_basic/.worktrees/t_e1f323ea/app/src/main/java/com/wm
s/basic/data/database/StockEntity.kt","content":"package com.wms.basic.data.database\n\nimport androidx.room.Entity\nimport androidx.room.PrimaryKey\n\n@Entity(tableName = \"stock\")\ndata class StockEntity(\n @PrimaryK
ey val id: Long = 0,\n val sku: String = \"\",\n val locationCode: String = \"\",\n val quantity: Int = 0\n)\n"}"}}]}
41.09.953.373 D srv operator(): http: streamed chunk: data: {"choices":[{"finish_reason":"tool_calls","index":0,"delta":{}}],"created":1785948005,"id":"chatcmpl-SJmIbCumCh4wpaRrZKXxk2u7kwQup9fY","model":"bottlecapai/ThinkingCap-Qwen3.
6-27B-GGUF:Q4_K_M","system_fingerprint":"b10216-06be260","object":"chat.completion.chunk"}

data: {"choices":[],"created":1785948005,"id":"chatcmpl-SJmIbCumCh4wpaRrZKXxk2u7kwQup9fY","model":"bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M","system_fingerprint":"b10216-06be260","object":"chat.completion.chunk","usage":{"completi
on_tokens":146,"prompt_tokens":46098,"total_tokens":46244,"prompt_tokens_details":{"cached_tokens":45804}},"timings":{"cache_n":45804,"prompt_n":294,"prompt_ms":14368.073,"prompt_per_token_ms":48.87099659863946,"prompt_per_second":20.462
034122460263,"predicted_n":146,"predicted_ms":34868.026,"predicted_per_token_ms":238.82209589041094,"predicted_per_second":4.187217251702176,"draft_n":117,"draft_n_accepted":113}}

41.09.953.376 D srv operator(): all results received, terminating stream
41.09.953.387 D srv operator(): http: streamed chunk: data: [DONE]

41.09.953.402 D srv operator(): http: stream ended
41.09.953.413 D res remove_waiti: remove task 1526 from waiting list. current waiting = 1 (before remove)
41.09.953.414 D srv stop: all tasks already finished, no need to cancel
41.42.566.511 D common_chat_templates_apply_jinja: using differential autoparser
41.42.566.514 D ESC[1mESC[38;5;126m=== Starting differential analysis ===
ESC[0m41.42.566.514 D ESC[1mESC[38;5;214mPhase 1: Reasoning analysis
ESC[0m41.42.567.714 D ESC[1mESC[38;5;214mPhase 2: Content analysis
ESC[0m41.42.568.674 D ESC[1mESC[38;5;214mPhase 3: Tool call analysis
ESC[0m41.42.569.926 D ESC[1mESC[38;5;214mPhase 3a: Function call analysis
ESC[0m41.42.570.560 D ESC[1mESC[38;5;214mPhase 3b: Argument analysis
ESC[0m41.42.572.928 D ESC[1mESC[38;5;214mPhase 3c: Call id analysis
ESC[0m41.42.573.853 D Template application failed:

While executing CallExpression at line 43, column 24 in source:
... not messages %}↵ {{- raise_exception('No messages provided.') }}↵{%- endif %...
^
Error: Jinja Exception: No messages provided.
41.42.573.960 D ESC[1mESC[38;5;214mdetect_user_start_marker: Template application failed, unsupported empty messages? trying complex variant
ESC[0m41.42.574.342 D
--- Reasoning & Content Structure ---
41.42.574.344 D user_msg_start: <|im_start|>user
41.42.574.344 D assistant_msg_start: <|im_start|>assistant
41.42.574.344 D reasoning_mode: TAG_BASED

As I hit this bug: https://github.com/NousResearch/hermes-agent/issues/20866 , I still running in debug mode, I need to find a solution to avoid this annoying http 400 error, which blocks the "flow" :)

>Not exactly this bug, but very similar, at this moment in Kanban task.
It seems with a modified chat template all http 400 issues I experienced are gone.
{%- set image_count = namespace(value=0) %}
{%- set video_count = namespace(value=0) %}
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
    {%- if content is string %}
        {{- content }}
    {%- elif content is iterable and content is not mapping %}
        {%- for item in content %}
            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
                {%- if is_system_content %}
                    {{- raise_exception('System message cannot contain images.') }}
                {%- endif %}
                {%- if do_vision_count %}
                    {%- set image_count.value = image_count.value + 1 %}
                {%- endif %}
                {%- if add_vision_id %}
                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
                {%- endif %}
                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
            {%- elif 'video' in item or item.type == 'video' %}
                {%- if is_system_content %}
                    {{- raise_exception('System message cannot contain videos.') }}
                {%- endif %}
                {%- if do_vision_count %}
                    {%- set video_count.value = video_count.value + 1 %}
                {%- endif %}
                {%- if add_vision_id %}
                    {{- 'Video ' ~ video_count.value ~ ': ' }}
                {%- endif %}
                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
            {%- elif 'text' in item %}
                {{- item.text }}
            {%- else %}
                {{- raise_exception('Unexpected item type in content.') }}
            {%- endif %}
        {%- endfor %}
    {%- elif content is none or content is undefined %}
        {{- '' }}
    {%- else %}
        {{- raise_exception('Unexpected content type.') }}
    {%- endif %}
{%- endmacro %}
{%- if not messages %}
    {{- '' }}
{%- endif %}
{%- if tools and tools is iterable and tools is not mapping %}
    {{- '<|im_start|>system\n' }}
    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>" }}
    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
    {%- if messages[0].role == 'system' %}
        {%- set content = render_content(messages[0].content, false, true)|trim %}
        {%- if content %}
            {{- '\n\n' + content }}
        {%- endif %}
    {%- endif %}
    {{- '<|im_end|>\n' }}
{%- else %}
    {%- if messages[0].role == 'system' %}
        {%- set content = render_content(messages[0].content, false, true)|trim %}
        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
    {%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
    {%- set index = (messages|length - 1) - loop.index0 %}
    {%- if ns.multi_step_tool and message.role == "user" %}
        {%- set content = render_content(message.content, false)|trim %}
        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
            {%- set ns.multi_step_tool = false %}
            {%- set ns.last_query_index = index %}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if ns.multi_step_tool %}
    {{- '' }}
{%- endif %}
{%- for message in messages %}
    {%- set content = render_content(message.content, true)|trim %}
    {%- if message.role == "system" %}
        {%- if not loop.first %}
            {{- '' }}
        {%- endif %}
    {%- elif message.role == "user" %}
        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
    {%- elif message.role == "assistant" %}
        {%- set reasoning_content = '' %}
        {%- if message.reasoning_content is string %}
            {%- set reasoning_content = message.reasoning_content %}
        {%- else %}
            {%- if '</think>' in content %}
                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
            {%- endif %}
        {%- endif %}
        {%- set reasoning_content = reasoning_content|trim %}
        {%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}
            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
        {%- else %}
            {{- '<|im_start|>' + message.role + '\n' + content }}
        {%- endif %}
        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
            {%- for tool_call in message.tool_calls %}
                {%- if tool_call.function is defined %}
                    {%- set tool_call = tool_call.function %}
                {%- endif %}
                {%- if loop.first %}
                    {%- if content|trim %}
                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
                    {%- else %}
                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
                    {%- endif %}
                {%- else %}
                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
                {%- endif %}
                {%- if tool_call.arguments is defined %}
                    {%- for args_name, args_value in tool_call.arguments|items %}
                        {{- '<parameter=' + args_name + '>\n' }}
                        {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
                        {{- args_value }}
                        {{- '\n</parameter>\n' }}
                    {%- endfor %}
                {%- endif %}
                {{- '</function>\n</tool_call>' }}
            {%- endfor %}
        {%- endif %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" %}
        {%- if loop.previtem and loop.previtem.role != "tool" %}
            {{- '<|im_start|>user' }}
        {%- endif %}
        {{- '\n<tool_response>\n' }}
        {{- content }}
        {{- '\n</tool_response>' }}
        {%- if not loop.last and loop.nextitem.role != "tool" %}
            {{- '<|im_end|>\n' }}
        {%- elif loop.last %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- else %}
        {{- raise_exception('Unexpected message role.') }}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
    {%- if enable_thinking is defined and enable_thinking is false %}
        {{- '<think>\n\n</think>\n\n' }}
    {%- else %}
        {{- '<think>\n' }}
    {%- endif %}
{%- endif %}

Sign up or log in to comment