Ternary-Bonsai-27B-Q2_0.gguf loads just fine

john@ai-debian:~/TND/1bit/llama.cpp$ ./build/bin/llama-server -ngl 1000 -c 8192 --no-mmap -fit off -m ~/TND/AI/Ternary-Bonsai-27B-Q2_0.gguf
0.00.020.108 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.020.109 I device_info:
0.00.066.540 I   - CUDA0   : NVIDIA GeForce RTX 3060 (12257 MiB, 12139 MiB free)
0.00.066.548 I   - CPU     : 12th Gen Intel(R) Core(TM) i5-12400F (64146 MiB, 64146 MiB free)
0.00.066.636 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.00.066.641 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.066.664 I srv          init: running without SSL
0.00.066.711 I srv          init: using 11 threads for HTTP server
0.00.066.820 I srv         start: binding port with default address family
0.00.067.991 I srv  llama_server: loading model
0.00.068.013 I srv    load_model: loading model '/home/john/TND/AI/Ternary-Bonsai-27B-Q2_0.gguf'
0.02.532.321 W llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.02.553.158 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.02.619.104 I srv    load_model: initializing slots, n_slots = 4
0.02.673.374 W srv    load_model: speculative decoding will use checkpoints
0.02.673.388 W common_speculative_init: no implementations specified for speculative decoding
0.02.673.390 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 8192
0.02.673.394 I slot   load_model: id  1 | task -1 | new slot, n_ctx = 8192
0.02.673.394 I slot   load_model: id  2 | task -1 | new slot, n_ctx = 8192
0.02.673.394 I slot   load_model: id  3 | task -1 | new slot, n_ctx = 8192
0.02.673.464 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
0.02.673.465 I srv    load_model: use `--cache-ram 0` to disable the prompt cache
0.02.673.465 I srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
0.02.673.465 I srv    load_model: context checkpoints enabled, max = 32, min spacing = 256
0.02.673.492 I srv          init: idle slots will be saved to prompt cache and cleared upon starting a new task
0.02.686.571 I init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
<think>
'
0.02.696.450 I srv          init: init: chat template, thinking = 1
0.02.696.526 I srv  llama_server: model loaded
0.02.696.532 I srv  llama_server: server is listening on http://127.0.0.1:8080
0.02.696.540 I srv  update_slots: all slots are idle

im guessing the g64 model isnt for cuda

Ternary-Bonsai-27B-Q2_0.gguf is the correct one to use right now.

Demo repo has more info, for ternary we still waiting for some backends to merge llama.cpp side then need to do a migration
https://github.com/PrismML-Eng/Bonsai-demo
See "Upstream Status for Ternary" section

It doesn't work for me either (Windows) :/

"G:**\llama-b9999-bin-win-hip-radeon-x64>llama-server -m G:****\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-Q2_0.gguf
0.00.080.913 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.342.941 I srv load_model: loading model 'G:****\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-Q2_0.gguf'
0.00.407.831 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
0.00.407.835 E gguf_init_from_reader: failed to read tensor data
0.00.414.951 E llama_model_load: error loading model: llama_model_loader: failed to load model from G:****\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-Q2_0.gguf
0.00.414.958 E llama_model_load_from_file_impl: failed to load model
0.00.414.998 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.489.008 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
0.00.489.014 E gguf_init_from_reader: failed to read tensor data
0.00.496.164 E llama_model_load: error loading model: llama_model_loader: failed to load model from G:****\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-Q2_0.gguf
0.00.496.172 E llama_model_load_from_file_impl: failed to load model
0.00.496.192 E cmn common_init_: failed to load model 'G:****\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-Q2_0.gguf'
0.00.496.196 E srv load_model: failed to load model, 'G:******\prism-ml\Ternary-Bonsai-27B-gguf\Ternary-Bonsai-27B-Q2_0.gguf'
0.00.496.223 I srv operator(): operator(): cleaning up before exit...
0.00.497.028
G:******\llama-b9999-bin-win-hip-radeon-x64>"

please post issues on github with details hardware/os/what is exactly being run, this is not easy to debug here
many things could be different
https://github.com/PrismML-Eng/Bonsai-demo

I built the newest llama.cpp with nix flakes with ROCm on a RX 6800 and got this going to try the prism fork

~/Projects/nixdots
❯ llama-server -ngl 99 -c 8192 -m ~/Downloads/Ternary-Bonsai-27B-Q2_0.gguf
0.00.014.543 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.014.699 W srv  llama_server: -----------------
0.00.014.701 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.014.701 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.014.701 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.014.701 W srv  llama_server: -----------------
0.00.015.810 I srv    load_model: loading model '/home/noah/Downloads/Ternary-Bonsai-27B-Q2_0.gguf'
0.00.051.573 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
0.00.051.580 E gguf_init_from_reader: failed to read tensor data
0.00.053.936 E llama_model_load: error loading model: llama_model_loader: failed to load model from /home/noah/Downloads/Ternary-Bonsai-27B-Q2_0.gguf
0.00.053.938 E llama_model_load_from_file_impl: failed to load model
0.00.053.956 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.085.616 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
0.00.085.619 E gguf_init_from_reader: failed to read tensor data
0.00.087.753 E llama_model_load: error loading model: llama_model_loader: failed to load model from /home/noah/Downloads/Ternary-Bonsai-27B-Q2_0.gguf
0.00.087.755 E llama_model_load_from_file_impl: failed to load model
0.00.087.758 E cmn  common_init_: failed to load model '/home/noah/Downloads/Ternary-Bonsai-27B-Q2_0.gguf'
0.00.087.760 E srv    load_model: failed to load model, '/home/noah/Downloads/Ternary-Bonsai-27B-Q2_0.gguf'
0.00.087.762 I srv    operator(): operator(): cleaning up before exit...
0.00.088.345 E srv  llama_server: exiting due to model loading error

Really hoping this release lives up to the hype. Curious about even smaller models, likely could get things working decently on super super super vintage computers for fun

Edit: Fork gives different issues? Might try pure CPU

~/Projects/nixdots took 8m15s
❯ llama-server -ngl 99 -c 8192 -m ~/Downloads/Ternary-Bonsai-27B-Q2_0.gguf
0.00.013.455 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.013.457 I device_info:
0.00.013.469 I   - ROCm0   : AMD Radeon RX 6800 (16368 MiB, 16306 MiB free)
0.00.013.505 I   - ROCm1   : AMD Ryzen 9 9950X 16-Core Processor (30964 MiB, 53344 MiB free)
0.00.013.508 I   - CPU     : AMD Ryzen 9 9950X 16-Core Processor (61929 MiB, 61929 MiB free)
0.00.013.520 I system_info: n_threads = 16 (n_threads_batch = 16) / 32 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.013.522 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.013.532 I srv          init: running without SSL
0.00.013.549 I srv          init: using 31 threads for HTTP server
0.00.013.617 I srv         start: binding port with default address family
0.00.014.687 I srv  llama_server: loading model
0.00.014.694 I srv    load_model: loading model '/home/noah/Downloads/Ternary-Bonsai-27B-Q2_0.gguf'
0.00.014.706 I common_init_result: fitting params to device memory ...
0.00.014.708 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.01.158.414 W llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.01.171.828 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
LLVM ERROR: CachedFileStream was not committed.

zsh: abort (core dumped)  llama-server -ngl 99 -c 8192 -m ~/Downloads/Ternary-Bonsai-27B-Q2_0.gguf

Edit: Yep works on CPU, just super slow like 2 tokens a second which feels a bit off for model that is only a few gigs and model. But might have to do some fixing on AMD side specifically, seems completely broken there unfortunately.

But here is a sample output. Not getting good results whatsoever, completely hilariously hallucinated output on a very simple question, maybe my expectations where blown out of proportion, or this is broken as well. Gave me a good laugh, and that's good enough I guess, read the full thing, the non-existent Beatles collab killed me

Who is Hatsune Miku?
---

Hatsune Miku is a **virtual singer** created by the Japanese audio development studio **Keiwa Corporation**. She is one of the most iconic characters in the Japanese otaku and anime culture, originating from a project called *Project Miku*, which began in 2007.

Here are some key facts about Miku:

*   **Purpose:** She was designed as a voice-acting AI, part of a series of virtual idols called "Hatsune" (meaning "celestial" or "heavenly" in Japanese). Her primary use was in video games (like *Soul Eater: The Last Rebirth*) and the *Eidol Project*, which allowed users to perform songs with her virtual avatar.
*   **Voice:** Initially, Miku’s voice was synthesized using a custom algorithm. In 2013, the studio released a version of Miku with a human voice, recorded by real vocalists, which is now the standard version.
*   **Miku Society:** The project evolved from a game to a community-driven cultural phenomenon known as the *Miku Society*. This group of fans and creators has written over 20,000 songs in various genres, created music videos, and organized live concerts with holographic projections of Miku.
*   **Global Fame:** Miku has achieved massive international popularity. She has been featured in the Guinness Book of World Records (for the most music videos on YouTube) and has collaborated with international artists, appearing on major stage productions like those of the Beatles, The Weeknd, and Lady Gaga.

In short, Hatsune Miku is a globally recognized **virtual idol and digital entity** that has significantly influenced pop culture, music, and gaming worldwide.

Edit: Same poor result with webgpu as well

Z:\llama-prism-b1-62061f9-bin-win-cuda-12.4-x64>llama-server.exe -m z:\Ternary-Bonsai-27B-PQ2_0.gguf --temp 0.7 --top-p 0.95 --top-k 20
error:
0.00.153.364 E gguf_init_from_reader: tensor 'output.weight' has invalid ggml type 142. should be in [0, 43)
0.00.153.367 E gguf_init_from_reader: failed to read tensor info
0.00.157.863 E llama_model_load: error loading model: llama_model_loader: failed to load model from z:\Ternary-Bonsai-27B-PQ2_0.gguf
0.00.157.867 E llama_model_load_from_file_impl: failed to load model
0.00.157.890 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.200.995 E gguf_init_from_reader: tensor 'output.weight' has invalid ggml type 142. should be in [0, 43)
0.00.200.998 E gguf_init_from_reader: failed to read tensor info
0.00.204.996 E llama_model_load: error loading model: llama_model_loader: failed to load model from z:\Ternary-Bonsai-27B-PQ2_0.gguf
0.00.204.999 E llama_model_load_from_file_impl: failed to load model
0.00.205.001 E common_init_from_params: failed to load model 'z:\Ternary-Bonsai-27B-PQ2_0.gguf'
0.00.205.002 E srv load_model: failed to load model, 'z:\Ternary-Bonsai-27B-PQ2_0.gguf'
0.00.205.009 I srv operator (): operator (): cleaning up before exit...
0.00.206.015 E srv llama_server: exiting due to model loading error

Yeah sorry for confusion, lots of moving parts, we are in a middle of migration for gguf formats that why shipped 3 variants of the "Q2_0", each only work with specific codebase.

• Q2_0.gguf with our llama.cpp fork
• Q2_0_g64.gguf can use with main llama.cpp (but only has metal and cpu backend right now)
  -PQ2_0 not yet supported, it will be after we finish the migration.

See more info here (should be more polished soon)
https://github.com/PrismML-Eng/Bonsai-demo#upstream-status-for-ternary

Just wanted everyone to know to not cause confusion, it not working on AMD was a user error. I forgot to disable my iGPU, for some reason Arch selected to correct GPU so it was never a issue, NixOS selected the wrong one. I thought I would make this a new comment so nobody would go hunting for a bug which doesn't exist. Still getting disappointing results

I also get this error with prism llama.cpp fork: [g64 gguf version]
gguf_init_from_reader: tensor 'output_norm.weight' has offset 357580800, expected 337715200.

with main llama.cpp it runs successfully but very slow, I think it runs on cpu instead of gpu, speed is around 1tokens/s

specs:
windows 11
Rtx 3080 ti
core i5 11th gen

Yeah sorry for confusion, lots of moving parts, we are in a middle of migration for gguf formats that why shipped 3 variants of the "Q2_0", each only work with specific codebase.

• Q2_0.gguf with our llama.cpp fork
• Q2_0_g64.gguf can use with main llama.cpp (but only has metal and cpu backend right now)
  -PQ2_0 not yet supported, it will be after we finish the migration.

See more info here (should be more polished soon)
https://github.com/PrismML-Eng/Bonsai-demo#upstream-status-for-ternary

@pashak
I hope you can pin this message, it's important.

Yeah sorry for confusion, lots of moving parts, we are in a middle of migration for gguf formats that why shipped 3 variants of the "Q2_0", each only work with specific codebase.

• Q2_0.gguf with our llama.cpp fork
• Q2_0_g64.gguf can use with main llama.cpp (but only has metal and cpu backend right now)
  -PQ2_0 not yet supported, it will be after we finish the migration.

See more info here (should be more polished soon)
https://github.com/PrismML-Eng/Bonsai-demo#upstream-status-for-ternary

@pashak
I hope you can pin this message, it's important.

Q2_0.gguf with our fork of llama.cpp -> does not work under Vulkan either (the version file contains a Vulkan file which I placed in my llama-cpp and which does not work either).