AMD works just wonderfully, here is how:
Predispositions:
() You MUST build; the pre-built executables are not working, they use CPU only (even if you remove the CUDA+HIPs and the script reaches Vulkan)
() Vulkan is faster, as always
Tools that are always helpful:
Git, CMake, and Ninja (Ninja comes with VS; also you may install it separately). Ask your loved one where to get them and how to install, it is straightforward.
For running first time:
git clone https://github.com/PrismML-Eng/llama.cpp.git
Or for updating over Ternary 1:
go to the directory of llama.cpp (the main one) and do
git pull
To run HIPs you need the workstation driver, the game adrenaline driver contains only Vulkan. HIPs maybe have better pp, but Vulkan have better tg (or it was the opposite? don't remember already, as summarized speed Vulkan is better)
Important notice:
Vulkan is multiplatform, you have my encouragement to try it with Intel videocards too!
I try everytime to enable HTTPS, but it doesn't work (with the main llama.cpp is the same), however, here is what I do:
install OpenSSL
Preparation to building:
under PowerShell (under Command prompt you have to use Path):
copy C:\Program Files\OpenSSL-Win64\lib\VC\x64\MD*.* C:\Program Files\OpenSSL-Win64\lib - this is only the 1st time after installing OpenSSL, just to make the libraries 'visible' to the compiler
$env:OPENSSL_ROOT_DIR = "C:\Program Files\OpenSSL-Win64"
$env:GGML_VULKAN_FORCE_COOPMAT="1" - this enables the new matrix cores in RDNA3/4, but also 'may' have effect on RDNA1/2, at least doesn't hurt for sure
$env:GGML_VK_SUBALLOCATION_BLOCK_SIZE="4294967296" - the default was maybe 512MB, setting to 4GB ... have some good effect, yes
cmake -S . -B build.Vulkan -G Ninja -DGGML_VULKAN=ON -DGGML_VULKAN_USE_COOPMAT=ON -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DOPENSSL_ROOT_DIR="C:\Program Files\OpenSSL-Win64" -DGGML_OPENMP=OFF -DLLAMA_BUILD_BORINGSSL=ON -DLLAMA_BUILD_TESTS=OFF
with the current fork of PrismML, the tests=OFF (last parameter) is mandatory, without it the compilation break and can't finish !
() -B build.Vulkan is the directory, where the compiled will go, use whatever you like (or need) here
() you may notice that I duplicate the enviroinment variables as command parameters, and also added BoringSSL for the HTTPS/SSL, and yet no success. Hope for you it will be better, I have no more ideas than to install and include them (as shown in the command).
Build:
cmake --build build.Vulkan --config Release --
after that, as usual in the directory there will be subdir bin\ where all the .exe files reside.
Again why to use this?
Coz the 'official' information is not fully true, just like it was for Ternary1. It works wonderfully, the pre-built executables are unusable for AMD (both ROCm and Vulkan, they have CPU offload only), they generate their own directory tree, and their command prompt is very basic. I think most of the team have no access to AMD hardware.
Here is what you may find useful as starting command for 16GB AMD Radeon card:
.\build.vulkan\bin\llama-server.exe -m "Ternary-Bonsai-2-27B-PQ2_0" -ngl 99 -fa on -c 262144 --host 0.0.0.0 -t 16 --dynatemp-range 0.15 --top-p 0.34 --top-k 12 --min-p 0.45 --repeat-penalty 1.12 --presence-penalty 0.0 -ctk q5_1 -ctv q5_1 --spec-type ngram-mod,ngram-map-k4v -np 1 --spec-draft-n-max 2 --spec-ngram-mod-n-match 20 --spec-ngram-mod-n-min 36 --spec-ngram-mod-n-max 68 --spec-ngram-map-k4v-size-n 10 --cache-ram 4096 --chat-template-file .\Qwen-3.8\Qwen-sharp-chat_template.jinja --reasoning-effort medium --perf --slot-save-path .\cache\ -b 8192 -ub 448 -cms 3172 -ctxcp 56 --lookup-cache-dynamic .\cache\n-gram.cache -lm dio
() --chat-template-file .\Qwen-3.8\Qwen-sharp-chat_template.jinja I'm using the sharp jinja template from peculiar-ragdoll https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates/tree/main
() -ub 448 is due to AMD RDNA 2 architecture (RX 6xxx cards), no idea how it behaves on RDNA3/4 (it will have benefit to be 768 or even 1024)
() -t 16 is to use 16 CPU threads if anything goes there, not actually needed
() I'm yet to experiment with MTPs, for Ternary1 DSpark was not working, despite it was paired with it.
In my case (RDNA2 with 16GB VRAM) from the dense QWEN 3.8 27B I can use Q3-K-XXS only, this baby here gives 6x pp and slightly lower tg. For some reason currently ngram is not working, only 5% approval, so still on the path to fix it. While with the dense model it was ~83% approval (same topics).
With this default 256K context my vram is 14.4 GB from 16 GB used, CPU usage is most of the time 1% to 3% (despite it shows 0.7 GB shared RAM usage).
So, again: all is working. The model is truly, visibly better than the Q3_XXS dense.
Their efforts (PrismML team) are monumental and their success is much, much more than that!