Instructions to use unsloth/Qwen3.8-27B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use unsloth/Qwen3.8-27B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

LM Studio
Jan
Ollama
How to use unsloth/Qwen3.8-27B-GGUF with Ollama:
```
ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
```

Unsloth Studio

How to use unsloth/Qwen3.8-27B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3.8-27B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3.8-27B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/Qwen3.8-27B-GGUF to start chatting

How to use unsloth/Qwen3.8-27B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use unsloth/Qwen3.8-27B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use unsloth/Qwen3.8-27B-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/Qwen3.8-27B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.Qwen3.8-27B-GGUF-UD-Q4_K_XL

List all available models

lemonade list

Hermes Agent

How to use unsloth/Qwen3.8-27B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

Run Hermes

hermes

Atomic Chat

tensor split crash

#43

by gregnetau - opened about 9 hours ago

Discussion

gregnetau

about 9 hours ago

i cannot run this model on tensor split under llama.cpp (2 x R9700 AI PRO's w/64GB RAM):

llamacpp-r9700 | /app/ggml/src/ggml.c:1804: GGML_ASSERT(obj_new) failed
llamacpp-r9700 | 0.03.811.075 W ggml_new_object: not enough space in the context's memory pool (needed 754032, available 753664)
llamacpp-r9700 | libggml-base.so.0(+0x1c20b)[0x7bc8d4d5820b]
llamacpp-r9700 | libggml-base.so.0(ggml_print_backtrace+0x21c)[0x7bc8d4d5868c]
llamacpp-r9700 | libggml-base.so.0(ggml_abort+0x15b)[0x7bc8d4d5886b]
llamacpp-r9700 | libggml-base.so.0(ggml_new_tensor+0x1f8)[0x7bc8d4d5ada8]
llamacpp-r9700 | libggml-base.so.0(+0x45e0f)[0x7bc8d4d81e0f]
llamacpp-r9700 | libggml-base.so.0(ggml_gallocr_alloc_graph+0x47b)[0x7bc8d4d6f44b]
llamacpp-r9700 | libggml-base.so.0(ggml_backend_sched_alloc_graph+0x111)[0x7bc8d4d75cc1]
llamacpp-r9700 | libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xed)[0x7bc8d4f122dd]
llamacpp-r9700 | libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x54d)[0x7bc8d4f1949d]
llamacpp-r9700 | libllama.so.0(llama_decode+0xf)[0x7bc8d4f1a9cf]
llamacpp-r9700 | libllama-common.so.0(_Z23common_init_from_paramsR13common_paramsb+0x43b)[0x7bc8d557fe0b]
llamacpp-r9700 | libllama-server-impl.so(_ZN19server_context_impl10load_modelER13common_params+0x57e)[0x7bc8d5e921ee]
llamacpp-r9700 | libllama-server-impl.so(_Z12llama_serverR13common_paramsiPPc+0x361f)[0x7bc8d5dc545f]
llamacpp-r9700 | libllama-server-impl.so(_Z12llama_serveriPPc+0xd6)[0x7bc8d5dc6966]
llamacpp-r9700 | /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7bc8d582c1ca]
llamacpp-r9700 | /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7bc8d582c28b]
llamacpp-r9700 | /app/llama-server(+0x12a5)[0x5a1fc743e2a5]

gregnetau

about 9 hours ago

(For ref, UD-Q6_K_XL gguf)

puchuu

about 9 hours ago

I have no issues running it in split mode on dual 9060 xt. You need to build latest llama.cpp with recent rocm release 7.14.0. Use my build script here https://gist.github.com/andrew-aladev/529c62d09c31e3e26dec36264fd9416f

gregnetau

about 9 hours ago

Thanks for the tip - gave that a crack, but the same crash:

✔ Container llamacpp-r9700 Recreated 0.2s
Attaching to llamacpp-r9700
llamacpp-r9700 | warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
llamacpp-r9700 | 0.00.896.570 I cmn common_param: common_params_print_info: build 1 (16d222f) with GNU 15.2.0 for Linux x86_64
llamacpp-r9700 | 0.00.896.573 I cmn common_param: common_params_print_info: verbosity = 4 (adjust with the -lv N CLI arg)
llamacpp-r9700 | 0.00.896.574 I cmn common_param: device_info:
llamacpp-r9700 | 0.00.896.598 I cmn common_param: - ROCm0 : AMD Radeon AI PRO R9700 (32624 MiB, 32546 MiB free)
llamacpp-r9700 | 0.00.896.599 I cmn common_param: - ROCm1 : AMD Radeon AI PRO R9700 (32624 MiB, 32546 MiB free)
llamacpp-r9700 | 0.00.896.602 I cmn common_param: - CPU : AMD Ryzen 9 9950X 16-Core Processor (255290 MiB, 255290 MiB free)
llamacpp-r9700 | 0.00.896.632 I cmn common_param: system_info: n_threads = 16 (n_threads_batch = 16) / 32 | ROCm : NO_VMM = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
llamacpp-r9700 | 0.00.896.963 I srv start: binding port with default address family
llamacpp-r9700 | 0.00.898.148 I srv load_model: loading model '/models/Qwen3.8-27B-UD-Q6_K_XL.gguf'
llamacpp-r9700 | 0.00.898.148 I srv load_model: local path '/models/Qwen3.8-27B-UD-Q6_K_XL.gguf'
llamacpp-r9700 | 0.02.123.708 I common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llamacpp-r9700 | 0.02.123.710 I common_memory_breakdown_print: | - Meta() (Meta()) | 65248 = 64508 + (24411 = 23718 + 500 + 193) + -23671 |
llamacpp-r9700 | 0.02.123.710 I common_memory_breakdown_print: | - Host | 1139 = 994 + 0 + 145 |
llamacpp-r9700 | 0.02.633.979 I srv load_model: [spec] estimated memory usage of MTP context is 693.02 MiB
llamacpp-r9700 | 0.02.633.999 I cmn common_init_: fitting params to device memory ...
llamacpp-r9700 | 0.02.634.000 I cmn common_init_: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
llamacpp-r9700 | 0.02.634.053 W common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR, abort
llamacpp-r9700 | 0.02.634.059 I common_fit_params: fitting params to free memory took 0.00 seconds
llamacpp-r9700 | 0.02.661.658 I llama_model_loader: loaded meta data with 51 key-value pairs and 866 tensors from /models/Qwen3.8-27B-UD-Q6_K_XL.gguf (version GGUF V3 (latest))
llamacpp-r9700 | 0.02.661.672 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llamacpp-r9700 | 0.02.661.677 I llama_model_loader: - kv 0: general.architecture str = qwen35
llamacpp-r9700 | 0.02.661.678 I llama_model_loader: - kv 1: general.type str = model
llamacpp-r9700 | 0.02.661.679 I llama_model_loader: - kv 2: general.sampling.top_k i32 = 20
llamacpp-r9700 | 0.02.661.682 I llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
llamacpp-r9700 | 0.02.661.683 I llama_model_loader: - kv 4: general.sampling.temp f32 = 1.000000
llamacpp-r9700 | 0.02.661.684 I llama_model_loader: - kv 5: general.name str = Qwen3.8-27B
llamacpp-r9700 | 0.02.661.684 I llama_model_loader: - kv 6: general.basename str = Qwen3.8-27B
llamacpp-r9700 | 0.02.661.685 I llama_model_loader: - kv 7: general.description str = Renewal of the beloved Qwen model, de...
llamacpp-r9700 | 0.02.661.685 I llama_model_loader: - kv 8: general.quantized_by str = Unsloth
llamacpp-r9700 | 0.02.661.685 I llama_model_loader: - kv 9: general.size_label str = 27B
llamacpp-r9700 | 0.02.661.685 I llama_model_loader: - kv 10: general.license str = apache-2.0
llamacpp-r9700 | 0.02.661.686 I llama_model_loader: - kv 11: general.repo_url str = https://huggingface.co/unsloth
llamacpp-r9700 | 0.02.661.686 I llama_model_loader: - kv 12: general.base_model.count u32 = 1
llamacpp-r9700 | 0.02.661.686 I llama_model_loader: - kv 13: general.base_model.0.name str = Qwen3.8 27B
llamacpp-r9700 | 0.02.661.687 I llama_model_loader: - kv 14: general.base_model.0.organization str = Qwen
llamacpp-r9700 | 0.02.661.687 I llama_model_loader: - kv 15: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen3.8-27B
llamacpp-r9700 | 0.02.661.696 I llama_model_loader: - kv 16: general.tags arr[str,1] = ["unsloth"]
llamacpp-r9700 | 0.02.661.697 I llama_model_loader: - kv 17: qwen35.block_count u32 = 65
llamacpp-r9700 | 0.02.661.697 I llama_model_loader: - kv 18: qwen35.context_length u32 = 262144
llamacpp-r9700 | 0.02.661.697 I llama_model_loader: - kv 19: qwen35.embedding_length u32 = 5120
llamacpp-r9700 | 0.02.661.697 I llama_model_loader: - kv 20: qwen35.feed_forward_length u32 = 17408
llamacpp-r9700 | 0.02.661.698 I llama_model_loader: - kv 21: qwen35.attention.head_count u32 = 24
llamacpp-r9700 | 0.02.661.698 I llama_model_loader: - kv 22: qwen35.attention.head_count_kv u32 = 4
llamacpp-r9700 | 0.02.661.699 I llama_model_loader: - kv 23: qwen35.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0]
llamacpp-r9700 | 0.02.661.707 I llama_model_loader: - kv 24: qwen35.rope.freq_base f32 = 10000000.000000
llamacpp-r9700 | 0.02.661.708 I llama_model_loader: - kv 25: qwen35.attention.layer_norm_rms_epsilon f32 = 0.000001
llamacpp-r9700 | 0.02.661.708 I llama_model_loader: - kv 26: qwen35.attention.key_length u32 = 256
llamacpp-r9700 | 0.02.661.709 I llama_model_loader: - kv 27: qwen35.attention.value_length u32 = 256
llamacpp-r9700 | 0.02.661.709 I llama_model_loader: - kv 28: qwen35.nextn_predict_layers u32 = 1
llamacpp-r9700 | 0.02.661.709 I llama_model_loader: - kv 29: qwen35.ssm.conv_kernel u32 = 4
llamacpp-r9700 | 0.02.661.709 I llama_model_loader: - kv 30: qwen35.ssm.state_size u32 = 128
llamacpp-r9700 | 0.02.661.709 I llama_model_loader: - kv 31: qwen35.ssm.group_count u32 = 16
llamacpp-r9700 | 0.02.661.710 I llama_model_loader: - kv 32: qwen35.ssm.time_step_rank u32 = 48
llamacpp-r9700 | 0.02.661.710 I llama_model_loader: - kv 33: qwen35.ssm.inner_size u32 = 6144
llamacpp-r9700 | 0.02.661.710 I llama_model_loader: - kv 34: qwen35.full_attention_interval u32 = 4
llamacpp-r9700 | 0.02.661.710 I llama_model_loader: - kv 35: qwen35.rope.dimension_count u32 = 64
llamacpp-r9700 | 0.02.661.710 I llama_model_loader: - kv 36: tokenizer.ggml.model str = gpt2
llamacpp-r9700 | 0.02.661.711 I llama_model_loader: - kv 37: tokenizer.ggml.pre str = qwen35
llamacpp-r9700 | 0.02.672.750 I llama_model_loader: - kv 38: tokenizer.ggml.tokens arr[str,248320] = ["!", """, "#", "$", "%", "&", "'", ...
llamacpp-r9700 | 0.02.675.692 I llama_model_loader: - kv 39: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llamacpp-r9700 | 0.02.686.671 I llama_model_loader: - kv 40: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llamacpp-r9700 | 0.02.686.673 I llama_model_loader: - kv 41: tokenizer.ggml.eos_token_id u32 = 248046
llamacpp-r9700 | 0.02.686.673 I llama_model_loader: - kv 42: tokenizer.ggml.padding_token_id u32 = 248055
llamacpp-r9700 | 0.02.686.674 I llama_model_loader: - kv 43: tokenizer.ggml.bos_token_id u32 = 248044
llamacpp-r9700 | 0.02.686.674 I llama_model_loader: - kv 44: general.quantization_version u32 = 2
llamacpp-r9700 | 0.02.686.674 I llama_model_loader: - kv 45: general.file_type u32 = 14
llamacpp-r9700 | 0.02.686.675 I llama_model_loader: - kv 46: quantize.imatrix.file str = /mnt/disks/unslothai/ubuntu/workspace...
llamacpp-r9700 | 0.02.686.675 I llama_model_loader: - kv 47: quantize.imatrix.dataset str = unsloth_calibration_Qwen3.8-27B.txt
llamacpp-r9700 | 0.02.686.676 I llama_model_loader: - kv 48: quantize.imatrix.entries_count u32 = 496
llamacpp-r9700 | 0.02.686.676 I llama_model_loader: - kv 49: quantize.imatrix.chunks_count u32 = 45
llamacpp-r9700 | 0.02.686.677 I llama_model_loader: - kv 50: tokenizer.chat_template str = {%- set image_count = namespace(value...
llamacpp-r9700 | 0.02.686.678 I llama_model_loader: - type f32: 360 tensors
llamacpp-r9700 | 0.02.686.678 I llama_model_loader: - type q8_0: 279 tensors
llamacpp-r9700 | 0.02.686.679 I llama_model_loader: - type q5_K: 96 tensors
llamacpp-r9700 | 0.02.686.679 I llama_model_loader: - type q6_K: 131 tensors
llamacpp-r9700 | 0.02.686.680 I print_info: file format = GGUF V3 (latest)
llamacpp-r9700 | 0.02.686.681 I print_info: file type = Q4_K - Small
llamacpp-r9700 | 0.02.686.684 I print_info: file size = 24.13 GiB (7.59 BPW)
llamacpp-r9700 | 0.02.686.693 I llama_prepare_model_devices: skipping CPU (AMD Ryzen 9 9950X 16-Core Processor) for tensor parallelism
llamacpp-r9700 | 0.02.686.693 I llama_prepare_model_devices: creating a Meta device for tensor parallelism from 2 devices:
llamacpp-r9700 | 0.02.686.694 I llama_prepare_model_devices: - device 0: ROCm0 (AMD Radeon AI PRO R9700)
llamacpp-r9700 | 0.02.686.694 I llama_prepare_model_devices: - device 1: ROCm1 (AMD Radeon AI PRO R9700)
llamacpp-r9700 | 0.02.686.723 I llama_prepare_model_devices: using device Meta() (Meta()) (unknown id) - 64552 MiB free
llamacpp-r9700 | 0.02.747.635 I load: 0 unused tokens
llamacpp-r9700 | 0.02.768.137 I load: printing all EOG tokens:
llamacpp-r9700 | 0.02.768.139 I load: - 248044 ('<|endoftext|>')
llamacpp-r9700 | 0.02.768.140 I load: - 248046 ('<|im_end|>')
llamacpp-r9700 | 0.02.768.140 I load: - 248063 ('<|fim_pad|>')
llamacpp-r9700 | 0.02.768.140 I load: - 248064 ('<|repo_name|>')
llamacpp-r9700 | 0.02.768.140 I load: - 248065 ('<|file_sep|>')
llamacpp-r9700 | 0.02.768.342 I load: special tokens cache size = 33
llamacpp-r9700 | 0.02.808.880 I load: token to piece cache size = 1.7581 MB
llamacpp-r9700 | 0.02.808.892 I print_info: arch = qwen35
llamacpp-r9700 | 0.02.808.892 I print_info: vocab_only = 0
llamacpp-r9700 | 0.02.808.892 I print_info: no_alloc = 0
llamacpp-r9700 | 0.02.808.893 I print_info: n_ctx_train = 262144
llamacpp-r9700 | 0.02.808.893 I print_info: n_embd_inp = 5120
llamacpp-r9700 | 0.02.808.894 I print_info: n_embd = 5120
llamacpp-r9700 | 0.02.808.894 I print_info: n_embd_out = 5120
llamacpp-r9700 | 0.02.808.894 I print_info: n_layer = 64
llamacpp-r9700 | 0.02.808.894 I print_info: n_layer_all = 65
llamacpp-r9700 | 0.02.808.902 I print_info: n_head = 24
llamacpp-r9700 | 0.02.808.903 I print_info: n_head_kv = 4
llamacpp-r9700 | 0.02.808.904 I print_info: n_rot = 64
llamacpp-r9700 | 0.02.808.904 I print_info: n_swa = 0
llamacpp-r9700 | 0.02.808.904 I print_info: is_swa_any = 0
llamacpp-r9700 | 0.02.808.904 I print_info: n_embd_head_k = 256
llamacpp-r9700 | 0.02.808.904 I print_info: n_embd_head_v = 256
llamacpp-r9700 | 0.02.808.905 I print_info: n_gqa = 6
llamacpp-r9700 | 0.02.808.906 I print_info: n_embd_k_gqa = 1024
llamacpp-r9700 | 0.02.808.908 I print_info: n_embd_v_gqa = 1024
llamacpp-r9700 | 0.02.808.908 I print_info: f_norm_eps = 0.0e+00
llamacpp-r9700 | 0.02.808.910 I print_info: f_norm_rms_eps = 1.0e-06
llamacpp-r9700 | 0.02.808.910 I print_info: f_clamp_kqv = 0.0e+00
llamacpp-r9700 | 0.02.808.910 I print_info: f_max_alibi_bias = 0.0e+00
llamacpp-r9700 | 0.02.808.910 I print_info: f_logit_scale = 0.0e+00
llamacpp-r9700 | 0.02.808.910 I print_info: f_attn_scale = 0.0e+00
llamacpp-r9700 | 0.02.808.911 I print_info: f_attn_value_scale = 0.0000
llamacpp-r9700 | 0.02.808.911 I print_info: n_ff = 17408
llamacpp-r9700 | 0.02.808.911 I print_info: n_expert = 0
llamacpp-r9700 | 0.02.808.912 I print_info: n_expert_used = 0
llamacpp-r9700 | 0.02.808.912 I print_info: n_expert_groups = 0
llamacpp-r9700 | 0.02.808.912 I print_info: n_group_used = 0
llamacpp-r9700 | 0.02.808.912 I print_info: causal attn = 1
llamacpp-r9700 | 0.02.808.913 I print_info: pooling type = -1
llamacpp-r9700 | 0.02.808.913 I print_info: rope type = 40
llamacpp-r9700 | 0.02.808.913 I print_info: rope scaling = linear
llamacpp-r9700 | 0.02.808.914 I print_info: freq_base_train = 10000000.0
llamacpp-r9700 | 0.02.808.915 I print_info: freq_scale_train = 1
llamacpp-r9700 | 0.02.808.915 I print_info: n_ctx_orig_yarn = 262144
llamacpp-r9700 | 0.02.808.915 I print_info: rope_yarn_log_mul = 0.0000
llamacpp-r9700 | 0.02.808.915 I print_info: rope_finetuned = unknown
llamacpp-r9700 | 0.02.808.915 I print_info: mrope sections = [11, 11, 10, 0]
llamacpp-r9700 | 0.02.808.916 I print_info: ssm_d_conv = 4
llamacpp-r9700 | 0.02.808.916 I print_info: ssm_d_inner = 6144
llamacpp-r9700 | 0.02.808.916 I print_info: ssm_d_state = 128
llamacpp-r9700 | 0.02.808.917 I print_info: ssm_dt_rank = 48
llamacpp-r9700 | 0.02.808.917 I print_info: ssm_n_group = 16
llamacpp-r9700 | 0.02.808.917 I print_info: ssm_dt_b_c_rms = 0
llamacpp-r9700 | 0.02.808.922 I print_info: model type = 27B
llamacpp-r9700 | 0.02.808.923 I print_info: model params = 27.32 B
llamacpp-r9700 | 0.02.808.923 I print_info: general.name = Qwen3.8-27B
llamacpp-r9700 | 0.02.808.924 I print_info: vocab type = BPE
llamacpp-r9700 | 0.02.808.925 I print_info: n_vocab = 248320
llamacpp-r9700 | 0.02.808.925 I print_info: n_merges = 247587
llamacpp-r9700 | 0.02.808.925 I print_info: BOS token = 248044 '<|endoftext|>'
llamacpp-r9700 | 0.02.808.925 I print_info: EOS token = 248046 '<|im_end|>'
llamacpp-r9700 | 0.02.808.925 I print_info: EOT token = 248046 '<|im_end|>'
llamacpp-r9700 | 0.02.808.926 I print_info: PAD token = 248055 '<|vision_pad|>'
llamacpp-r9700 | 0.02.808.926 I print_info: LF token = 198 'Ċ'
llamacpp-r9700 | 0.02.808.926 I print_info: FIM PRE token = 248060 '<|fim_prefix|>'
llamacpp-r9700 | 0.02.808.926 I print_info: FIM SUF token = 248062 '<|fim_suffix|>'
llamacpp-r9700 | 0.02.808.927 I print_info: FIM MID token = 248061 '<|fim_middle|>'
llamacpp-r9700 | 0.02.808.927 I print_info: FIM PAD token = 248063 '<|fim_pad|>'
llamacpp-r9700 | 0.02.808.927 I print_info: FIM REP token = 248064 '<|repo_name|>'
llamacpp-r9700 | 0.02.808.927 I print_info: FIM SEP token = 248065 '<|file_sep|>'
llamacpp-r9700 | 0.02.808.927 I print_info: EOG token = 248044 '<|endoftext|>'
llamacpp-r9700 | 0.02.808.928 I print_info: EOG token = 248046 '<|im_end|>'
llamacpp-r9700 | 0.02.808.928 I print_info: EOG token = 248063 '<|fim_pad|>'
llamacpp-r9700 | 0.02.808.928 I print_info: EOG token = 248064 '<|repo_name|>'
llamacpp-r9700 | 0.02.808.928 I print_info: EOG token = 248065 '<|file_sep|>'
llamacpp-r9700 | 0.02.808.928 I print_info: max token length = 256
llamacpp-r9700 | 0.02.808.949 I load_tensors: loading model tensors, this can take a while... (load_mode = mmap)
llamacpp-r9700 | 0.03.158.354 I load_tensors: offloading output layer to GPU
llamacpp-r9700 | 0.03.158.358 I load_tensors: offloading 64 repeating layers to GPU
llamacpp-r9700 | 0.03.158.358 I load_tensors: offloaded 66/66 layers to GPU
llamacpp-r9700 | 0.03.158.365 I load_tensors: CPU_Mapped model buffer size = 994.63 MiB
llamacpp-r9700 | 0.03.158.367 I load_tensors: Meta() model buffer size = 11886.94 MiB
llamacpp-r9700 | 0.04.941.236 I cmn common_init_: added <|endoftext|> logit bias = -inf
llamacpp-r9700 | 0.04.941.241 I cmn common_init_: added <|im_end|> logit bias = -inf
llamacpp-r9700 | 0.04.941.242 I cmn common_init_: added <|fim_pad|> logit bias = -inf
llamacpp-r9700 | 0.04.941.242 I cmn common_init_: added <|repo_name|> logit bias = -inf
llamacpp-r9700 | 0.04.941.243 I cmn common_init_: added <|file_sep|> logit bias = -inf
llamacpp-r9700 | 0.04.941.346 I llama_context: constructing llama_context
llamacpp-r9700 | 0.04.941.351 I llama_context: n_seq_max = 1
llamacpp-r9700 | 0.04.941.351 I llama_context: n_ctx = 128000
llamacpp-r9700 | 0.04.941.352 I llama_context: n_ctx_seq = 128000
llamacpp-r9700 | 0.04.941.352 I llama_context: n_batch = 2048
llamacpp-r9700 | 0.04.941.352 I llama_context: n_ubatch = 512
llamacpp-r9700 | 0.04.941.352 I llama_context: causal_attn = 1
llamacpp-r9700 | 0.04.941.353 I llama_context: flash_attn = enabled
llamacpp-r9700 | 0.04.941.353 I llama_context: kv_unified = false
llamacpp-r9700 | 0.04.941.357 I llama_context: freq_base = 10000000.0
llamacpp-r9700 | 0.04.941.358 I llama_context: freq_scale = 1
llamacpp-r9700 | 0.04.941.358 I llama_context: n_rs_seq = 16
llamacpp-r9700 | 0.04.941.358 I llama_context: n_outputs_max = 17
llamacpp-r9700 | 0.04.941.359 I llama_context: n_outputs_max_per_seq = 17
llamacpp-r9700 | 0.04.941.359 I llama_context: n_ctx_seq (128000) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llamacpp-r9700 | 0.05.103.395 I llama_context: ROCm_Host output buffer size = 0.95 MiB
llamacpp-r9700 | 0.05.210.306 I llama_kv_cache: Meta() KV buffer size = 4000.00 MiB
llamacpp-r9700 | 0.05.225.316 I llama_kv_cache: size = 8000.00 MiB (128000 cells, 16 layers, 1/1 seqs), K (f16): 4000.00 MiB, V (f16): 4000.00 MiB
llamacpp-r9700 | 0.05.225.318 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
llamacpp-r9700 | 0.05.225.318 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
llamacpp-r9700 | 0.05.267.396 I llama_memory_recurrent: Meta() RS buffer size = 1271.81 MiB
llamacpp-r9700 | 0.05.267.408 I llama_memory_recurrent: size = 2543.62 MiB ( 1 cells, 64 layers, 1 seqs 16 rs_seq), R (f32): 95.62 MiB, S (f32): 2448.00 MiB
llamacpp-r9700 | 0.05.267.417 I sched_reserve: reserving ...
llamacpp-r9700 | 0.05.271.668 I resolve_fused_ops: resolving fused Gated Delta Net support:
llamacpp-r9700 | 0.05.272.808 I resolve_fused_ops: fused Gated Delta Net (autoregressive) enabled
llamacpp-r9700 | 0.05.273.560 I resolve_fused_ops: fused Gated Delta Net (chunked) enabled
llamacpp-r9700 | 0.05.273.561 I resolve_fused_ops: resolving fused Lightning Indexer support:
llamacpp-r9700 | 0.05.274.288 I resolve_fused_ops: Lightning Indexer enabled
llamacpp-r9700 | 0.05.274.289 I resolve_fused_ops: resolving fused DeepSeek V4 HC support:
llamacpp-r9700 | 0.05.275.002 I resolve_fused_ops: fused DeepSeek V4 HC pre enabled
llamacpp-r9700 | 0.05.275.708 I resolve_fused_ops: fused DeepSeek V4 HC comb enabled
llamacpp-r9700 | 0.05.276.407 I resolve_fused_ops: fused DeepSeek V4 HC post enabled
llamacpp-r9700 | 0.05.320.551 I sched_reserve: Meta() compute buffer size = 268.25 MiB
llamacpp-r9700 | 0.05.320.556 I sched_reserve: ROCm_Host compute buffer size = 145.02 MiB
llamacpp-r9700 | 0.05.320.556 I sched_reserve: graph nodes = 5959
llamacpp-r9700 | 0.05.320.556 I sched_reserve: graph splits = 2
llamacpp-r9700 | 0.05.320.557 I sched_reserve: reserve took 53.14 ms, sched copies = 1
llamacpp-r9700 | 0.05.320.616 I cmn init: llama threadpool init, n_threads = 16
llamacpp-r9700 | 0.05.320.652 I cmn common_init_: warming up the model with an empty run - please wait ... (--no-warmup to disable)
llamacpp-r9700 | /app/ggml/src/ggml.c:1804: GGML_ASSERT(obj_new) failed
llamacpp-r9700 | 0.05.333.736 W ggml_new_object: not enough space in the context's memory pool (needed 754032, available 753664)
llamacpp-r9700 | libggml-base.so.0(+0x1b076) [0x78d27fe9f076]
llamacpp-r9700 | libggml-base.so.0(ggml_print_backtrace+0x20d) [0x78d27fe9f4fd]
llamacpp-r9700 | libggml-base.so.0(ggml_abort+0x166) [0x78d27fe9f6e6]
llamacpp-r9700 | libggml-base.so.0(ggml_new_tensor+0x202) [0x78d27fea1c92]
llamacpp-r9700 | libggml-base.so.0(+0x4a884) [0x78d27fece884]
llamacpp-r9700 | libggml-base.so.0(+0x4cc7a) [0x78d27fed0c7a]
llamacpp-r9700 | libggml-base.so.0(ggml_gallocr_alloc_graph+0x4c3) [0x78d27feb6c73]
llamacpp-r9700 | libggml-base.so.0(ggml_backend_sched_alloc_graph+0x111) [0x78d27febd611]
llamacpp-r9700 | libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xed) [0x78d28005f47d]
llamacpp-r9700 | libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x476) [0x78d2800666c6]
llamacpp-r9700 | libllama.so.0(llama_decode+0x12) [0x78d280067d92]
llamacpp-r9700 | libllama-common.so.0(_Z23common_init_from_paramsR13common_paramsb+0x388) [0x78d2806fe5b8]
llamacpp-r9700 | libllama-server-impl.so(_ZN19server_context_impl10load_modelER13common_params+0x85b) [0x78d28106707b]
llamacpp-r9700 | libllama-server-impl.so(_Z12llama_serverR13common_paramsiPPc+0x3c55) [0x78d280f88aa5]
llamacpp-r9700 | libllama-server-impl.so(_Z12llama_serveriPPc+0x1197) [0x78d280f8b097]
llamacpp-r9700 | /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a601) [0x78d2809b7601]
llamacpp-r9700 | /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x88) [0x78d2809b7718]
llamacpp-r9700 | /app/llama-server(+0x1325) [0x5696daa9d325]

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment