Instructions to use bartowski/ukisai_Swift-Qwen3.8-27b-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use bartowski/ukisai_Swift-Qwen3.8-27b-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M

Use Docker

docker model run hf.co/bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use bartowski/ukisai_Swift-Qwen3.8-27b-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bartowski/ukisai_Swift-Qwen3.8-27b-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bartowski/ukisai_Swift-Qwen3.8-27b-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M

Ollama
How to use bartowski/ukisai_Swift-Qwen3.8-27b-GGUF with Ollama:
```
ollama run hf.co/bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M
```
Unsloth Desktop

How to use bartowski/ukisai_Swift-Qwen3.8-27b-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use bartowski/ukisai_Swift-Qwen3.8-27b-GGUF with Docker Model Runner:
```
docker model run hf.co/bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M
```

Lemonade

How to use bartowski/ukisai_Swift-Qwen3.8-27b-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.ukisai_Swift-Qwen3.8-27b-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use bartowski/ukisai_Swift-Qwen3.8-27b-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use bartowski/ukisai_Swift-Qwen3.8-27b-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Model not working

by l0rdraiden - opened 4 days ago

Discussion

l0rdraiden

4 days ago

With the same configuration but using -hf "ukisai/Swift-Qwen3.8-27B-GGUF:Q4_K_M" ^
it just works.

If I use your model bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_L it extremly slow 10 toks vs 70 with ukisai

@echo off
setlocal

cd /d D:\AI\llama.cpp

set "GGML_VK_SUBALLOCATION_BLOCK_SIZE=4294967296"

set "LLAMA_ARG_HOST=10.10.10.21"
set "LLAMA_ARG_PORT=8080"

set "LLAMA_ARG_CTX_SIZE=147456"
set "LLAMA_ARG_CACHE_RAM=8192"
set "LLAMA_ARG_KV_UNIFIED=1"

set "LLAMA_ARG_FIT=on"
set "LLAMA_ARG_FIT_TARGET=256"

set "LLAMA_ARG_N_GPU_LAYERS=all"

set "LLAMA_ARG_FLASH_ATTN=on"

set "LLAMA_ARG_CACHE_TYPE_K=q8_0"
set "LLAMA_ARG_CACHE_TYPE_V=q5_0"

set "LLAMA_ARG_BATCH=2048"
set "LLAMA_ARG_UBATCH=512"

set "LLAMA_ARG_N_PARALLEL=1"
set "LLAMA_ARG_THREADS=12"

set "LLAMA_ARG_JINJA=1"
set "LLAMA_ARG_CHAT_TEMPLATE_FILE=D:\AI\templates\chat_template_froggeric_22.5.jinja"

set "LLAMA_ARG_REASONING=on"
set "LLAMA_ARG_REASONING_PRESERVE=off"
set "LLAMA_ARG_REASONING_FORMAT=deepseek"

set "LLAMA_ARG_SPEC_TYPE=draft-mtp"
set "LLAMA_ARG_SPEC_DRAFT_N_MAX=3"
set "LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_K=q8_0"
set "LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_V=q5_0"

set "LLAMA_ARG_TEMPERATURE=1"
set "LLAMA_ARG_TOP_P=0.95"
set "LLAMA_ARG_TOP_K=20"
set "LLAMA_ARG_MIN_P=0"
set "LLAMA_ARG_PRESENCE_PENALTY=0"

set "LLAMA_ARG_CHAT_TEMPLATE_KWARGS={"reasoning_effort":"medium"}"

llama-server.exe ^
-hf "bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_L" ^
--mmproj "D:\AI\models\hub\models--bartowski--ukisai_Swift-Qwen3.8-27b-GGUF\snapshots\d8175c5357a565bfd1514ecf6f37201cc8b99acd\mmproj-ukisai_Swift-Qwen3.8-27b-bf16.gguf" ^
--no-mmproj-offload

echo.
echo llama-server exited with code %ERRORLEVEL%

pause

0.01.953.220 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.01.953.493 W srv llama_server: -----------------
0.01.953.497 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.01.953.499 W srv llama_server: this can be a security risk (cross-origin attacks)
0.01.953.499 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.01.953.499 W srv llama_server: -----------------
0.01.961.641 I srv load_model: loading model 'bartowski/ukisai_Swift-Qwen3.8-27b-GGUF:Q4_K_L'
0.16.103.191 I cmn init: llama threadpool init, n_threads = 12
0.16.265.992 I common_speculative_init_result: creating MTP draft context against the target model 'D:\AI\models\hub\models--bartowski--ukisai_Swift-Qwen3.8-27b-GGUF\snapshots\d8175c5357a565bfd1514ecf6f37201cc8b99acd\ukisai_Swift-Qwen3.8-27b-Q4_K_L.gguf'
0.16.671.842 W load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
0.16.671.847 W load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
0.16.671.847 W load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

0.19.729.820 I srv load_model: loaded multimodal model, 'D:\AI\models\hub\models--bartowski--ukisai_Swift-Qwen3.8-27b-GGUF\snapshots\d8175c5357a565bfd1514ecf6f37201cc8b99acd\mmproj-ukisai_Swift-Qwen3.8-27b-bf16.gguf'
0.19.937.177 I srv load_model: initializing, n_slots = 1, n_ctx_slot = 147456, kv_unified = 'true'
0.20.012.669 I srv init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.20.013.298 I srv llama_server: model loaded
0.20.013.304 I srv llama_server: listening on http://10.10.10.21:8080
0.20.013.305 W srv llama_server: NOTICE: server default port will be changed to :9931 in a future release
0.20.013.305 W srv llama_server: ref: https://github.com/ggml-org/llama.cpp/pull/26508
0.22.567.225 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
0.22.568.498 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
0.35.677.545 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 2048, progress = 0.08, t = 9.04 s / 226.46 tokens per second
0.46.255.550 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 4096, progress = 0.16, t = 20.99 s / 195.10 tokens per second
0.56.969.797 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 6144, progress = 0.24, t = 31.63 s / 194.26 tokens per second
1.07.619.120 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 8192, progress = 0.31, t = 42.32 s / 193.58 tokens per second
1.18.434.080 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 10240, progress = 0.39, t = 53.06 s / 192.98 tokens per second
1.29.276.119 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 12288, progress = 0.47, t = 63.93 s / 192.22 tokens per second
1.40.180.157 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 14336, progress = 0.55, t = 74.79 s / 191.70 tokens per second
1.51.234.691 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 16384, progress = 0.63, t = 85.83 s / 190.88 tokens per second
2.02.372.180 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 18432, progress = 0.71, t = 96.92 s / 190.17 tokens per second
2.13.579.460 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 20480, progress = 0.79, t = 108.13 s / 189.40 tokens per second
2.24.979.942 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 22528, progress = 0.86, t = 119.49 s / 188.53 tokens per second
2.36.461.270 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 24576, progress = 0.94, t = 130.92 s / 187.72 tokens per second
2.42.255.333 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 25554, progress = 0.98, t = 136.79 s / 186.81 tokens per second
2.44.002.785 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 25857, progress = 0.99, t = 139.77 s / 185.00 tokens per second
2.44.961.753 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 25993, progress = 1.00, t = 142.01 s / 183.04 tokens per second
2.45.603.650 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 26066, progress = 1.00, t = 142.49 s / 182.93 tokens per second
2.53.331.243 I slot print_timing: id 0 | task 0 | prompt eval time = 143244.39 ms / 26070 tokens ( 5.49 ms per token, 182.00 tokens per second)
2.53.331.247 I slot print_timing: id 0 | task 0 | eval time = 7518.12 ms / 62 tokens ( 123.25 ms per token, 8.11 tokens per second)
2.53.331.248 I slot print_timing: id 0 | task 0 | total time = 150762.51 ms / 26132 tokens
2.53.331.248 I slot print_timing: id 0 | task 0 | graphs reused = 28
2.53.331.253 I slot print_timing: id 0 | task 0 | draft acceptance = 0.40230 ( 35 accepted / 87 generated), mean len = 2.21
2.53.335.104 I slot release: id 0 | task 0 | stop processing: n_tokens = 26134, truncated = 0
2.53.335.113 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 171804985
2.53.962.229 I slot launch_slot_: id 0 | task 2 | processing task, is_child = 0
3.00.252.706 I slot print_timing: id 0 | task 2 | prompt eval time = 3168.52 ms / 500 tokens ( 6.34 ms per token, 157.80 tokens per second)
3.00.252.710 I slot print_timing: id 0 | task 2 | eval time = 3121.78 ms / 33 tokens ( 97.56 ms per token, 10.25 tokens per second)
3.00.252.711 I slot print_timing: id 0 | task 2 | total time = 6290.30 ms / 533 tokens
3.00.252.712 I slot print_timing: id 0 | task 2 | graphs reused = 38
3.00.252.717 I slot print_timing: id 0 | task 2 | draft acceptance = 0.61111 ( 22 accepted / 36 generated), mean len = 2.83
3.00.252.848 I slot release: id 0 | task 2 | stop processing: n_tokens = 534, truncated = 0
3.00.336.659 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.949 (> 0.100 thold), f_keep = 0.933
3.00.336.906 I slot launch_slot_: id 0 | task 65 | processing task, is_child = 0
3.07.735.484 I slot print_timing: id 0 | task 65 | prompt eval time = 617.35 ms / 29 tokens ( 21.29 ms per token, 46.97 tokens per second)
3.07.735.489 I slot print_timing: id 0 | task 65 | eval time = 6781.00 ms / 65 tokens ( 105.95 ms per token, 9.44 tokens per second)
3.07.735.490 I slot print_timing: id 0 | task 65 | total time = 7398.35 ms / 94 tokens
3.07.735.490 I slot print_timing: id 0 | task 65 | graphs reused = 63
3.07.735.494 I slot print_timing: id 0 | task 65 | draft acceptance = 0.48718 ( 38 accepted / 78 generated), mean len = 2.46
3.07.735.538 I slot release: id 0 | task 65 | stop processing: n_tokens = 589, truncated = 0
3.07.810.634 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.929 (> 0.100 thold), f_keep = 0.888
3.07.810.937 I slot launch_slot_: id 0 | task 95 | processing task, is_child = 0
3.13.128.816 I slot print_timing: id 0 | task 95 | prompt eval time = 631.25 ms / 42 tokens ( 15.03 ms per token, 66.54 tokens per second)
3.13.128.821 I slot print_timing: id 0 | task 95 | eval time = 4686.46 ms / 56 tokens ( 85.21 ms per token, 11.74 tokens per second)
3.13.128.822 I slot print_timing: id 0 | task 95 | total time = 5317.70 ms / 98 tokens
3.13.128.823 I slot print_timing: id 0 | task 95 | graphs reused = 80
3.13.128.827 I slot print_timing: id 0 | task 95 | draft acceptance = 0.68519 ( 37 accepted / 54 generated), mean len = 3.06
3.13.128.888 I slot release: id 0 | task 95 | stop processing: n_tokens = 618, truncated = 0

battataka

4 days ago

•

edited 4 days ago

LLAMA_ARG_CACHE_TYPE_K=q8_0
LLAMA_ARG_CACHE_TYPE_V=q5_0

If you are using the pre-compiled llama-server from the official GitHub repo, I believe the K (Q8_0) and V (Q5_0) caches will fall back to the CPU.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment