Instructions to use unsloth/DeepSeek-V4-Flash-0731-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

LM Studio
Jan
Ollama
How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Ollama:
```
ollama run hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
```

Unsloth Studio

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/DeepSeek-V4-Flash-0731-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/DeepSeek-V4-Flash-0731-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/DeepSeek-V4-Flash-0731-GGUF to start chatting

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/DeepSeek-V4-Flash-0731-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.DeepSeek-V4-Flash-0731-GGUF-UD-Q4_K_XL

List all available models

lemonade list

CUDA ERROR for omp

#23

by Butterfly-314 - opened 1 day ago

Discussion

Butterfly-314

1 day ago

ERROR during inference for V100 x 8

[ma-user work]$bin/llama-server -m /data/obs/model-2608/unsloth_DeepSeek-V4-Flash-0731-GGUF/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf -a DS-V4-F
lash -ngl 99 --temp 1.0 --top-p 1.0 --no-mmap -c 384000 --chat-template-kwargs '{"reasoning_effort":"max"}'  && date
0.00.550.682 W DEPRECATED: --mmap and --no-mmap are deprecated. use --load-mode mmap instead
0.00.551.957 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.14.997.092 W srv  llama_server: -----------------
[ma-user work]$bin/llama-server -m /data/obs/model-2608/unsloth_DeepSeek-V4-Flash-0731-GGUF/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf -a DS-V4-Flash -ngl 99 --temp 1.0 --top-p 1.0 --no-mmap -c 384000 --chat-template-kwargs '{"reasoning_effort":"max"}'  && date
0.00.569.482 W DEPRECATED: --mmap and --no-mmap are deprecated. use --load-mode mmap instead
0.00.570.796 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.15.327.465 W srv  llama_server: -----------------
0.15.327.473 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.15.327.473 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.15.327.474 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.15.327.474 W srv  llama_server: -----------------
0.15.331.128 I srv    load_model: loading model '/data/obs/model-2608/unsloth_DeepSeek-V4-Flash-0731-GGUF/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf'
15.28.693.546 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 384000, kv_unified = 'true'
15.28.702.800 I srv  llama_server: model loaded
15.28.702.811 I srv  llama_server: listening on http://127.0.0.1:8080
16.53.919.442 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
16.53.919.497 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
17.01.660.268 I slot get_availabl: id  2 | task -1 | selected slot by LRU, t_last = -1
17.01.662.488 I slot launch_slot_: id  2 | task 2 | processing task, is_child = 0
17.10.283.167 I slot print_timing: id  2 | task 2 | prompt processing, n_tokens =   2048, progress = 0.10, t =   8.61 s / 237.96 tokens per second
17.18.815.495 I slot print_timing: id  2 | task 2 | prompt processing, n_tokens =   4096, progress = 0.20, t =  17.14 s / 238.99 tokens per second
17.18.941.433 /home/ma-user/work/codes/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:106: CUDA error
E CUDA error: invalid configuration argument
17.18.941.464 E   current device: 0, in function argsort_f32_i32_cuda_cub at /home/ma-user/work/codes/llama.cpp/ggml/src/ggml-cuda/argsort.cu:150
17.18.941.467 E   DeviceSegmentedSort::SortPairsDescending(d_temp_storage, temp_storage_bytes, temp_keys, temp_keys, temp_indices, dst, ncols * nrows, nrows, offset_iterator, offset_iterator + 1, stream)
bin/llama-server(+0x151a05b)[0x55e55ec8505b]
bin/llama-server(+0x151a64d)[0x55e55ec8564d]
bin/llama-server(+0x151a81e)[0x55e55ec8581e]
bin/llama-server(+0x825b06)[0x55e55df90b06]
bin/llama-server(+0x11272c0)[0x55e55e8922c0]
bin/llama-server(+0x9c2bba)[0x55e55e12dbba]
bin/llama-server(+0x83b54b)[0x55e55dfa654b]
bin/llama-server(+0x153ea1a)[0x55e55eca9a1a]
bin/llama-server(+0x5731c1)[0x55e55dcde1c1]
bin/llama-server(+0x57a6eb)[0x55e55dce56eb]
bin/llama-server(+0x57f848)[0x55e55dcea848]
bin/llama-server(+0x580fd0)[0x55e55dcebfd0]
bin/llama-server(+0x20963d)[0x55e55d97463d]
bin/llama-server(+0x20ca95)[0x55e55d977a95]
bin/llama-server(+0x1a7c4e)[0x55e55d912c4e]
bin/llama-server(+0x14842d)[0x55e55d8b342d]
bin/llama-server(+0x14957f)[0x55e55d8b457f]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7efef824c083]
bin/llama-server(+0x141a5e)[0x55e55d8aca5e]
Aborted

anathor logs:

195.40.538.107 W Wrong type supplied for parameter 'tool_choice'. Expected 'string', using default value: [json.exception.type_error.302] type must be string, but is object
195.40.574.202 I slot get_availabl: id  2 | task -1 | selected slot by LRU, t_last = -1
195.40.574.560 I slot launch_slot_: id  2 | task 132231 | processing task, is_child = 0
195.42.836.450 I slot get_availabl: id  1 | task -1 | selected slot by LRU, t_last = -1
195.42.840.942 I slot launch_slot_: id  1 | task 132233 | processing task, is_child = 0
195.51.204.161 I slot print_timing: id  1 | task 132233 | prompt processing, n_tokens =   2048, progress = 0.07, t =   8.36 s / 244.96 tokens per second
195.59.658.057 I slot print_timing: id  1 | task 132233 | prompt processing, n_tokens =   4096, progress = 0.14, t =  16.81 s / 243.60 tokens per second
/home/ma-user/work/codes/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:106: CUDA error
195.59.969.478 E CUDA error: an illegal memory access was encountered
195.59.969.495 E   current device: 0, in function ggml_cuda_mul_mat_id at /home/ma-user/work/codes/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:1937
195.59.969.497 E   cudaMemcpyAsync(ids_host.data(), ids->data, ggml_nbytes(ids), cudaMemcpyDeviceToHost, stream)
bin/llama-server(+0x1519f5b)[0x55e79b044f5b]
bin/llama-server(+0x151a54d)[0x55e79b04554d]
bin/llama-server(+0x151a71e)[0x55e79b04571e]
bin/llama-server(+0x825a06)[0x55e79a350a06]
bin/llama-server(+0x837b34)[0x55e79a362b34]
bin/llama-server(+0x83b0eb)[0x55e79a3660eb]
bin/llama-server(+0x153e91a)[0x55e79b06991a]
bin/llama-server(+0x5730c1)[0x55e79a09e0c1]
bin/llama-server(+0x57a5eb)[0x55e79a0a55eb]
bin/llama-server(+0x57f748)[0x55e79a0aa748]
bin/llama-server(+0x580ed0)[0x55e79a0abed0]
bin/llama-server(+0x20961d)[0x55e799d3461d]
bin/llama-server(+0x20ca75)[0x55e799d37a75]
bin/llama-server(+0x1a7c2e)[0x55e799cd2c2e]
bin/llama-server(+0x14840d)[0x55e799c7340d]
bin/llama-server(+0x14955f)[0x55e799c7455f]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f481c251083]
bin/llama-server(+0x141a3e)[0x55e799c6ca3e]
Aborted

Butterfly-314

1 day ago

prompt processing 到 ~4096 tokens 处崩溃（CUDA error: invalid configuration argument）

DeepSeek-V4 的 DSA (DeepSeek Sparse Attention) indexer 的 ggml_top_k 操作：

DSA indexer 对 LID KV cache 的 score 做 top_k 选择
top-k.cu 中 CCCL < 3.2 时回退到 argsort
ncols = n_lid > 1024 → 走 CUB DeviceSegmentedSort
DeviceSegmentedSort 在 V100 sm_70 上 kernel 配置非法 → 崩溃

修复（1 行代码）

在 ~/work/codes/llama.cpp/ggml/src/ggml-cuda/argsort.cu 第 84 行后添加：

  is_capturing = true;  // Force DeviceSegmentedRadixSort (DeviceSegmentedSort crashes on V100 sm_70)

然后重编译：
cd build && cmake --build . --config Release -j$(nproc) --target llama-server

  cd ~/work/codes/llama.cpp
  # backup file
  cp ggml/src/ggml-cuda/argsort.cu ggml/src/ggml-cuda/argsort.cu.bak

  # Insert Fix（After line 84 #endif  // USE_CUDA_GRAPH）
  sed -i '/#endif  \/\/ USE_CUDA_GRAPH/a\    is_capturing = true;  // Force DeviceSegmentedRadixSort (DeviceSegmentedSort crashes on V100 sm_70)'
  ggml/src/ggml-cuda/argsort.cu

  # Validation
  diff ggml/src/ggml-cuda/argsort.cu.bak ggml/src/ggml-cuda/argsort.cu

  # Compile
  cd build && cmake --build . -j$(nproc) --target llama-server

Onion008

1 day ago

how is the token generation speed in V100 x 8 platform?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment