danielhanchen commited on
Commit
0bdd42d
·
verified ·
1 Parent(s): cf07878

Add vLLM run instructions

Browse files
Files changed (1) hide show
  1. README.md +14 -0
README.md CHANGED
@@ -31,6 +31,20 @@ Gemma 4 can now be run and fine-tuned in [Unsloth Studio](https://unsloth.ai/doc
31
 
32
  See all versions of Gemma 4 (GGUF, 16-bit etc.) [in our collection](https://huggingface.co/collections/unsloth/gemma-4).
33
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
34
  ---
35
 
36
  <div align="center">
 
31
 
32
  See all versions of Gemma 4 (GGUF, 16-bit etc.) [in our collection](https://huggingface.co/collections/unsloth/gemma-4).
33
 
34
+ ## Run in vLLM
35
+
36
+ ```shell
37
+ uv venv unsloth-nvfp4-env --python 3.13
38
+ source unsloth-nvfp4-env/bin/activate
39
+ uv pip install "vllm>=0.25.0" "flashinfer-python>=0.6.13" "nvidia-cutlass-dsl>=4.5.2" --torch-backend=auto
40
+ ```
41
+
42
+ ```shell
43
+ vllm serve unsloth/gemma-4-E2B-it-NVFP4
44
+ ```
45
+
46
+ Do not use the Marlin backend (around 2x slower); let vLLM auto-select the NVFP4 kernel.
47
+
48
  ---
49
 
50
  <div align="center">