Instructions to use gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4")
model = AutoModelForMultimodalLM.from_pretrained("gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4

SGLang

How to use gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4 with Docker Model Runner:
```
docker model run hf.co/gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4
```

Linux/SGLang/LMHead4/TextOnly/DSpark/236k context single 5090 GODSPEED recipe

by cosmicnag - opened 18 days ago

Discussion

cosmicnag

18 days ago

•

edited 18 days ago

# ENV VARS:
   PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

 # Full sglang args:
   --model-path /path/to/gittensor-model-hub--Qwen3.8-27B-NVFP4-RTX5090-LMHead4
   --host 127.0.0.1 --port ${PORT}
   --trust-remote-code --tp-size 1
   --context-length 237568
   --max-total-tokens 237568
   --kv-cache-dtype fp8_e4m3
   --attention-backend flashinfer
   --chunked-prefill-size 2048
   --mamba-radix-cache-strategy extra_buffer_lazy
   --mamba-ssm-dtype bfloat16
   --max-mamba-cache-size 5
   --mem-fraction-static 0.985
   --max-running-requests 1
   --speculative-algorithm DSPARK
   --speculative-draft-model-path /path/to/gittensor-model-hub--Qwen3.8-27B-DSpark-NVFP4
   --speculative-dspark-block-size 7
   --speculative-draft-model-quantization modelopt_fp4
   --reasoning-parser qwen3
   --tool-call-parser qwen3_coder
   --mm-feature-transport cpu
   --language-only

Hitting 150-260 tok/sec @525 W (avg about 180-200 depending on kind of task). This is the fastest possible 5090 27b with whatever fp8 kv context I could fit.
Nvtop shows 31.202Gi/31.843G usage once model is loaded and inferencing.
Note that this needs the gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4 model (not the main one).

cosmicnag changed discussion title from Linux/SGLang/LMHead4/DSpark/236k context single 5090 GODSPEED recipe to Linux/SGLang/LMHead4/TextOnly/DSpark/236k context single 5090 GODSPEED recipe 18 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment