Qwen
/

Qwen3.8-2.4T-A95B

Text Generation

qwen3_5_moe_text

Model card Files Files and versions

Instructions to use Qwen/Qwen3.8-2.4T-A95B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3.8-2.4T-A95B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/Qwen3.8-2.4T-A95B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.8-2.4T-A95B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.8-2.4T-A95B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Qwen/Qwen3.8-2.4T-A95B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3.8-2.4T-A95B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-2.4T-A95B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3.8-2.4T-A95B

How to use Qwen/Qwen3.8-2.4T-A95B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3.8-2.4T-A95B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-2.4T-A95B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3.8-2.4T-A95B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-2.4T-A95B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3.8-2.4T-A95B with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3.8-2.4T-A95B
```

978 Downloads? 🤔

#12

by cimilarkes - opened 24 days ago

Vram rich people share your specs :D

I don't think its real people, I think its bots.

Actually - i really shouldnt say this... but streaming downloads, ie streaming huge datasets... count each shard streamed as one download...
ie - 50,000 shard db = 50000 downloads per full streaming. What i think it is is some provider company streaming the model to their servers to run it.

thats just from my personal experience tho.

Yeahhhh, Bots...

why would a bot download like... 6tb repo?

likely a bunch of package manager solutions as well as clones of HF. Also likely a bunch of benchmark sites and model providers.

i run it on my samsung s24 FE locally with termux idk the specs tho but it work fine 200 toks

i run it on my samsung s24 FE locally with termux idk the specs tho but it work fine 200 toks

Proof

i run it on my samsung s24 FE locally with termux idk the specs tho but it work fine 200 toks

Proof

how to take picture of phone while phone is currently being used

so actually - its possible... technically. To do it you have to use something like colibri (github project to run glm on 24gb) but like... stream each shard when needed and then delete it.

i looked up the specs - 8gb in ram and 128gb disk - so its technially possible... but very very slow.

so actually - its possible... technically. To do it you have to use something like colibri (github project to run glm on 24gb) but like... stream each shard when needed and then delete it.

i looked up the specs - 8gb in ram and 128gb disk - so its technially possible... but very very slow.

What would the approximate tokens per second for this be on a Nokia?? I'm thinking it's close to one token every 6 months maybe?

so actually - its possible... technically. To do it you have to use something like colibri (github project to run glm on 24gb) but like... stream each shard when needed and then delete it.

i looked up the specs - 8gb in ram and 128gb disk - so its technially possible... but very very slow.

its super fast for me idk bro

so actually - its possible... technically. To do it you have to use something like colibri (github project to run glm on 24gb) but like... stream each shard when needed and then delete it.

i looked up the specs - 8gb in ram and 128gb disk - so its technially possible... but very very slow.

What would the approximate tokens per second for this be on a Nokia?? I'm thinking it's close to one token every 6 months maybe?

it get 100 toks for me not that fast like datacenter fast but it fast for locally

well then you prob aint running it on a phone lol... maybe local llm setup? since its a larger model id say smth like REAP (remove un needed experts) + dgx station?

well then you prob aint running it on a phone lol... maybe local llm setup? since its a larger model id say smth like REAP (remove un needed experts) + dgx station?

i would run it on my laptop but i broke my school laptop so i only have samsung phone right now sometimes when termux is slow i just put the model into base64 and put it inside of a html becauae html has easier access to my phones gpu and run it faster 320 avg toks but i use python in termux for agenticness

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment