Instructions to use Jab1718/qwen3.8-flash-coder-85gb-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jab1718/qwen3.8-flash-coder-85gb-bf16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Jab1718/qwen3.8-flash-coder-85gb-bf16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Jab1718/qwen3.8-flash-coder-85gb-bf16")
model = AutoModelForCausalLM.from_pretrained("Jab1718/qwen3.8-flash-coder-85gb-bf16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Jab1718/qwen3.8-flash-coder-85gb-bf16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jab1718/qwen3.8-flash-coder-85gb-bf16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jab1718/qwen3.8-flash-coder-85gb-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Jab1718/qwen3.8-flash-coder-85gb-bf16

SGLang

How to use Jab1718/qwen3.8-flash-coder-85gb-bf16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jab1718/qwen3.8-flash-coder-85gb-bf16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jab1718/qwen3.8-flash-coder-85gb-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jab1718/qwen3.8-flash-coder-85gb-bf16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jab1718/qwen3.8-flash-coder-85gb-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Jab1718/qwen3.8-flash-coder-85gb-bf16 with Docker Model Runner:
```
docker model run hf.co/Jab1718/qwen3.8-flash-coder-85gb-bf16
```

Incredible project questions about total download size,coding quality,benchmarks

by Abdulazizv - opened 6 days ago

Discussion

Abdulazizv

6 days ago

•

edited 6 days ago

Awesome project , and
Am i correct that You are trying to make model size smaller while preserving agentic coding quality it will be close to the original model right ?
And also will the final model be able to run as fast as other 35b moe models.
do you plan to do benchmarks
I am really hyped for this project .
Fully supporting you !!

Jab1718

Owner 5 days ago

yep, but im not so sure if im going to do a full benchmark comparison since i only have around 76gb vram. (will consider about that later on )

labdhee

5 days ago

following this thread

Abdulazizv

4 days ago

•

edited 4 days ago

Hi ,
saw your discussion where you asked and I would lean toward option B, because the single gpu version would make this much more accesible to users with limited vram.

However, i also think the bf16 dora work is also really valuable.if the failures are like mostly edge cases rather than capability issues it means tuning can could make it bf16 checkpoint a much stronger foundation.
I think ideal path is to keep the bf16 checkpoint as the main reference model and make int4 int8 variabts from the improved version.
It would be also interesting to see int4 vs int8 results on the same 100 tasks
This is already a pretty fascinating thing.

Jab1718

Owner 4 days ago

good idea, imma try that soon

Abdulazizv

about 23 hours ago

Hey Jab, just wondering — are you still planning to release the ~19.7GB INT4/smaller version?

I saw that the current model file is around 40GB, while the README mentions ~19.7GB. I'd really love to test the smaller version locally on my limited-VRAM setup.

No pressure, just curious if you're still planning to make it. Really cool project btw!

Jab1718

Owner about 21 hours ago

•

edited about 21 hours ago

ye im still working on it, it is just that i have been quite busy recently so i cant release it yet. i still need to finetune the model first then i can quantize it down to Q4. i will try release it asap for you guys

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment