Instructions to use twhitworth/gpt-oss-120b-fp16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use twhitworth/gpt-oss-120b-fp16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="twhitworth/gpt-oss-120b-fp16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("twhitworth/gpt-oss-120b-fp16")
model = AutoModelForCausalLM.from_pretrained("twhitworth/gpt-oss-120b-fp16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use twhitworth/gpt-oss-120b-fp16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "twhitworth/gpt-oss-120b-fp16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "twhitworth/gpt-oss-120b-fp16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/twhitworth/gpt-oss-120b-fp16

SGLang

How to use twhitworth/gpt-oss-120b-fp16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "twhitworth/gpt-oss-120b-fp16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "twhitworth/gpt-oss-120b-fp16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "twhitworth/gpt-oss-120b-fp16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "twhitworth/gpt-oss-120b-fp16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use twhitworth/gpt-oss-120b-fp16 with Docker Model Runner:
```
docker model run hf.co/twhitworth/gpt-oss-120b-fp16
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Precision: FP32 vs FP16 (and BF16)

This project saves dequantized checkpoints in FP16 (bf16 -> fp16)

FP32 (single precision, 32-bit, 4 bytes/param) Reference/default precision in many frameworks. Highest numerical range/precision, largest memory.
FP16 (half precision, 16-bit, 2 bytes/param) Half the memory of FP32. Great for inference on modern GPUs; may underflow/overflow more easily than BF16.
BF16 (bfloat16, 16-bit, 2 bytes/param) Same memory as FP16, wider exponent like FP32, often more numerically robust than FP16; slightly less precision in mantissa.

In this repo, output precision is FP16 (default) or BF16 via --dtype. FP32 output is not offered because it doubles disk/RAM vs FP16/BF16 with minimal inference benefit on modern hardware.

Memory math (example: 120B parameters)

Each parameter stores one number:

Format	Bits	Bytes/param	Approx size for 120B params
FP32	32	4	~ 447 GiB
FP16	16	2	~ 224 GiB
BF16	16	2	~ 224 GiB

Calculation (GiB): params * bytes_per_param / 1024^3 For 120,000,000,000 params: FP32: 480e9 B ≈ 447.03 GiB FP16/BF16: 240e9 B ≈ 223.52 GiB

When to use which

Inference on modern NVIDIA GPUs (Turing+/Ampere+/Ada/Hopper): Use FP16 (default here) or BF16. You’ll get large memory savings and typically equal or faster throughput than FP32 thanks to tensor cores.
Training / Finetuning: Use mixed precision (BF16 or FP16 compute with an FP32 master copy of weights/optimizer states). If your GPU supports BF16 well (e.g., A100/H100), BF16 is preferred for numeric stability. (This tool focuses on exporting dequantized checkpoints, not training loops.)
If you hit numeric issues in FP16: Try BF16 (--dtype bf16). Same size as FP16 but usually more stable due to FP32-like exponent range.

Notes

FP32 remains the gold standard for numeric headroom and deterministic baselines, but for inference it’s typically unnecessary and costly (2× memory vs FP16/BF16).
Tensor cores accelerate FP16/BF16 GEMMs on most modern NVIDIA GPUs; FP32 is often slower and more memory-bound.
If a downstream runtime expects a specific dtype, export to that: FP16 for speed/memory, BF16 for robustness.

WIP

Upcoming models: cleaned FP16 release (uniform fp16 with fp32 LayerNorms), compressed variants (W8A8, W4A16, mixed experts), 2:4 sparse checkpoints.
Evals: MMLU, HellaSwag, TruthfulQA, GSM8K, BBH, MT‑Bench; plus latency/throughput and memory footprint on 3090/A100.
Extras: scripted upload tooling, detailed model cards, and reproducible Docker workflows.