keisuke-miyako/h2o-danube3-4b-chat-onnx-int4-NG

⚠️ Do not use with onnxruntime-genai; model architecture has a non-standard head size of 120 which is not divisible by 16.

To save memory during export & quantisation

hf download h2oai/h2o-danube3-4b-chat --local-dir ~/.onnx/h2o-danube3-4b-chat

from optimum.onnxruntime import ORTModelForCausalLM
import torch

model_id = "~/.onnx/h2o-danube3-4b-chat"

print(f"Loading {model_id} in FP16 to save RAM...")

# 1. Load model in FP16 directly (Crucial for preventing crashes)
model = ORTModelForCausalLM.from_pretrained(
    model_id,
    export=True,
    torch_dtype=torch.float16,  # This tells PyTorch to use half the memory
    trust_remote_code=True
)

# 2. Save the ONNX model
print("Exporting to ONNX...")
model.save_pretrained("~/.onnx/h2o-danube3-4b-chat-onnx")
print("Done!")

python -m onnxruntime_genai.models.builder \
  -m ~/.onnx/h2o-danube3-4b-chat-onnx \
  -o ~/.onnx/h2o-danube3-4b-chat-onnx-int4-cpu \
  -p int4 \
  -e cpu

¥