geoffsee/octen-embedding-0.6b-onnx-int4

Octen-Embedding-0.6B ONNX (INT4)

ONNX export of Octen/Octen-Embedding-0.6B for inference with ONNX Runtime. This repo contains only the INT4 variant for smaller downloads. Other variants: fp16, int8, int4.

• Base model: Octen/Octen-Embedding-0.6B
• Max sequence length: 512
• Hidden size: 1024

Files

File	Description
model.int4.onnx (+ .onnx.data)	INT4 Bnb4 weight-only, ~537 MB
tokenizer/	Same as base model

Usage

from huggingface_hub import hf_hub_download
import onnxruntime as ort
from transformers import AutoTokenizer

repo_id = "geoffsee/octen-embedding-0.6b-onnx-int4"
path = hf_hub_download(repo_id=repo_id, filename="model.int4.onnx")
tokenizer = AutoTokenizer.from_pretrained(repo_id)
session = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
# ... encode and run ...