geoffsee/octen-embedding-0.6b-onnx-int8

Octen-Embedding-0.6B ONNX (INT8)

ONNX export of Octen/Octen-Embedding-0.6B for inference with ONNX Runtime. This repo contains only the INT8 variant for smaller downloads. Other variants: fp16, int8, int4.

• Base model: Octen/Octen-Embedding-0.6B
• Max sequence length: 512
• Hidden size: 1024

Files

File	Description
model.int8.onnx	INT8 quantized, ~560 MB
tokenizer/	Same as base model

Usage

from huggingface_hub import hf_hub_download
import onnxruntime as ort
from transformers import AutoTokenizer

repo_id = "geoffsee/octen-embedding-0.6b-onnx-int8"
path = hf_hub_download(repo_id=repo_id, filename="model.int8.onnx")
tokenizer = AutoTokenizer.from_pretrained(repo_id)
session = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
# ... encode and run ...