# Load model directly
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("TigreGotico/opus-mt-ko-en-onnx")
model = AutoModelForSeq2SeqLM.from_pretrained("TigreGotico/opus-mt-ko-en-onnx", device_map="auto")Quick Links
opus-mt-ko-en-onnx
ONNX export (fp32 + dynamic int8 quantized) of Helsinki-NLP/opus-mt-ko-en, a Marian translation model from the Helsinki-NLP OPUS-MT project.
License: apache-2.0 (inherited from the base model; verify at the source link above).
Export
optimum-cli export onnx --model Helsinki-NLP/opus-mt-ko-en --task text2text-generation-with-past <out>
Quantized to int8 with onnxruntime.quantization.quantize_dynamic (QUInt8 weights).
File layout
./ fp32 ONNX graphs (encoder_model.onnx, decoder_model.onnx, decoder_with_past_model.onnx) + tokenizer files
./int8/ int8 dynamic-quantized ONNX graphs
fp32 size: ~1246.4 MB | int8 size: ~579.5 MB
Parity check
Compared PyTorch (MarianMTModel) vs ONNX fp32 (ORTModelForSeq2SeqLM) on 2 sentences, greedy and beam=4 (max_new_tokens=64). Overall: greedy PASS, beam4 PASS.
- src: 안녕하세요, 오늘 어떻게 지내세요?
- pytorch greedy: Hi. How are you today?
- onnx fp32 greedy: Hi. How are you today? (match)
- pytorch beam4: Hi. How are you today?
- onnx fp32 beam4: Hi. How are you today? (match)
- onnx int8 greedy: Hello. How are you today?
- src: 대단히 감사합니다.
- pytorch greedy: Thank you very much.
- onnx fp32 greedy: Thank you very much. (match)
- pytorch beam4: Thank you very much.
- onnx fp32 beam4: Thank you very much. (match)
- onnx int8 greedy: Thank you very much.
Usage
from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer
repo = "TigreGotico/opus-mt-ko-en-onnx"
tok = AutoTokenizer.from_pretrained(repo)
model = ORTModelForSeq2SeqLM.from_pretrained(repo) # fp32; pass subfolder="int8" for the quantized graphs
inputs = tok("안녕하세요, 오늘 어떻게 지내세요?", return_tensors="pt")
out = model.generate(**inputs, num_beams=4, max_new_tokens=64)
print(tok.decode(out[0], skip_special_tokens=True))
Exported for the OVOS / TigreGotico offline translation stack.
- Downloads last month
- 11
Model tree for TigreGotico/opus-mt-ko-en-onnx
Base model
Helsinki-NLP/opus-mt-ko-en
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # 'pip install "transformers<5.0.0' from transformers import pipeline pipe = pipeline("translation", model="TigreGotico/opus-mt-ko-en-onnx")