justinchuby/deepseek-r1-distill-qwen-1.5b-onnx-genai

DeepSeek-R1-Distill-Qwen-1.5B for onnx-genai

An INT4 ONNX package that runs on CPU with the nxrt Python API. It demonstrates onnx-genai's canonical inference_metadata.yaml format, which is the authoritative contract in this package. genai_config.json is retained only for ecosystem compatibility.

The metadata is a fully serialized pipeline.workflow. The autoregressive loop is data, not runtime-specific code: the decoder graph and the ten token-policy graphs under policies/ are declared as workflow components, all 28 key/value cache pairs are threaded through serving.state_service groups, and sampling, termination, and length bookkeeping are ONNX graphs the runtime executes like any other component. The package therefore runs on the generic workflow runtime with no decoder-specific lowering step.

There is no model.io section. That legacy form described a single decoder in a shape only a special-cased decoder loader could execute; it is not supported.

Run

python3 -m pip install nxrt
ONNX_GENAI_KV_MAX_LEN=128 python3 - <<'PY'
import nxrt

engine = nxrt.genai.Engine.from_dir(".")
result = engine.generate(
    "Answer in one concise sentence: What is Rust?\nAnswer:",
    max_tokens=18,
    temperature=0.0,
)
print(result.text)
print(result.token_ids)
print(result.finish_reason)
PY

Verified with nxrt==0.1.0.dev3 and CPUExecutionProvider:

 Rust is a programming language that allows for high performance and is widely used in various industries.
[33789, 374, 264, 15473, 4128, 429, 6147, 369, 1550, 5068, 323, 374, 13570, 1483, 304, 5257, 19102, 382]
max_tokens

ONNX_GENAI_KV_MAX_LEN limits the shared KV allocation for this short example; the model metadata preserves the full 131,072-token context limit.

Attribution and license

Derived from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B. The model is distributed under the MIT License and is distilled from Qwen2.5-Math-1.5B, which is distributed under Apache-2.0. See LICENSE and the upstream model card.

Annotated inference metadata

Review inference_metadata.annotated.yaml for inline explanations of this package's workflow, tensor/state/cache contracts, and fail-closed omissions. inference_metadata.yaml remains the canonical machine-authored contract; automated validation confirms both files parse to the same metadata object.