jina-colbert-v2-int8-onnx

Dynamic INT8 quantization of jinaai/jina-colbert-v2, produced for fast/enterprise-friendly deployment of KOMPAS-3D_GUARD's optional ColBERT retrieval backend. This is a derivative artifact, not an official Jina AI release.

Why this exists

The upstream onnx/model.onnx + onnx/model.onnx_data is FP32, ~2.24 GB combined, and requires an external-data file. That is a slow, unreliable download for CI/enterprise networks and inflates install size. This repo hosts a single self-contained INT8 ONNX file quantized from that same upstream checkpoint, so it can be fetched, verified, and cached deterministically.

Provenance / reproducibility

• Base checkpoint: jinaai/jina-colbert-v2, revision 4552c4dc1ffd7d7a635b6a41a1077fe9c9cdd974 (onnx/model.onnx + onnx/model.onnx_data, verified against upstream SHA256 before quantizing).
• Pipeline: onnxruntime.quantization.shape_inference.quant_pre_process (shape inference, skip_symbolic_shape=True, external data preserved) then onnxruntime.quantization.quantize_dynamic(weight_type=QuantType.QInt8).
• Tool versions: onnxruntime==1.27.0, onnx (current at build time), Python 3.14.
• No calibration data / static quantization was used (dynamic weight-only INT8); activations remain computed in floating point at inference time.

Files and SHA256

file	bytes	sha256
model_int8.onnx	564161690	b4c0b1e2db67bdd70a8bffd6c2e5ac0435e48cb127644b20a78567ce92dc505a
config.json	1577	be60be0e1395e40b2726212570fced67b599b41abb8ad27194fe1bbfc36f6029
tokenizer.json	17083370	60aeacc24f3d75c34628dfee5b7fed4bcce9a55def52b49f62a0b09d65a44b0b
tokenizer_config.json	1593	758ee60daee223039d1c04a38f595382b2d42cf63a27ad3ac88b3fa87a4f602b
special_tokens_map.json	1298	dd40018cab7225d09fe9dedadb9c70292ca1db91656772f886627857f86c1073

config.json/tokenizer*.json/special_tokens_map.json are copied verbatim from the upstream revision above (unchanged, same hashes as upstream).

Measured quality delta (quick sanity check, not a full eval)

Per-token cosine similarity between FP32 and this INT8 model's late-interaction embeddings on a short KOMPAS API sentence: mean 0.975, min 0.963 (single sample, CPU InferenceSession). This is a smoke check, not a retrieval-quality (hit@k) evaluation; validate on your own eval set before relying on it for ranking decisions.

License

The base model jinaai/jina-colbert-v2 is licensed CC-BY-NC-4.0 (non-commercial). This quantized derivative inherits the same license: non-commercial use only, with attribution to Jina AI's original model. If you need to use this in a commercial/enterprise product, you must obtain a commercial license from Jina AI for the base model — quantization does not change the licensing terms.

Usage

from fastembed import LateInteractionTextEmbedding

LateInteractionTextEmbedding.add_custom_model(
    model="dwnmf/jina-colbert-v2-int8-onnx",
    sources={"hf": "dwnmf/jina-colbert-v2-int8-onnx"},
    model_file="model_int8.onnx",
    dim=128,
    description="INT8 dynamic-quantized jina-colbert-v2 ONNX",
    license="cc-by-nc-4.0",
    size_in_GB=0.56,
)
model = LateInteractionTextEmbedding("dwnmf/jina-colbert-v2-int8-onnx")