jinaai/jina-reranker-v3.5-GGUF

jina-reranker-v3.5 — GGUF

GGUF quantizations of jinaai/jina-reranker-v3.5, a domain-ready multilingual listwise reranker from Jina AI.

Available quantizations

	BF16	Q8_0	Q6_K	Q5_K_M	Q5_K_S	Q4_K_M	Q4_K_S	IQ4_NL	IQ4_XS	IQ3_S	IQ3_XS	IQ3_XXS	IQ2_S	IQ2_XS	IQ2_XXS	IQ1_M	IQ1_S
Size	1.2 GB	610 MB	473 MB	424 MB	417 MB	379 MB	366 MB	366 MB	353 MB	309 MB	299 MB	267 MB	243 MB	231 MB	219 MB	207 MB	199 MB

Files

• jina-reranker-v3.5-*.gguf — Quantized model weights (see table above for sizes)
• projector.safetensors — MLP projector weights (not baked into the GGUF; required for scoring)
• tokenizer.json — Tokenizer required by rerank.py for in-process tokenization and block splitting
• rerank.py — Reranker implementation
• imatrix.dat — Importance matrix used to guide quantization of lower-bit variants

Requirements

• Python 3.8+
• llama-embedding binary — built from the fork below (see Building llama.cpp)
• projector.safetensors from this repo (the scoring MLP — not baked into the GGUF)
• Python packages: numpy, safetensors, torch, tokenizers

Building llama.cpp

This model requires a non-causal encoder mode and a custom --output-token-ids flag that are not yet in the official llama.cpp release. A PR is open upstream:

ggml-org/llama.cpp#26286

Until the PR is merged, build llama-embedding from the fork:

git clone https://github.com/littlewine/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # or -DGGML_METAL=ON on macOS
cmake --build build --config Release -j$(nproc) --target llama-embedding

The resulting binary is at build/bin/llama-embedding.

CPU-only build: omit the -DGGML_CUDA=ON flag.

Usage

from rerank import GGUFReranker

reranker = GGUFReranker(
    model_path="jina-reranker-v3.5-Q4_K_M.gguf",
    projector_path="projector.safetensors",
    llama_embedding_path="/path/to/llama-embedding",
    tokenizer_path="tokenizer.json",  # omit if tokenizer.json is next to the .gguf
)

results = reranker.rerank(
    query="What causes inflammation in rheumatoid arthritis?",
    documents=[
        "Rheumatoid arthritis is caused by immune system attacking joints...",
        "Osteoarthritis results from mechanical wear on cartilage...",
    ],
)

for r in results:
    print(f"[{r['index']}] {r['relevance_score']:.4f}  {r['document'][:60]}")

Citation

If you find jina-reranker-v3.5 useful in your research, please cite the original paper:

@misc{nasika2026jinarerankerv35efficientlistwisereranker,
      title={jina-reranker-v3.5: An Efficient Listwise Reranker with Hybrid Attention and Self-Distillation}, 
      author={Christina Nasika and Feng Wang and Antonis Krasakis and Han Xiao},
      year={2026},
      eprint={2607.18152},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2607.18152}, 
}

License

This GGUF implementation follows the same CC BY-NC 4.0 license as the original model. For commercial usage inquiries, please contact Jina AI.