ai-babai/giga-embeddings-0826-480m-gguf

Giga Embeddings 0826 480M — GGUF for llama.cpp and Ollama

Русская карточка · GGUF collection · Original model · Larger 3B GGUF · Original paper

Compact local Russian and English text embeddings for semantic search, RAG, text similarity, clustering, and classification. This repository contains a BF16 reference and three direct GGUF quantizations of the bidirectional Giga Embeddings 0826 480M encoder for stock llama.cpp and Ollama.

Start with Q8_0. It is the recommended default and keeps quality closest to BF16 while cutting the download to 0.520 GB. Use BF16 for maximum fidelity. Q6_K is an alternative compact format for users who specifically need Q6, but it is not recommended over Q8_0. Q4_K_M is the minimum-size experimental option and has measurable representation and retrieval loss.

This is an independent ai-babai GGUF conversion, not an official ai-sage release.

Choose a quant in 10 seconds

Variant	Role	File size	Saving vs BF16	CPU peak RSS	Metal allocation	Metal 1×512
BF16	reference / maximum fidelity	0.974 GB	—	4.159 GB	2.269 GB	4,981 tok/s
Q8_0	recommended / default	0.520 GB	46.6%	3.556 GB	1.816 GB	4,868 tok/s
Q6_K	alternative compact; prefer Q8	0.403 GB	58.6%	3.321 GB	1.699 GB	4,594 tok/s
Q4_K_M	experimental / minimum size	0.325 GB	66.6%	3.166 GB	1.621 GB	4,647 tok/s

Apple Silicon uses unified memory. Metal allocation and process RSS are different views of the same shared memory and must not be added. The measured Metal peak RSS values were 2.465 / 2.011 / 1.885 / 1.814 GB for BF16 / Q8_0 / Q6_K / Q4_K_M. All capacities use decimal GB (1 GB = 10^9 bytes).

SHA256SUMS · Machine-readable manifest

Quick start with llama.cpp

Download the recommended file:

hf download ai-babai/giga-embeddings-0826-480m-gguf \
  giga-embeddings-0826-480m-q8_0.gguf \
  --local-dir .

Run a recent stock llama.cpp server:

llama-server \
  -m giga-embeddings-0826-480m-q8_0.gguf \
  --embeddings \
  -c 2048 -b 2048 -ub 2048 -np 1 \
  --cache-type-k f32 --cache-type-v f32 \
  --flash-attn auto -ngl 99 \
  --host 127.0.0.1 --port 8080

Use -ngl 0 for CPU-only execution. Embed an instruction-prefixed query and plain documents through the OpenAI-compatible endpoint:

curl http://127.0.0.1:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "giga-embeddings-0826-480m-q8_0.gguf",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации.",
      "Париж — столица Франции."
    ]
  }'

The server returns normalized 1024-dimensional embeddings. Compare them with cosine similarity, or equivalently with a dot product after normalization.

Quick start with Ollama

Create Modelfile next to the downloaded GGUF:

FROM ./giga-embeddings-0826-480m-q8_0.gguf

Import it and call the embeddings API:

# Start this in a separate terminal if the Ollama app/service is not running:
ollama serve

ollama create giga-embeddings-0826-480m-q8 -f Modelfile

curl http://127.0.0.1:11434/api/embed \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "giga-embeddings-0826-480m-q8",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации."
    ]
  }'

The four files were tested with Ollama 0.33.3 on Apple M4 Pro Metal. They expose the embedding capability only; they are not generative chat models.

Prompt, pooling, and normalization

For retrieval and other asymmetric tasks, add a task instruction to the query and embed documents without it:

Instruct: {one-sentence task description}
Query: {query text}

For symmetric tasks such as semantic similarity or deduplication, use the same instruction for both sides or no instruction. The GGUF metadata selects the required mean pooling and L2 normalization; do not substitute CLS or last-token pooling.

The sentence-transformers tag points to the compatible upstream model and training/usage ecosystem. These .gguf files themselves are intended for llama.cpp or Ollama and are not loaded by Python SentenceTransformer.

Quality at a glance

The original authors report 70.98 Russian MTEB, 69.52 English MTEB, 72.87 code MTEB, and 56.97 multilingual MTEB for the source BF16 model. Those scores belong to the upstream model; we did not rerun the complete MTEB suites for these GGUF files.

Our full pinned evaluation used all 1,692 RuBQ and 300 SciFact test queries, instruction-prefixed queries, title + "\n" + text documents, a 512-token limit, and equal weighting of the two tasks.

Variant	NDCG@10	MRR@10	Recall@10	NDCG change vs BF16
BF16	0.724512	0.720000	0.839233	reference
Q8_0	0.725245	0.719655	0.843399	+0.0732 points¹
Q6_K	0.725051	0.719660	0.842728	+0.0538 points¹
Q4_K_M	0.717830	0.710399	0.839366	−0.6682 points

¹Small positive differences are normal ranking variation and are not evidence that quantization improves the model. This evaluation is not a complete MTEB run, a leaderboard submission, or a cross-model comparison.

The stricter frozen RU/EN/code/multilingual holdout compared each quantized GGUF with this repository's BF16 GGUF across 512 aligned texts and a 256-query/2,048-document ranking set:

Variant	Min / mean vector cosine	Top-1 agreement	Mean top-10 overlap
Q8_0	0.993771 / 0.999333	99.609%	98.164%
Q6_K	0.976087 / 0.993303	98.047%	94.570%
Q4_K_M	0.844764 / 0.928026	91.406%	82.227%

Q8_0 is the revision 2 recommended default after explicit review of both full retrieval and the frozen holdout. The original default-role rule recorded one English top-1 disagreement: 63/64 (98.4375%) versus its predeclared 99.0% family floor. Forensic review found that sole changed top result beneficial on the labelled holdout: Q8 selected a qrel-relevant SciFact document where BF16 did not. The measurement and threshold miss remain disclosed; the revision 2 role is a product decision, not a rewritten test result.

Q6_K has no clear quality/size advantage over Q8_0 and shows broader representation/ranking drift. Q4_K_M is published for minimum-size use with a clear experimental label: its full aggregate NDCG loss is 0.6682 points, its SciFact loss is 1.2942 points, and its holdout drift is substantially larger.

Measured speed

Median total throughput after two warmups and five measured repetitions, context 2048 and parallelism 1:

Backend / workload	BF16	Q8_0	Q6_K	Q4_K_M
Apple M4 Pro Metal, 1×512	4,981 tok/s	4,868 tok/s	4,594 tok/s	4,647 tok/s
Apple M4 Pro Metal, 16×1024	4,545 tok/s	4,358 tok/s	3,887 tok/s	3,876 tok/s
Apple M4 Pro CPU, 1×512	846 tok/s	1,651 tok/s	1,011 tok/s	1,155 tok/s
Apple M4 Pro CPU, 16×1024	1,024 tok/s	1,183 tok/s	854 tok/s	934 tok/s

Quantization primarily saves storage and memory on the tested Metal backend; it was not a universal speedup. CUDA resource throughput and peak VRAM were not measured for the 480M release and are intentionally not estimated.

Tested runtimes

Validation used clean stock ggml-org/llama.cpp commit e750b887a82719c27200b71545f63ed78ec24719 (build 10763).

Runtime/backend	BF16	Q8_0	Q6_K	Q4_K_M
macOS Apple Silicon / Metal, CLI + server	tested	tested	tested	tested
macOS Apple Silicon / CPU, CLI + server	tested	tested	tested	tested
Clean Linux x86 / CUDA, CLI + server	tested	tested	tested	tested
Clean Linux x86 / CPU, CLI + server	tested	tested	tested	tested
Ollama 0.33.3 / Apple Metal	tested	tested	tested	tested

All tested lanes returned finite, unit-normalized 1024-dimensional embeddings and passed single/batch, repeat, permutation, and fresh-process checks. Remote Linux copies matched the local SHA-256 hashes exactly.

Limitations and numerical caveats

• This is a dense embedding model for retrieval, similarity, clustering, and classification. It is not a generative model or a cross-encoder reranker.
• BF16 is the frozen reference, not bit-identical to upstream PyTorch CPU/F32. The comparison recorded minimum aligned cosine 0.99782324 and maximum absolute score delta 0.00982617, concentrated in one Spanish 768-token sample; 511/512 aligned samples cleared the former 0.9985 floor.
• Cross-backend arithmetic is not bit-identical. Every artifact was stable within each backend, but the three quants exceeded a deliberately strict eight-prompt Mac-Metal-to-Linux diagnostic. This is a portability diagnostic, not corruption or a functional-runtime failure.
• In Ollama, Q8_0/Q6_K/Q4_K_M import as the exact GGUF blobs. Ollama 0.33.3 rewrote BF16 into a byte-different COPY layer of the same size with preserved model metadata, so it did not execute the frozen BF16 file byte-for-byte.
• Resource testing used context 2048. Full external retrieval used a 512-token contract; 394/56,826 RuBQ documents and 534/5,183 SciFact documents were truncated. Queries were not truncated.
• Windows, LM Studio, Jan, and older llama.cpp/Ollama builds were not tested.

Artifact integrity and provenance

File	Bytes	SHA-256
giga-embeddings-0826-480m-bf16.gguf	973,636,768	6dbe5d1121329cfd73b99fd71199541063081814c3a147a79cd1ebb1526c0cae
giga-embeddings-0826-480m-q8_0.gguf	520,209,568	f0fee824896fbb2fe3960e57434d05630037ceaa7c1edab58721cf93071a3e38
giga-embeddings-0826-480m-q6_k.gguf	403,074,208	6471e94b8be7a92d09cb253cdb73fea8fcb84bb4fef2105a772530d4c9483d0d
giga-embeddings-0826-480m-q4_k_m.gguf	325,487,776	cb9de6581c1701300fbe82df51ddd67cf8f08f558c9e8fc7b4c1f55a08a3725d

• Source: ai-sage/Giga-Embeddings-instruct-480M-0826
• Exact source revision: 2d0c1a92716eef0e5b6972df85b5883eb5b4f57a
• Source license: MIT
• Source model.safetensors SHA-256: 9ce03c6c5ae02baebb42ce3015b6f3e628c5fec7b7745bc2490f6ff961a654a5
• Source tokenizer.json SHA-256: 0870f9c0f06a677f7e939e7765ae3a5d9582f2a6d7e7258176ecd1a3eab96cef
• Source tokenizer_config.json SHA-256: 57981f331fd010c7be87f58e7570b2061dd14d98be53063ad8972e9c4db0b644
• Converter: bidirectional-model patch 409723a88b12071974ed5924a2dc1c8b2b2064f7 over upstream llama.cpp e750b887a82719c27200b71545f63ed78ec24719
• Validation workbench: 30272b4a0c2b52037a0134f9638d3a329d069e8b
• Q8_0, Q6_K, and Q4_K_M were quantized directly from the frozen BF16 GGUF; no cascade or requantization was used.
• Architecture preserved: bidirectional Qwen3, 310 tensors, mean pooling, 1024-dimensional L2-normalized output.

Machine-readable release metadata is available in manifest.json; package hashes are listed in SHA256SUMS.

Русские текстовые эмбеддинги

Локальная GGUF-версия Giga Embeddings 0826 480M для семантического поиска, RAG, сравнения текстов, кластеризации и классификации на русском и английском языках. Полное описание на русском.

Citation

Please cite both this GGUF release and the original Giga-Embeddings work:

@software{popkov2026gigaembeddings480mgguf,
  author  = {Maksim Popkov},
  title   = {Giga Embeddings 0826 480M GGUF},
  year    = {2026},
  url     = {https://huggingface.co/ai-babai/giga-embeddings-0826-480m-gguf}
}

@misc{kolodin2026gigaembeddings,
  title         = {Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings},
  author        = {Egor Kolodin and Egor Krasnoperov and Evgeniy Kosarev and Fyodor Minkin},
  year          = {2026},
  eprint        = {2608.23806},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2608.23806}
}