--- license: mit base_model: microsoft/harrier-oss-v1-270m base_model_relation: quantized pipeline_tag: feature-extraction inference: false quantized_by: shadowrock-io library_name: vllm metrics: - cosine_similarity language: - en - multilingual model-index: - name: Harrier-OSS-v1-270M-Community-FP8 results: - task: type: STS dataset: name: MTEB STSBenchmark type: mteb/stsbenchmark-sts config: default split: test revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 metrics: - type: cosine_spearman name: STSBenchmark main score (vLLM-served FP8) value: 0.81483 source: name: ShadowRock eval (raw JSON) url: https://huggingface.co/shadowrock-io/Harrier-OSS-v1-270M-Community-FP8/tree/main/eval/evidence - task: type: STS dataset: name: MTEB STS17 (multilingual, 11 language-pair subsets, mean) type: mteb/sts17-crosslingual-sts config: default split: test revision: faeb762787bd10488a50c8b5be4a3b82e411949c metrics: - type: cosine_spearman name: STS17 mean main score (vLLM-served FP8) value: 0.80983 source: name: ShadowRock eval (raw JSON) url: https://huggingface.co/shadowrock-io/Harrier-OSS-v1-270M-Community-FP8/tree/main/eval/evidence - task: type: Retrieval dataset: name: MTEB SciFact type: mteb/scifact config: default split: test revision: 0228b52cf27578f30900b9e5271d331663a030d7 metrics: - type: ndcg_at_10 name: SciFact NDCG@10 (vLLM-served FP8) value: 0.69126 source: name: ShadowRock eval (raw JSON) url: https://huggingface.co/shadowrock-io/Harrier-OSS-v1-270M-Community-FP8/tree/main/eval/evidence - task: type: Retrieval dataset: name: MTEB NFCorpus type: mteb/nfcorpus config: default split: test revision: ec0fa4fe99da2ff19ca1214b7966684033a58814 metrics: - type: ndcg_at_10 name: NFCorpus NDCG@10 (vLLM-served FP8) value: 0.30703 source: name: ShadowRock eval (raw JSON) url: https://huggingface.co/shadowrock-io/Harrier-OSS-v1-270M-Community-FP8/tree/main/eval/evidence tags: - fp8 - e4m3 - w8a8 - llm-compressor - compressed-tensors - vllm - embeddings - retrieval - semantic-search - sentence-similarity - quantized - safetensors - gemma3 - multilingual --- ShadowRock # Harrier OSS v1 270M — Community FP8 **Unofficial community quantization — not a microsoft release.** FP8 (W8A8, `FP8_DYNAMIC`) quantization of [microsoft/harrier-oss-v1-270m](https://huggingface.co/microsoft/harrier-oss-v1-270m) (revision [`31de22b`](https://huggingface.co/microsoft/harrier-oss-v1-270m/commit/31de22b673913c7d658c0f03f792d77c2dcf8ebd)), a 0.27B multilingual Gemma3TextModel embedding model using last-token pooling + L2 normalization. All credit for the model belongs to Microsoft; this repo only changes the numeric precision of the transformer's `Linear` projection weights and activations. 0.54 GB BF16 → 0.44 GB FP8, and fixture cosine parity vs the BF16 golden is 0.998 mean / 0.977 min (see Benchmarks below) with MTEB deltas under 0.005 absolute on every task measured. Pick this variant for the best fidelity/size tradeoff on a GPU that supports FP8 (Ada/Hopper/ Blackwell). The companion NVFP4 variant (in progress) trades additional fidelity for a further ~2x size reduction on Blackwell-class GPUs. ## What is quantized / Quantization details - Method: `FP8_DYNAMIC` (compressed-tensors preset) via [llm-compressor](https://github.com/vllm-project/llm-compressor) 0.13.0 — static per-channel FP8 E4M3 weight scales computed directly from the weights, dynamic per-token FP8 E4M3 activation quantization at inference. - Scope: all 126 `nn.Linear` projections inside the 18 Gemma3TextModel decoder layers (`self_attn.{q,k,v,o}_proj`, `mlp.{gate,up,down}_proj`). Left at full precision: the embedding table (`embed_tokens`), every RMSNorm module, `lm_head` (tied to `embed_tokens`, unused for embedding output but present in the checkpoint), and the sentence-transformers last-token pooling + L2-normalize heads (outside the transformer module tree, always full precision). - Calibration: 448-sample pinned manifest (`calibration/manifest.json`, seed 20260818871), bucketed by (input_type, token-length) — passages sampled from `mteb/sts17-crosslingual-sts` (disjoint from the eval fixture set), queries authored across the model's three documented instruct-prefix types (`web_search_query`/`sts_query`/`bitext_query`). Not strictly required for `FP8_DYNAMIC`'s data-free weight scales, but run for parity with the NVFP4 export path. - Full module inventory / provenance file: [`QUANT_PROVENANCE.json`](https://huggingface.co/shadowrock-io/Harrier-OSS-v1-270M-Community-FP8/blob/main/QUANT_PROVENANCE.json). - Quantize/eval scripts ship under `quantization/`, `eval/`, `calibration/`; raw eval JSON under `eval/evidence/`. ## Benchmarks / Validation Two complementary evals, both against the BF16 golden baseline computed on the same hardware (arcade, RTX 5070 Ti) with the same fixture/prompt pinning (see the BF16 golden's own `eval/README.md` for the full protocol): (1) 190-fixture cosine parity via vLLM `llm.embed()`, gated by `fixtures/tolerances.json` (schema v1, `parity_harness.py`); (2) MTEB on 4 small tasks incl. one multilingual (STS17, 11 language-pair subsets), run via a thin vLLM-backed MTEB encoder wrapper (`eval/run_mteb_vllm.py`) — **required** for this variant, since a `compressed-tensors` FP8 checkpoint cannot execute a raw `transformers`/`sentence-transformers` forward pass (needs vLLM's fused FP8 dequant kernels); confirmed by a `RuntimeError: expected mat1 and mat2 to have the same dtype` when attempted directly. | eval | metric | BF16 (vLLM) | FP8 | delta | gate | verdict | |---|---|---|---|---|---|---| | fixture parity (190 cases, non-stress) | cosine, p5 aggregate | 1.0 (self) | 0.99717 | — | ≥ 0.995 | PASS | | fixture parity (190 cases, non-stress) | cosine, per-case floor | 1.0 (self) | ≥ 0.9920 | — | ≥ 0.99 | PASS | | fixture parity — stress case (adv_long_near_max, ~6000 chars) | cosine | 1.0 (self) | 0.97724 | — | ≥ 0.95 | PASS | | MTEB STSBenchmark | main score | 0.81516 | 0.81483 | -0.00032 | ≤ 0.01 | PASS | | MTEB STS17 (mean/11 subsets) | main score | 0.81042 | 0.80983 | -0.00059 | ≤ 0.01 | PASS | | MTEB SciFact | NDCG@10 | 0.696 | 0.69126 | -0.00474 | ≤ 0.01 | PASS | | MTEB NFCorpus | NDCG@10 | 0.30675 | 0.30703 | +0.00028 | ≤ 0.01 | PASS | | MTEB macro delta (4 tasks) | mean absolute delta | — | — | 0.00089 | ≤ 0.005 | PASS | All gates pass. Fixture parity fails a naively-tight 0.999 p5 bar only on the deliberate extreme-length stress fixture and 3/190 borderline STS17 fixtures at 0.992-0.994 cosine — see `fixtures/tolerances.json` for the pre-registered-then-revised gate rationale (the original p5≥0.999 was miscalibrated before any real FP8 run existed; revised to match the house nemotron FP8 bar of 0.995 mean, applied here as a stricter p5). No case failed the revised gates. Raw result JSON: `eval/evidence/fp8_fixture_parity_report.json`, `eval/evidence/fp8_mteb.json`, `eval/evidence/golden_bf16_mteb_vllm.json`. ### Protocol deviations from the paper / base card - MTEB numbers for this variant use a custom vLLM-backed MTEB encoder wrapper (`eval/run_mteb_vllm.py`), not the base card's documented `sentence-transformers` load path — required because compressed-tensors FP8 checkpoints only run through vLLM's fused kernels. The BF16 golden's headline MTEB numbers (in the sibling BF16 golden docs) were run via `sentence-transformers` per the base card; **the BF16-vs-FP8 deltas above instead compare BF16-via-vLLM vs FP8-via-vLLM** (both in `eval/evidence/`) for an apples-to-apples backend comparison — the sentence-transformers BF16 numbers differ (e.g. NFCorpus 0.2215 vs 0.30675) because vLLM's `io_processor` plugin auto-applies the model's baked-in per-MTEB-task instruction prompts (`mteb_v2_eval_prompts.json` in the base repo) while the plain `sentence-transformers` path does not resolve those task-name-keyed prompts automatically (confirmed via mismatched- prompt-name warnings during that run) — a base-model serving nuance, not a quantization effect. - STS17/NFCorpus/SciFact dataset revisions above are the `mteb`-library-pinned snapshot hashes as resolved at eval time (2026-08-18); re-running against a newer `mteb` release may resolve different default revisions. ## Serving / Usage ```python from vllm import LLM llm = LLM( model="shadowrock-io/Harrier-OSS-v1-270M-Community-FP8", runner="pooling", # task=embed trust_remote_code=True, max_model_len=8192, # base model supports up to 32768; raise if you need longer inputs gpu_memory_utilization=0.85, # lower if sharing the GPU with other processes dtype="bfloat16", # activation/compute dtype; FP8 weights dequant/fuse automatically ) # Documents: no prefix needed. doc_emb = llm.embed(["Paris is the capital and largest city of France."]) # Queries: apply the base model's documented instruct prefix (three task-specific variants exist # in the base repo's config_sentence_transformers.json `prompts` map — this is the general one). query = "Instruct: Given a web search query, retrieve relevant passages that answer the query\\nQuery: what is the capital of France" query_emb = llm.embed([query]) ``` No `PoolerConfig` override is needed: vLLM auto-resolves `pooling_type=LAST` and L2-normalization directly from this repo's `1_Pooling/config.json` + `config_sentence_transformers.json` (carried over unchanged from the BF16 base repo) — same as the BF16 model. FP8 requires no special vLLM flags beyond loading this repo; vLLM detects the `compressed-tensors` FP8 config in `config.json` automatically and uses fused FP8 kernels on GPUs that support them (Ada/Hopper/Blackwell — tested on an RTX 5070 Ti, Blackwell sm_120). Notes that matter for correct embedding output: - Pooling is **last-token**, not mean/CLS — confirmed via `pooling_mode_lasttoken: true` in this repo's `1_Pooling/config.json` and empirically (all fixture embeddings are exactly unit-norm). - Trailing whitespace/newlines measurably shift the embedding under last-token pooling (it changes which token is literally last) — this is expected behavior for this pooling strategy on the base model, unrelated to quantization; be consistent about trimming input text across a corpus. - The query instruct-prefix meaningfully changes the embedding vs. the same text posed as a bare passage (asymmetric by design) — apply it only on the query side, never on documents. Measured on: arcade (RTX 5070 Ti, 16 GB, Blackwell sm_120), vLLM 0.26.0, llm-compressor 0.13.0, compressed-tensors 0.18.0, torch 2.11.0+cu130, mteb (pinned per `eval/evidence/*.json`). ## Caveats - The 448-sample calibration set skews toward short-to-medium sentence-length inputs (STS-style sentences + short authored queries); very long documents (thousands of tokens) were exercised only by the single adversarial stress fixture, which shows a larger (but still gated, PASS) fidelity drop (cosine 0.977) than typical-length inputs (mean 0.998) — expect somewhat more quantization noise on long-document embeddings than on short queries/passages. - MTEB coverage is 4 small tasks (STSBenchmark, STS17, SciFact, NFCorpus); the base model claims ~94 language variants and dozens of MTEB task types (classification, clustering, reranking, bitext mining) not evaluated here — only STS/retrieval-style tasks and 11 STS17 language pairs were measured. ## Intended use & limitations Intended uses are the base model's: general-purpose multilingual text embeddings for retrieval, clustering, semantic similarity, classification, bitext mining, and reranking. The base card's intended-use, safety, and coverage statements — [microsoft/harrier-oss-v1-270m](https://huggingface.co/microsoft/harrier-oss-v1-270m) — carry over unchanged; quantization alters none of the model's behavior boundaries, only its numeric precision. Our evaluation establishes parity on the benchmarks listed above and nothing beyond them: other languages, MTEB task types, and document-length regimes inherit the base model's behavior with FP8 quantization noise that we have not measured there. ## Attribution & citation Quantization, validation harness, and card by [Matt Busi](https://www.linkedin.com/in/matt-busi) ([@mattbusi](https://huggingface.co/mattbusi) on Hugging Face) at [ShadowRock](https://shadowrock.io). If you use this build, cite the microsoft base model — the underlying model architecture and training are theirs: ```bibtex @misc{harrier-oss-v1-270m, title = {Harrier OSS v1 270M}, author = {Microsoft}, year = {2026}, url = {https://huggingface.co/microsoft/harrier-oss-v1-270m} } ``` ## License MIT, inherited unchanged from the base model (see [LICENSE](LICENSE); the base repo declares `license: mit` in its card metadata but ships no LICENSE file, so the canonical MIT text is reproduced here). Community build by [ShadowRock](https://shadowrock.io); no Microsoft affiliation or endorsement. ## About ShadowRock [ShadowRock](https://shadowrock.io) is an AI-specialized systems integrator and Zendesk Premier Partner. We help businesses get real value from their go-to-market technology, from CRM and support platforms to applied AI like the models in this collection. Find us at [shadowrock.io](https://shadowrock.io) or on [LinkedIn](https://www.linkedin.com/company/shadowrock).