# Qwen3-Embedding-4B — RBLN (ATOM CA25) compiled

`Qwen/Qwen3-Embedding-4B` pre-compiled for Rebellions RBLN-CA25 NPUs with
`optimum-rbln`, ready to serve with `vllm-rbln`. This repository contains the
compiled artifact only — RBLN NPUs and the RBLN SDK are required to run it.

## Hardware & software

| Component | Version / configuration |
|---|---|
| NPU | 2× RBLN-CA25 chiplets (32 GiB total; half of one 64 GiB CA25 card) |
| Compiler | `rebel-compiler==0.11.2` |
| Runtime | `optimum-rbln==0.11.2`, `vllm-rbln==0.11.2.post1` |
| Framework | `transformers==5.8.1`, `vllm==0.24.0+cpu` |

## Compile configuration

| Option | Value |
|---|---:|
| `max_seq_len` | 40,960 |
| `batch_size` | 1 |
| `num_devices` | 2 |
| `attn_impl` | `flash_attn` |
| `kvcache_partition_len` | 8,192 |
| `prefill_chunk_size` | 128 |
| `kvcache_num_blocks` | 5 (one complete 40,960-token sequence) |
| phases | `prefill` only |

The batch size is intentionally one. In the current Optimum pooling path,
larger batches are dispatched through a Python-side per-item prefill loop and
do not improve NPU throughput. A 64 GiB CA25 card can host two independent
2-chiplet replicas for real request-level parallelism.

## Serving

```bash
vllm serve <this-repo-or-local-path> \
  --runner pooling \
  --max-num-seqs 1 \
  --max-model-len 40960 \
  --no-enable-prefix-caching
```

Do not pass `--tensor-parallel-size 2`; the two-device split is already baked
into the compiled model and `vllm-rbln` uses tensor parallel size 1 for the
serving process.

The OpenAI-compatible embedding endpoint is `/v1/embeddings`. Embeddings have
2,560 dimensions and are returned L2-normalized.

## License

Derived from `Qwen/Qwen3-Embedding-4B`; use is subject to the original model's
license and terms.