---
library_name: optimum-rbln
license: gemma
pipeline_tag: image-text-to-text
base_model:
  - google/gemma-4-31B-it
  - lokeshe09/gemma-4-31B-it-INT8
tags:
  - gemma4
  - multimodal
  - rbln
  - ca25
  - int8
---

# Gemma4-31B-Int8-RBLN-0.11.2

`google/gemma-4-31B-it` (multimodal, image-text-to-text), using the INT8
checkpoint from `lokeshe09/gemma-4-31B-it-INT8`, pre-compiled for Rebellions
RBLN-CA25 NPUs with `optimum-rbln` and ready to serve with `vllm-rbln`.

This repository contains the compiled artifact only. RBLN NPUs and the RBLN
SDK are required to run it.

## Hardware and software

| Component | Configuration |
|---|---|
| NPU | 16× RBLN-CA25 (language model: 16 devices; vision tower and multimodal projector: device 0, shared) |
| Compiler | `rebel-compiler==0.11.2` |
| Runtime | `optimum-rbln==0.11.2`, `vllm-rbln==0.11.2.post1`, `vllm==0.24.0`, `transformers==5.8.1` |
| Quantization | INT8 weights / FP16 activations (W8A16, weight-only) |

## Compile configuration

| Option | Value |
|---|---|
| `max_seq_len` | 262,144 |
| `batch_size` | 8 |
| `decoder_batch_sizes` | `[1, 2, 4, 8]` |
| `attn_impl` / `kvcache_partition_len` | `flash_attn` / 16,384 |
| `cache_impl` / `sliding_window` | `hybrid` / 1,024 |
| `prefill_chunk_size` | 128 |
| `kvcache_num_blocks` | 36 (589,824 shared KV tokens) |
| Vision `max_soft_tokens` / image prefill | 280 / `image_prefill_384` |

The 36 KV blocks were selected by the SDK estimator with `memory_budget=0.9`.
The shared KV capacity is approximately 2.25 full 262K contexts; shorter
requests can use the compiled dynamic batches up to batch 8.

## Serving

```bash
export VLLM_PLUGINS=rbln
export VLLM_RBLN_USE_VLLM_MODEL=0
export VLLM_RBLN_NUM_DEVICES_PER_LOCAL_RANK=16
export VLLM_RBLN_SUB_BLOCK_CACHE=false

vllm serve <this-repo-or-local-path> \
  --dtype float32 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 128 \
  --enable-chunked-prefill \
  --no-enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4
```

Notes:

- Do not pass `--tensor-parallel-size`; the 16-device split is baked into the
  compiled language model and `vllm-rbln` runs it with TP=1.
- Keep `--max-num-seqs 8` and `--max-num-batched-tokens 128` aligned with the
  compiled decoder and prefill graphs.
- Prefix caching must remain disabled. Gemma4 uses multi-group hybrid KV caches,
  which sub-block prefix caching does not support in this SDK version.
- Thinking is controlled per request with
  `chat_template_kwargs={"enable_thinking": true}`. Parsed reasoning is returned
  separately from the final response content.
- The artifact accepts one image per prompt with the compiled vision profile.

## License

Derived from `google/gemma-4-31B-it`. Use is subject to the Gemma Terms of Use.