Tensor G4 Edge TPU — On-device Gemma + RAG (LiteRT-LM)
Collection
int4 AOT Gemma 3 1B + on-device EmbeddingGemma RAG (Termux engine + MCP) on the Pixel 9 Tensor G4 Edge TPU via LiteRT-LM. • 2 items • Updated
Access Gemma3-1B-IT (Tensor G4 NPU) on Hugging Face
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
These are quantized, repackaged derivatives of google/gemma-3-1b-it and are governed by the Gemma license. To access them you must review and agree to Google's Gemma Terms of Use. Please ensure you are logged in to Hugging Face and click below.
Log in or Sign Up to review the conditions and access this model content.
Gemma3-1B-IT — Tensor G4 NPU (AOT, int4 per-channel + Hadamard)
A quantized, repackaged variant of
google/gemma-3-1b-itthat runs on the Google Tensor G4 Edge TPU ("rio") NPU — the SoC in the Pixel 9 / 9 Pro / 9 Pro XL / 9 Pro Fold. Packaged as a.litertlmbundle for the LiteRT-LM runtime.
This is, to our knowledge, the first publicly available int4 AOT Tensor-G4 NPU build of Gemma 3 1B —
and it is ~half the size of the only other Tensor-NPU Gemma 3 1B bundle published
(litert-community/Gemma3-1B-IT_q8_ekv1280_Google_Tensor_G5.litertlm is 1,678 MB; this build is
875 MB = 0.52×), because it is int4-per-channel rather than int8.
litert-community/Gemma3-1B-IT ships Tensor G5 and G6 builds. There is no G4 build there.
The build
| File | Gemma3-1B-IT_int4hadamard_aot_ekv4096_G4.litertlm |
| Mode | AOT — precompiled DarwiNN DGC |
| Quantization | int4 per-channel + Hadamard rotation (all FC / BMM / embed) |
| NPU share | ~100% |
| Embedded KV cache | 4096 |
| Size | 874,631,968 B (875 MB) |
26-layer Gemma 3 1B-IT, embedded KV cache, embedded tokenizer + chat template, .litertlm v1.
Built from Google's mobile-QAT checkpoint
google/gemma-3-1b-it-qat-int4-unquantized.
Performance (Pixel 9 Pro XL, Tensor G4)
Measured on this exact bundle, 2026-09-10:
| prefill | decode | prefill tok/s | decode tok/s |
|---|---|---|---|
| 128 | 64 | 608.83 | 15.93 |
| 512 | 128 | 659.22 | 16.17 |
| 1024 | 64 | 663.99 | 15.89 |
Prefill scales; decode is pinned. Prefill rises 608 → 659 → 664 tok/s as the batch grows 128 → 1024, while decode holds at 15.89–16.17 tok/s regardless (±0.9% across an 8× change in prefill length). That is a clean confirmation that decode is dispatch-bound, not compute-bound — more work per dispatch makes prefill faster, but decode always does exactly one token per dispatch, so nothing moves it.
The rise from 608 to 664 is fixed per-dispatch overhead amortizing: at 128 tokens you pay it once per 128, at 1024 once per 1024. Extrapolating, the per-dispatch prefill floor is roughly ~670 tok/s for this model.
AOT profile = prefill-monster / decode-laggard. The Edge TPU crushes batched matmuls (prefill) but a single-token decode underutilizes it — so this build is the fastest option on the phone for prefill-heavy work (RAG, long-context, summarization, classification), and a strong speculative-decode (MTP) candidate. Speculative decode is the real next perf lever and is not wired into this build.
Behaviour/"feel" differences vs a reference are runtime config (system prompt + sampling), not the quant — a faithful int4 copy matches the reference once system prompt and top-k / top-p / temperature match.
Quantization
The rank-2 FC surgery — the key enabler (affects G4 and G5)
The DarwiNN AOT compiler OP_FAILEDs on rank-3 FULLY_CONNECTED ops, which Gemma-3 emits ~360× — so
the AOT compile fails outright without a fix. recipe/fc_rank2_surgery.py wraps every rank-3 FC as
RESHAPE → (rank-2) FC → RESHAPE (same math, a rank the compiler accepts). This is what lets a full int4
Gemma 3 actually compile to a single G4 DGC. We are not aware of an upstream fix.
Details in RANK2_FC_SURGERY.md.
Hadamard rotation
On top of int4 per-channel, this build applies a Hadamard rotation to the weights before quantization
(a DECOMPOSED_HADAMARD_ROTATION-style outlier-suppression transform). Spreading outliers across the
rotated basis tightens the per-channel int4 grid — most visibly on the 262k-vocab head, where it improves
logit precision and coherence over plain per-channel int4, at a small size cost.
Per-quant AOT support is compiler-build-dependent
There is no fixed per-bit-width support matrix for the DarwiNN AOT compiler — what a given toolchain
build accepts varies by compiler version. The shipping gemma4-E2B G4-NPU bundle is a 2-subgraph model
carrying int2, int4, and int8 weight tensors (verified directly), so mixed int2/int4/int8 does
AOT-compile. The verified enabler for compiling a full Gemma to a G4 DGC is the rank-2 FC surgery,
not any particular bit width.
Granularity is a build choice, not a compiler wall:
- int4 per-channel + Hadamard (one scale per output channel, rotated) → this build.
- int4 per-channel (one scale per output channel) → also coherent.
- int4 per-tensor also AOT-compiles, but one scale over the 262k-vocab head → word salad.
Running
This is a precompiled DarwiNN DGC. It requires a compatible GoogleTensor dispatch library to load on-device; a stock app's dispatch may reject it as a firmware mismatch. Runtime/dispatch specifics are out of scope for this repo.
| Surface | Support |
|---|---|
LiteRT-LM CLI (--backend=npu) |
✅ with a compatible dispatch runtime |
| Google AI Edge Gallery (NPU) | needs a compatible dispatch runtime |
| Tensor G4 (Pixel 9 series) | ✅ |
| Tensor G5 (Pixel 10) | ❌ DGC is G4-targeted |
| Older Tensor (G2 / G3) | ❌ |
| llama.cpp / GGUF, Web, iOS, desktop, non-Pixel | ❌ .litertlm is Pixel-NPU-only |
Files
| File | Purpose |
|---|---|
Gemma3-1B-IT_int4hadamard_aot_ekv4096_G4.litertlm |
the model — per-channel int4 + Hadamard, 4096 ctx, ~100% NPU |
RANK2_FC_SURGERY.md |
the rank-2 FC surgery writeup + why per-quant AOT support is compiler-build-dependent |
recipe/fc_rank2_surgery.py |
rank-3 → rank-2 FC surgery (the DarwiNN AOT enabler) |
MANIFEST.md |
checksum + size |
License & attribution
Base model Gemma 3 1B-IT © Google, under the Gemma Terms of Use. This quantized/repackaged derivative inherits the Gemma license. Runtime: LiteRT-LM / LiteRT (Apache-2.0). Quantization, rank-2 FC surgery, per-channel int4 + Hadamard AOT compile, and G4 NPU packaging by @xThr45hx.
- Downloads last month
- 25