Bonsai-27B LiteRT-LM — Experimental Path
Shipped artifacts
| File | Size | Cache | Notes |
|---|---|---|---|
experimental_artifacts/Qwen3.5-0.8B-hybrid-exact-c2048.litertlm |
1.08 GB | 2048 | hybrid graph proof |
experimental_artifacts/Qwen3.5-0.8B-hybrid-exact-c32k.litertlm |
1.08 GB | 32768 | long-context hybrid |
experimental_artifacts/Qwen3.5-0.8B-hybrid-exact-c64k.litertlm |
1.08 GB | 65536 | long-context hybrid |
experimental_artifacts/Bonsai-slice-4L-Q1dequant-fixed-c2048.litertlm |
3.81 GB | 2048 | Q1 weights + layout fix |
experimental_artifacts/Bonsai-slice-4L-Q1dequant-fixed-c32k.litertlm |
3.81 GB | 32768 | fixed + long ctx |
experimental_artifacts/Bonsai-slice-4L-Q1dequant-fixed-c64k.litertlm |
3.81 GB | 65536 | fixed + long ctx |
Repo: https://huggingface.co/Tdamre/Bonsai-27B-litert-lm
Breakthroughs
Hybrid GatedDeltaNet export
- Dual-state KV cache (
experimental/patch_linear_attention_cache.py) - Float-only GatedDelta inverse (
experimental/neumann_gated_delta.py) export_hf+dynamic_wi8_afp32+ external embedder- Mobile ChatML template
Bonsai Q1 real weights
- Source:
Bonsai-27B-Q1_0.gguf(3.80 GB non-ternary 1-bit) - Custom Q1_0_g128 dequant
- GGUF→HF layout fix (
experimental/fix_weight_layout.py): transpose Linears + reshape conv1d - Without layout fix, transformers reinitialized weights (garbage)
Host limits (RTX 4090 / 31 GB RAM / 24 GB swap)
| Attempt | Result |
|---|---|
| 4L fixed c2048/c32k/c64k | OK |
| 6L / 8L fixed | OOM at LiteRT converter passes |
| Full 64L | needs more RAM/disk strategy |
Quality note
Prefer *-fixed-* artifacts. 4L is a structural milestone (real Bonsai dims + Q1 weights), not full 27B chat. Vision packaging fails on qwen3_5; text+embedder graphs complete.
Run
litert-lm run \
--from-huggingface-repo Tdamre/Bonsai-27B-litert-lm \
experimental_artifacts/Bonsai-slice-4L-Q1dequant-fixed-c32k.litertlm \
--backend=cpu --max-num-tokens=256 --prompt="What is 2+2?"
Toolchain
litert-torch-nightly 0.10.0.dev20260714
ai-edge-quantizer-nightly 0.8.0
transformers (qwen3_5)
- Downloads last month
- 176
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for Tdamre/Bonsai-27B-litert-lm
Base model
Qwen/Qwen3.6-27B
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli litert-lm run \ --from-huggingface-repo=Tdamre/Bonsai-27B-litert-lm \ --prompt="Write me a poem"