Image-Text-to-Text
Safetensors
Trellis
exllamav3
inkling_mm_model
exl3
inkling
quantized
Mixture of Experts
multimodal
conversational
4-bit precision
Instructions to use 0xSero/Inkling-Small-EXL3-4.0bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Trellis
How to use 0xSero/Inkling-Small-EXL3-4.0bpw with Trellis:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Clarify 4.0 bpw structural and runtime validation status
Browse files
README.md
CHANGED
|
@@ -22,6 +22,10 @@ Calibrated EXL3 trellis quantization of the routed MoE experts in
|
|
| 22 |
|
| 23 |
> **Status:** The assembled weight archive has been uploaded. Consult `EXL3_MANIFEST.json` for the current structural and runtime validation state.
|
| 24 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
## What is quantized
|
| 26 |
|
| 27 |
| Component | Storage |
|
|
@@ -85,10 +89,14 @@ assembly alone.
|
|
| 85 |
|
| 86 |
The source model is loaded once for calibration. Hidden states and natural
|
| 87 |
expert assignments are captured for all forty routed layers. Each expert's
|
| 88 |
-
gate, up, and down projections are calibrated, Hadamard-transformed,
|
| 89 |
-
as EXL3/MCG trellis weights
|
| 90 |
-
|
| 91 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
|
| 93 |
The target bpw applies to routed-expert trellis weights. The complete
|
| 94 |
repository is larger than a whole-model quantization at the same nominal bpw
|
|
|
|
| 22 |
|
| 23 |
> **Status:** The assembled weight archive has been uploaded. Consult `EXL3_MANIFEST.json` for the current structural and runtime validation state.
|
| 24 |
|
| 25 |
+
> **Runtime validation:** Text generation, multimodal generation, and MTP
|
| 26 |
+
> validation are pending. The uploaded archive and its packed tensors have
|
| 27 |
+
> structural validation only.
|
| 28 |
+
|
| 29 |
## What is quantized
|
| 30 |
|
| 31 |
| Component | Storage |
|
|
|
|
| 89 |
|
| 90 |
The source model is loaded once for calibration. Hidden states and natural
|
| 91 |
expert assignments are captured for all forty routed layers. Each expert's
|
| 92 |
+
gate, up, and down projections are calibrated, Hadamard-transformed, and
|
| 93 |
+
encoded as EXL3/MCG trellis weights. The full sweep checks finite Hessians and
|
| 94 |
+
scales, exact trellis byte counts, safetensor key counts, and per-file
|
| 95 |
+
checksums. Before the sweep, a bounded H200 proof on a real Inkling expert
|
| 96 |
+
also passed trellis pack/unpack/repack equality and finite reconstruction.
|
| 97 |
+
That bounded kernel proof is not a full-model generation test. Integer-K
|
| 98 |
+
caches are reused to assemble the seven public variants without repeating
|
| 99 |
+
the full model calibration.
|
| 100 |
|
| 101 |
The target bpw applies to routed-expert trellis weights. The complete
|
| 102 |
repository is larger than a whole-model quantization at the same nominal bpw
|