Image-Text-to-Text
MLX
Safetensors
English
Korean
Chinese
inkling_mm_model
audio-text-to-text
quantized
mixture-of-experts
Mixture of Experts
multimodal
apple-silicon
mixed-precision
3-bit
dwq
conversational
Instructions to use avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw") config = load_config("avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "avlp12/Inkling-975B-Alis-MLX-Dynamic-3.7bpw" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -24,17 +24,17 @@ base_model: thinkingmachines/Inkling
|
|
| 24 |
|
| 25 |
# Inkling-975B-Alis-MLX-Dynamic-3.7bpw
|
| 26 |
|
| 27 |
-
> The **quality / golden-spot** tier of the Inkling · Alis MLX Dynamic family — siblings: a ~2.7 bpw size-optimal build and a ~6.5 bpw two-box Q6 performance build (uploading as their certification completes).
|
| 28 |
|
| 29 |
**Apple Silicon (MLX) mixed-precision quantization of [thinkingmachines/Inkling](https://huggingface.co/thinkingmachines/Inkling)** — a 975B-class multimodal Mixture-of-Experts model (66 hybrid decoder layers, **256 routed experts (top-6) + 2 shared** per MoE layer, hidden 6144, sliding-window attention + short-convolution hybrid, vision + audio front-ends, 201K vocab).
|
| 30 |
|
| 31 |
-
This build targets the **golden spot for a single 512 GB M3 Ultra: near-teacher quality in one box** — ~3.71 bits/weight, with the entire non-expert skeleton kept at **exact BF16**, and quantization scales certified by a **fully receipt-sealed, layer-local ALIS-DWQ pass** run as a distributed two-Mac pipeline.
|
| 32 |
|
| 33 |
---
|
| 34 |
|
| 35 |
## Quality-first recipe
|
| 36 |
|
| 37 |
-
Bits go where the parameters are; exactness stays where the control flow is. ~96% of weights (the routed expert bank) take the cheap bits; **everything a token's routing, attention, or modality path depends on stays BF16** — a stricter split than most mixed builds, which quantize attention too.
|
| 38 |
|
| 39 |
| Component | Precision | Share | Why |
|
| 40 |
|---|---|---|---|
|
|
@@ -62,7 +62,7 @@ Bits go where the parameters are; exactness stays where the control flow is. ~96
|
|
| 62 |
Most public quants ship converted weights. This one ships converted weights **plus a machine-checkable certification trail**. The quantization scales/biases were passed through **layer-local ALIS-DWQ** ([alis-dwq](https://github.com/avlp12/alis-dwq)) against exact **BF16 teacher activations**, under a guard-and-receipt harness originally built for reproducible two-box runs:
|
| 63 |
|
| 64 |
- **BF16 teacher boundaries, not logits-only:** the full-precision teacher was run as a **distributed pipeline across two 512 GB Macs** (layers 0–32 / 33–65), dumping per-layer input/target activations (h₀…h₆₆) for 72 calibration batches across **text, image, and audio** — so every decoder layer trains against its own exact teacher boundary, per modality.
|
| 65 |
-
- **Layer-local, memory-bounded training:** each layer is strict-loaded alone (never the full model) and its affine scales/biases tuned with stop-gradient teacher boundaries and valid-token NMSE. Differentiating a 256-expert quantized gather naively materializes ~150 GiB of dequantized workspace; this pass uses an **expert-group × token-block serialized backward** that provably matches the fused gradients (cosine ≥ 0.9985) at a **~23 GiB peak** — the whole optimization ran inside a watchdog envelope of *min 90% system RAM free, zero swap growth*.
|
| 66 |
- **Do-no-harm acceptance, per layer:** a layer's new scales are kept **only if held-out boundary NMSE does not regress at all** (allowed regression: 0.0); otherwise the layer rolls back to baseline, byte-exact. Of 63 tuned layers, 62 certified neutral and **layer 40 committed a genuine improvement** — that accepted delta is what distinguishes these weights from the raw conversion.
|
| 67 |
- **Every step is evidence:** each layer ran as a sealed process chain (frozen runtime bundle → launch pin → guarded watchdog → atomic no-clobber receipts), and the 63 terminal receipts link into a hash chain closed by an **`advanced-completion` receipt (status: pass)**. Nothing in this repo was produced by an unaudited script run.
|
| 68 |
|
|
@@ -75,8 +75,8 @@ The result is conservative by construction: **you get provably-not-worse-than-ba
|
|
| 75 |
| Build | bpw | Size | For |
|
| 76 |
|---|---|---|---|
|
| 77 |
| **this** — quality / golden spot | **3.71** | 409 GiB | single 512 GB Mac, best quality in one box |
|
| 78 |
-
| capacity (uploading next) | ~2.72 | 299 GiB | single Mac with generous headroom / smaller boxes |
|
| 79 |
-
| Q6 teacher (two-box) | ~6.5 | ~790 GiB | maximum fidelity, 2 × 512 GB pipeline serving |
|
| 80 |
|
| 81 |
---
|
| 82 |
|
|
|
|
| 24 |
|
| 25 |
# Inkling-975B-Alis-MLX-Dynamic-3.7bpw
|
| 26 |
|
| 27 |
+
> The **quality / golden-spot** tier of the Inkling · Alis MLX Dynamic family — siblings: a \~2.7 bpw size-optimal build and a \~6.5 bpw two-box Q6 performance build (uploading as their certification completes).
|
| 28 |
|
| 29 |
**Apple Silicon (MLX) mixed-precision quantization of [thinkingmachines/Inkling](https://huggingface.co/thinkingmachines/Inkling)** — a 975B-class multimodal Mixture-of-Experts model (66 hybrid decoder layers, **256 routed experts (top-6) + 2 shared** per MoE layer, hidden 6144, sliding-window attention + short-convolution hybrid, vision + audio front-ends, 201K vocab).
|
| 30 |
|
| 31 |
+
This build targets the **golden spot for a single 512 GB M3 Ultra: near-teacher quality in one box** — \~3.71 bits/weight, with the entire non-expert skeleton kept at **exact BF16**, and quantization scales certified by a **fully receipt-sealed, layer-local ALIS-DWQ pass** run as a distributed two-Mac pipeline.
|
| 32 |
|
| 33 |
---
|
| 34 |
|
| 35 |
## Quality-first recipe
|
| 36 |
|
| 37 |
+
Bits go where the parameters are; exactness stays where the control flow is. \~96% of weights (the routed expert bank) take the cheap bits; **everything a token's routing, attention, or modality path depends on stays BF16** — a stricter split than most mixed builds, which quantize attention too.
|
| 38 |
|
| 39 |
| Component | Precision | Share | Why |
|
| 40 |
|---|---|---|---|
|
|
|
|
| 62 |
Most public quants ship converted weights. This one ships converted weights **plus a machine-checkable certification trail**. The quantization scales/biases were passed through **layer-local ALIS-DWQ** ([alis-dwq](https://github.com/avlp12/alis-dwq)) against exact **BF16 teacher activations**, under a guard-and-receipt harness originally built for reproducible two-box runs:
|
| 63 |
|
| 64 |
- **BF16 teacher boundaries, not logits-only:** the full-precision teacher was run as a **distributed pipeline across two 512 GB Macs** (layers 0–32 / 33–65), dumping per-layer input/target activations (h₀…h₆₆) for 72 calibration batches across **text, image, and audio** — so every decoder layer trains against its own exact teacher boundary, per modality.
|
| 65 |
+
- **Layer-local, memory-bounded training:** each layer is strict-loaded alone (never the full model) and its affine scales/biases tuned with stop-gradient teacher boundaries and valid-token NMSE. Differentiating a 256-expert quantized gather naively materializes \~150 GiB of dequantized workspace; this pass uses an **expert-group × token-block serialized backward** that provably matches the fused gradients (cosine ≥ 0.9985) at a **\~23 GiB peak** — the whole optimization ran inside a watchdog envelope of *min 90% system RAM free, zero swap growth*.
|
| 66 |
- **Do-no-harm acceptance, per layer:** a layer's new scales are kept **only if held-out boundary NMSE does not regress at all** (allowed regression: 0.0); otherwise the layer rolls back to baseline, byte-exact. Of 63 tuned layers, 62 certified neutral and **layer 40 committed a genuine improvement** — that accepted delta is what distinguishes these weights from the raw conversion.
|
| 67 |
- **Every step is evidence:** each layer ran as a sealed process chain (frozen runtime bundle → launch pin → guarded watchdog → atomic no-clobber receipts), and the 63 terminal receipts link into a hash chain closed by an **`advanced-completion` receipt (status: pass)**. Nothing in this repo was produced by an unaudited script run.
|
| 68 |
|
|
|
|
| 75 |
| Build | bpw | Size | For |
|
| 76 |
|---|---|---|---|
|
| 77 |
| **this** — quality / golden spot | **3.71** | 409 GiB | single 512 GB Mac, best quality in one box |
|
| 78 |
+
| capacity (uploading next) | \~2.72 | 299 GiB | single Mac with generous headroom / smaller boxes |
|
| 79 |
+
| Q6 teacher (two-box) | \~6.5 | \~790 GiB | maximum fidelity, 2 × 512 GB pipeline serving |
|
| 80 |
|
| 81 |
---
|
| 82 |
|