Instructions to use EigenLabs/Qwen3.8-27B-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use EigenLabs/Qwen3.8-27B-4bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("EigenLabs/Qwen3.8-27B-4bit") config = load_config("EigenLabs/Qwen3.8-27B-4bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use EigenLabs/Qwen3.8-27B-4bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "EigenLabs/Qwen3.8-27B-4bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "EigenLabs/Qwen3.8-27B-4bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use EigenLabs/Qwen3.8-27B-4bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "EigenLabs/Qwen3.8-27B-4bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default EigenLabs/Qwen3.8-27B-4bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use EigenLabs/Qwen3.8-27B-4bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "EigenLabs/Qwen3.8-27B-4bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "EigenLabs/Qwen3.8-27B-4bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Model card: conversion provenance, architecture, MTP head pointer, challenge role
Browse files
README.md
CHANGED
|
@@ -2,6 +2,46 @@
|
|
| 2 |
language: en
|
| 3 |
library_name: mlx
|
| 4 |
pipeline_tag: text-generation
|
|
|
|
|
|
|
| 5 |
tags:
|
| 6 |
- mlx
|
|
|
|
|
|
|
| 7 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
language: en
|
| 3 |
library_name: mlx
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
+
license: apache-2.0
|
| 6 |
+
base_model: Qwen/Qwen3.8-27B
|
| 7 |
tags:
|
| 8 |
- mlx
|
| 9 |
+
- 4-bit
|
| 10 |
+
- qwen3_5
|
| 11 |
---
|
| 12 |
+
|
| 13 |
+
# Qwen3.8-27B-4bit
|
| 14 |
+
|
| 15 |
+
A 4-bit MLX conversion of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B)
|
| 16 |
+
(revision `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0`), quantized affine 4-bit with
|
| 17 |
+
group size 64 under a pinned **mlx 0.32.0** toolchain via `mlx_lm.convert`.
|
| 18 |
+
|
| 19 |
+
## Architecture
|
| 20 |
+
|
| 21 |
+
- 64 layers on a 4-layer hybrid repeat: three linear-attention layers, then one
|
| 22 |
+
full-attention layer (`full_attention_interval: 4`)
|
| 23 |
+
- hidden size 5120, 24 attention heads / 4 KV heads, head dim 256
|
| 24 |
+
- vocabulary 248,320
|
| 25 |
+
- `model_type: qwen3_5`
|
| 26 |
+
|
| 27 |
+
## Multi-token-prediction head
|
| 28 |
+
|
| 29 |
+
The upstream checkpoint ships a native MTP (multi-token-prediction) head as 15
|
| 30 |
+
`mtp.*` tensors. Those tensors are **not** in this repository — they are published
|
| 31 |
+
separately, in their released bf16 dtype, as
|
| 32 |
+
[EigenLabs/Qwen3.8-27B-MTP-bf16](https://huggingface.co/EigenLabs/Qwen3.8-27B-MTP-bf16).
|
| 33 |
+
|
| 34 |
+
## Role
|
| 35 |
+
|
| 36 |
+
This is the pinned reference checkpoint for the Qwen 3.8 native-MTP
|
| 37 |
+
speculative-decode challenge track (`qwen3.8-27b-mtp-v1`) at
|
| 38 |
+
[Layr-Labs/qwen-3.8-mtp-challenge](https://github.com/Layr-Labs/qwen-3.8-mtp-challenge).
|
| 39 |
+
The track pins this repository by revision; scoring artifacts are byte-verified
|
| 40 |
+
against it.
|
| 41 |
+
|
| 42 |
+
## Usage
|
| 43 |
+
|
| 44 |
+
```bash
|
| 45 |
+
pip install mlx-lm
|
| 46 |
+
mlx_lm.generate --model EigenLabs/Qwen3.8-27B-4bit --prompt "Hello"
|
| 47 |
+
```
|