Instructions to use sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8 with MLX:

# Make sure mlx-vlm is installed
# pip install --upgrade mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model, processor = load("sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8")
config = load_config("sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8")

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

# Generate output
output = generate(model, processor, formatted_prompt, image)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8 with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent

How to use sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8 with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8 with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Huihui-Qwen3.8-27B-abliterated-MXFP8 / README.md

sahilchachra

Upload README.md with huggingface_hub

0d9afd6 verified 10 days ago

preview code

Raw

History Blame Contribute Delete

4.28 kB

	---
	license: apache-2.0
	base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
	base_model_relation: quantized
	pipeline_tag: image-text-to-text
	library_name: mlx
	tags:
	- mlx
	- qwen3_5
	- image-text-to-text
	- mxfp8
	- apple-silicon
	- uncensored
	---

	# Huihui-Qwen3.8-27B-abliterated — MLX MXFP8

	MLX MXFP8 quantization of [`huihui-ai/Huihui-Qwen3.8-27B-abliterated`](https://huggingface.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated),
	uncensored via abliteration (refusal-direction removal on text layers 18–51; the README notes MTP and the vision tower were left unmodified) of Qwen3.8-27B. Qwen3.8-27B is a `qwen3_5` vision-language model with
	a hybrid GatedDeltaNet linear-attention + full-attention text backbone (64
	layers, full attention every 4th) and a Qwen3-VL vision tower. Runs on Apple
	Silicon via [mlx-vlm](https://github.com/Blaizzy/mlx-vlm). Stays
	image-text-to-text — the vision tower is kept in bf16; only the text backbone
	is quantized.

	\| \| \|
	\| :---- \| :---- \|
	\| Precision \| MXFP8 (E4M3 + E8M0 shared scale, group size 32) \|
	\| Bits per weight \| 8.381 bpw \|
	\| On-disk size \| 27 GB (27 shards) \|
	\| Quantized \| text backbone (incl. the ~1.27B `lm_head`) \|
	\| Kept in bf16 \| Qwen3-VL vision tower \|

	## Quantizations

	\| Variant \| Bits \| Size \| \|
	\| :---- \| :---- \| :---- \| :---- \|
	\| [`Huihui-Qwen3.8-27B-abliterated-MXFP4`](https://huggingface.co/sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP4) \| 4.449 bpw \| 14 GB \| smaller / for 16 GB+ \|
	\| [`Huihui-Qwen3.8-27B-abliterated-MXFP8`](https://huggingface.co/sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8) \| 8.381 bpw \| 27 GB \| ← this repo \|

	## Verification

	This higher-fidelity build was verified structurally (correct tensor shapes,
	`format: mlx` metadata, consistent shard index, vision tower intact in bf16). Full
	token-by-token generation was not benchmarked on the 24 GB test machine because
	27 GB exceeds its RAM; on a 32 GB+ Mac it runs at normal speed. Since MXFP8 uses
	more bits than the **MXFP4 build — which passed text + vision smoke tests
	end-to-end** — it is at least as faithful to the base model.

	See the [MXFP4 build](https://huggingface.co/sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP4) for the
	full generation/vision smoke-test results.

	## Usage (mlx-vlm)

	```bash
	pip install -U mlx-vlm # needs the qwen3_5 architecture (>= 0.6.12)
	```

	```python
	from mlx_vlm import load, generate
	from mlx_vlm.prompt_utils import apply_chat_template

	model, processor = load("sahilchachra/Huihui-Qwen3.8-27B-abliterated-MXFP8")
	config = model.config

	prompt = apply_chat_template(processor, config, "What is the capital of France?")
	print(generate(model, processor, prompt, max_tokens=256, verbose=True))
	```

	This is a reasoning model; it produces a `<think>` channel before its answer.

	## Run in LM Studio

	Loads and runs in LM Studio (tested on 0.4.20, mlx-llm runtime): the
	`qwen3_5` architecture is recognized, the model indexes cleanly (`format: mlx`
	present), and the ChatML template runs as-is. This is a reasoning model — it
	emits a thinking channel (`reasoning_content`) before the final `content`, so give
	it enough `max_tokens` (e.g. 200+) or the answer can be empty while it is still
	thinking. Verified: "capital of France" → reasoning + `content` = "Paris".

	Note: at 27 GB this needs a 32 GB+ Mac to load under LM Studio's default guardrails.

	## Notes & limitations

	- Uncensored model. This is a deliberately uncensored/abliterated derivative and
	will not refuse requests the way the original might. Use responsibly and in line
	with the base model's license and your local laws.
	- MTP head dropped. The base model's multi-token-prediction (speculative
	decoding) head is not included — mlx-vlm strips the `mtp.*` weights on load. Text
	and vision are unaffected; only spec-decode is not available.
	- `tie_word_embeddings=false`, so the ~1.27B `lm_head` is a separate matrix and is
	quantized. Verified correct on the MXFP4 build above.
	- Inherits all capabilities and limitations of the base model. See the
	[original model card](https://huggingface.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated).
	- Quantized by [@sahilchachra](https://huggingface.co/sahilchachra) with MLX.
	Apache-2.0.