heswithme/Huihui-Qwen3.8-27B-Abliterated-Gittensor-Style-NVFP4-RTX5090

Qwen3.8-27B NVFP4 RTX5090 — Huihui Abliterated

The goal of this release was simple: take the original BF16 Huihui abliterated weights and quantize them with the same practical NVFP4 layout used by Gittensor's RTX 5090 release.

This is an independent derivative, not an official Huihui, Gittensor, Qwen, NVIDIA, or SGLang release.

Provenance

Role	Repository	Pinned revision
BF16 source	huihui-ai/Huihui-Qwen3.8-27B-abliterated	739e3c5b89849f6c238ce1e5b70008612ae42cdd
NVFP4 recipe/layout reference	gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090	b8ca3826548c9a7735642feb05c3c473f1fede1f
Quantizer	NVIDIA Model Optimizer	c4129b6e03d3c564e04359e6d0c6057c9a59183f
Qualified optional drafter	gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4	eba1ac5a66c74902eaa95a4000a7c5eda96d8e95

What was done

1. Downloaded the pinned raw BF16 Huihui checkpoint: 18 shards and 55,563,006,216 weight-file bytes.
2. Removed the unchanged native MTP head from the deployment view; production uses either normal autoregressive decoding or an external drafter.
3. Ran NVIDIA ModelOpt PTQ with Gittensor's Qwen3.8 all-linear NVFP4 recipe:
   • NVFP4 W4A4, group size 16;
   • 128 image-text calibration samples;
   • calibration sequence length 512;
   • vision, embeddings, and Gated-DeltaNet conv1d, in_proj_a, and in_proj_b kept out of NVFP4.
4. Normalized the output to Gittensor's tensor names, shapes, dtypes, shard map, and quantization metadata.
5. Used Gittensor's NVFP4 lm_head byte-for-byte. Huihui states that lm_head was not modified, so this preserves the qualified Gittensor head rather than independently recalibrating it.
6. Used Gittensor's chat template and tokenizer metadata so stock and abliterated deployments have the same serving format.
7. Verified the finished artifact against the pinned Gittensor layout.

All model tensors other than the explicitly described lm_head come from a fresh PTQ export of the pinned Huihui source.

Huihui's card says layers 18–51 were ablated. Direct tensor comparison found changed zero-based mlp.down_proj layers 17–51. Both observations are recorded without trying to reinterpret Huihui's numbering convention.

Artifact checks

• 2 safetensor shards
• 2,387 tensors
• 17,915,815,528 tensor payload bytes
• 17,916,112,584 safetensor file bytes
• no MTP tensors
• exact Gittensor tensor layout and quantization exclusions
• SHA256 hashes in SHA256SUMS
• machine-readable details in PROVENANCE.json

Run with SGLang: native 262k, no speculation

This is the cleanest way to run the model. The command below was qualified with one RTX 5090 32 GB and one request:

MODEL_DIR=/path/to/Qwen3.8-27B-NVFP4-RTX5090-Gittensor-Abliterated-Huihui

docker run --rm --gpus all --ipc=host --shm-size=32g \
  -p 127.0.0.1:8000:30000 \
  -v "$MODEL_DIR:/model:ro" \
  --entrypoint python3 \
  lmsysorg/sglang@sha256:febfb971c7352570fc445c466ebd6ffc9d896024958e544a60f2137fd85856b1 \
  -m sglang.launch_server \
    --model-path /model \
    --served-model-name qwen3.8-27b-abliterated-nvfp4 \
    --trust-remote-code --tp-size 1 \
    --host 0.0.0.0 --port 30000 \
    --context-length 262144 \
    --kv-cache-dtype fp8_e4m3 \
    --attention-backend flashinfer \
    --chunked-prefill-size 1024 \
    --disable-prefill-cuda-graph \
    --cuda-graph-max-bs-decode 1 \
    --mamba-radix-cache-strategy extra_buffer_lazy \
    --mamba-ssm-dtype bfloat16 \
    --max-mamba-cache-size 5 \
    --mem-fraction-static 0.93 \
    --max-running-requests 1 \
    --reasoning-parser qwen3 \
    --default-chat-template-kwargs '{"reasoning_effort":"xhigh","preserve_thinking":true}' \
    --tool-call-parser qwen3_coder \
    --mm-feature-transport cpu \
    --language-only \
    --enable-metrics

On WSL with NVIDIA Container Toolkit CDI, replace --gpus all with --device nvidia.com/gpu=all if required.

Faster agent setup: SGLang + DSpark

For faster decoding, download the separate Gittensor DSpark checkpoint linked above and add these flags to the same SGLang command:

-v /path/to/Qwen3.8-27B-DSpark-NVFP4:/model_dspark:ro

--context-length 202000 \
--speculative-algorithm DSPARK \
--speculative-draft-model-path /model_dspark \
--speculative-dspark-block-size 7 \
--speculative-draft-model-quantization modelopt_fp4 \
--speculative-draft-attention-backend flashinfer

Our qualified single-agent client budget is:

shared context: 202,000
max output:      32,000
prompt + tools:  170,000
physical pool:   203,125

It is 170k, not 180k, because output, reasoning, prompts, tool schemas, and tool results all share the same 202k sequence budget.

The qualified drafter is DSpark, not DFlash. DFlash was tested separately but was not the recommended production configuration.

Arbitrage implementation benchmark

This was an end-to-end implementation task, not a token-speed microbenchmark. Starting from a repository of stubs, the agent had to build:

• a fee-aware XYK WETH/USDC AMM;
• a two-coin StableSwap USDC/USDT AMM with robust invariant and post-swap solvers;
• an infinite-depth WETH/USDT CEX leg;
• both atomic AMM-AMM-CEX triangular arbitrage directions;
• a bounded global size optimizer with a strict objective-call budget;
• stateful event processing, execution, PnL and reserve accounting, deterministic trade-log hashing, CLI support, tests, and source documentation.

A private grader checked AMM math on difficult states, optimizer regret and evaluation counts, then replayed every submitted arbitrage action through an independent transition model.

Agent configuration	Fair grade	Wall time	Median decode
target-only, medium, 262k / 32k output	99.375/100	53m09s	72.1 tok/s
DSpark, medium, 202k / 32k output	100/100	21m26s	148.2 tok/s
DSpark, xhigh, 202k / 32k output	99.375/100	15m52s	154.6 tok/s

Both DSpark runs earned full correctness and optimization marks. The xhigh run lost 0.625 points only for its numerical-method source selection. It had no output-limit stop; a one-shot length-recovery policy was enabled but did not fire. These are single runs on one implementation benchmark, not a general leaderboard claim.

Limitations

• Abliteration reduces refusal behavior and safety filtering. Treat outputs as untrusted.
• The 262,144-token value is the target-only architectural/qualified serving window. External speculation uses extra memory and reduces the practical shared window on a 32 GB card.
• Speculative decoding accelerates decode, not cold prefill.
• The qualified deployment was language-only; multimodal behavior was not evaluated here.
• Do not expose the unauthenticated API publicly.

License

Apache License 2.0. See LICENSE.

Credits: Qwen team; Huihui AI for the abliterated source; Gittensor Model Hub for the NVFP4 recipe/layout and DSpark; NVIDIA for Model Optimizer. Independent quantization, validation, and packaging by heswithme.