N8Programs's picture
Add files using upload-large-folder tool
aa17594 verified
|
Raw
History Blame
3.69 kB
metadata
library_name: transformers
license: apache-2.0
license_link: >-
  https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/995ad96eacd98c81ed38be0c5b274b04031597b0/LICENSE
base_model: Qwen/Qwen3.6-35B-A3B
pipeline_tag: image-text-to-text
tags:
  - qwen3.6
  - moe
  - lora
  - merged
  - antidoom

Qwen3.6-35B-A3B-AntiLoop

This is a BF16 merged-weight variant of Qwen/Qwen3.6-35B-A3B, fine-tuned to recover from pathological self-verification and enumeration loops while preserving ordinary long-form reasoning.

The artifact contains a full Transformers checkpoint; no PEFT adapter is needed at inference time. It retains the base model's multimodal architecture, tokenizer, chat template, MTP weights, and 262,144-token native context length.

Provenance

  • Base: Qwen/Qwen3.6-35B-A3B
  • Pinned base revision: 995ad96eacd98c81ed38be0c5b274b04031597b0
  • Fine-tuning: rank-32 LoRA, alpha 64, dropout 0
  • Targets: 190 attention-side projections across all 40 language-model layers (150 Gated DeltaNet projections and 40 full-attention projections)
  • Objective: 0.9 * supervised CE + 0.1 * KL(base || tuned)
  • Training: one epoch / 178 optimizer steps, seed 0
  • Training run: n8programs/qwen-antidoom, run dw7mbmce

The LoRA was merged as BF16(W_FP32 + B_FP32 @ (A_FP32 * 2)), one target tensor at a time in small row tiles. merge_manifest.json records the exact adapter hash, base revision, environment, per-target effective deltas, and every output-shard SHA-256. merge_verification.json certifies that all 190 intended tensors equal that formula bit-for-bit after BF16 rounding, every target differs from the base, and every non-target byte is unchanged.

Evaluation

The behavior evaluations below used the same round-2 adapter served on the Qwen3.6-35B-A3B FP8 base with vLLM. They validate the adapter behavior; the merged BF16 artifact was separately validated at the tensor level.

Evaluation Base/control AntiLoop
Held-out enumeration-loop rate (285 prompts, calibrated judge) 25% 2%
Answer delivery on the same prompts 77% 98%
GPQA Diamond, thinking enabled 167/198 (84.3%) 166/198 (83.8%)

The paired GPQA difference was -0.5 percentage points (exact two-sided McNemar p = 1.0), consistent with preserved capability at this sample size. The loop evaluation is task-specific and judge-based; it should not be read as a general safety or truthfulness guarantee.

Usage

Use a recent Transformers release with Qwen3.6 support:

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "N8Programs/Qwen3.6-35B-A3B-AntiLoop"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
)

The checkpoint can also be served directly with recent vLLM/SGLang builds that support the Qwen3.6 MoE architecture. Follow the upstream Qwen3.6 model card for chat templating, thinking-mode controls, multimodal inputs, deployment details, and the base model's limitations.

Limitations

  • This is a narrow behavioral fine-tune, not a general alignment or safety model.
  • The merged BF16 checkpoint has not been exhaustively evaluated across every language, modality, tool-use setting, or long-context regime.
  • Quantization can change behavior; validate the exact deployed quantization.
  • Outputs may still be incorrect, overconfident, repetitive, or unsafe.

License

Apache 2.0, following the base checkpoint. See LICENSE and the upstream Qwen3.6-35B-A3B model card.