MiniCPM-V-4.6-35B-Abliterated

A multimodal vision-language model combining:

• Vision: openbmb/MiniCPM-V-4.6 vision tower (SigLIP 400M, 27 encoder layers + ViT merger)
• Language: huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated (Qwen3.5-35B-A3B MoE, abliterated for uncensored text generation)
• Merger: Trained MLP bridge (4608→2048) connecting vision to language, trained across a 4-stage curriculum (pretrain → instruction-tune → UI grounding → SFT polish)

⚙️ Latest update (2026-06-05): model weights now ship with the Stage 4 / step 23706 trained merger (val_loss 1.0280). See Training below.

Quick Stats

Property	Value
Total params	~35B (3B active per token, MoE 256 experts × 8 active)
Vision tower	1152 hidden, 27 layers, SigLIP-400M + ViT merger
Merger	4608 → 2048, single DownsampleMLP (LayerNorm → Linear → GELU → Linear)
LLM hidden size	2048
Precision	BF16 (also ships Q4_K_M GGUF for llama.cpp)
Disk	65.6 GB safetensors, 21.2 GB GGUF Q4_K_M
Inference	~75 GB BF16 VRAM, ~28 GB NF4 (bitsandbytes)

How It Was Made

1. Assembled by splicing weights:
   • Vision tower + ViT merger from openbmb/MiniCPM-V-4.6
   • Language model + lm_head from huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated
   • Merger MLP Xavier-initialized
2. Merger trained in 4 stages on NVIDIA GB10 (128 GB unified memory) using a custom standalone trainer (~2.4 GB GPU footprint — only vision tower + merger + embed_tokens loaded, LLM frozen).

Training

The merger MLP (~22 tensors, 270 MB) was trained in a 4-stage curriculum. Vision tower and LLM remain frozen throughout. Each stage uses next-token cross-entropy loss against the LLM's frozen embedding/lm_head (Stages 2-4) or proxy MSE loss (Stage 1).

Stage Curriculum

Stage	Name	Dataset	Steps	LR	Final val_loss
1	Pretrain Alignment	LLaVA-Pretrain (558K image-caption)	558,128	2e-4	— (MSE only)
2	Instruction Tuning	LLaVA-Instruct-150k	157,712	5e-5	1.0855
3	UI Grounding	ScreenSpot + UI variants	1,211	5e-5	4.1910
4	SFT Polish	agentsea/wave-ui-25k	23,706	5e-5	1.0280 ✓

Each stage trains on top of the previous stage's converged merger.

Stage 4 Eval Trajectory (the shipped checkpoint)

Step	val_loss
16000	1.0318
18000	1.0302
20000	1.0286
22000	1.0282
23706	1.0280 ✓ shipped

Validation loss is monotonically decreasing; the final checkpoint is the best by val_loss and is the one packaged in model-00015-of-00015.safetensors and mmproj-stage4.gguf.

Hyperparameters (all stages)

• Optimizer: AdamW (β1=0.9, β2=0.999, weight_decay=0.01)
• LR schedule: linear warmup + cosine decay
• Gradient clipping: max_norm=1.0
• Batch size: 1 with grad accumulation (Stages 1: 12 / 2: 16 / 3: 8 / 4: 8)
• Hardware: NVIDIA GB10 (128 GB unified memory)
• Only merger + ViT-merger weights trained; vision tower core and LLM frozen

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoProcessor
from PIL import Image

model = AutoModelForCausalLM.from_pretrained(
    "jduartedj/MiniCPM-V-4.6-35B-Abliterated",
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(
    "jduartedj/MiniCPM-V-4.6-35B-Abliterated",
    trust_remote_code=True,
)

image = Image.open("your_image.jpg").convert("RGB")
messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": "Describe this image in detail."},
    ]},
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(output[0], skip_special_tokens=True))

NF4 (bitsandbytes) for ~28 GB inference

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "jduartedj/MiniCPM-V-4.6-35B-Abliterated",
    trust_remote_code=True,
    quantization_config=bnb,
    device_map={"": 0},
    dtype=torch.bfloat16,
    # CRITICAL: keep vision tower, ViT merger, projector merger, and lm_head
    # in bf16. Skipping only via llm_int8_skip_modules silently fails on the
    # 4-bit code path for sub-module dotted names. See caveat below.
    modules_to_not_convert=[
        "merger",                   # model.merger.mlp.0.*
        "vision_tower.vit_merger",  # model.vision_tower.vit_merger.*
        "lm_head",
    ],
)

⚠️ NF4 loader caveat — use modules_to_not_convert. BitsAndBytesConfig(llm_int8_skip_modules=[...]) alone is not honored on the 4-bit code path for sub-module dotted names. Without the kwarg below, 8 of the 22 merger Linear layers get silently wrapped as bitsandbytes.Linear4bit, and any post-load overlay (state_dict.copy_(...)) will silently fail on the 4-bit-packed .weight tensors. The fix is to pass modules_to_not_convert=["merger", "vision_tower.vit_merger", "lm_head"] directly to from_pretrained, as shown in the NF4 snippet above. The shipped weights already contain the trained merger, so no overlay is needed — the kwarg above is enough.

llama.cpp

# Use the matched stage-4 mmproj:
./llama-mtmd-cli \
  -m ggml-model-Q4_K_M.gguf \
  --mmproj mmproj-stage4.gguf \
  --image your_image.jpg \
  -p "Describe this image."

Requirements

• transformers >= 5.7.0 (native minicpmv4_6 support)
• torch >= 2.1.0
• torchvision
• bitsandbytes >= 0.43 (for NF4 path)
• ~67 GB disk for safetensors, ~21 GB for GGUF
• ~75 GB VRAM (bf16) or ~28 GB (NF4) for inference

Limitations

• Domain bias: the final merger checkpoint was polished on wave-ui-25k, a UI-screenshot dataset. Strongest grounding is on screenshots / UI elements / OCR-heavy images. Natural-image captioning works but is not as sharp as the instruction-tuned-only stage.
• Abliteration removes safety refusals from the LLM. Use responsibly.
• Merger is BF16, LLM is recommended NF4 for sane VRAM. Loading both in BF16 demands ~75 GB.
• Val_loss has plateaued (1.0318 → 1.0280 across the last 7K steps of Stage 4). Further training would need a new dataset or different objective.

Files in This Repo

File	Size	Purpose
model-*-of-00015.safetensors	~66 GB total	BF16 full weights (LLM + vision + trained merger)
ggml-model-Q4_K_M.gguf	21.2 GB	Quantized LLM for llama.cpp
mmproj-stage4.gguf	1.1 GB	Stage-4 trained mmproj for llama.cpp (matches the safetensors merger)
mmproj-model-f16.gguf	1.1 GB	Legacy alias (identical content)
chat_template.jinja	7 KB	Chat template
tokenizer.json + tokenizer_config.json	20 MB	Tokenizer

Credits

• openbmb — MiniCPM-V-4.6 vision architecture & weights
• huihui-ai — Abliterated Qwen3.5-35B-A3B language model
• liuhaotian — LLaVA-Pretrain & LLaVA-Instruct-150k datasets
• ScreenSpot — UI grounding dataset
• agentsea — wave-ui-25k SFT dataset
• Assembly & 4-stage merger training by jduartedj

License

Apache 2.0 (consistent with all upstream components).