Qwen3.5-4B-M4-v2-ex-LRP-turbo

Re-merge of M4-ex-LRP using the updated mergekit PR #682 (Tusm11's feature/ex-lrp branch, HEAD 8d989f6 — "supercharged ex-LRP with turbo optimizations and multimodal support").

Same AttnLRP signal as M4-orig, same sources, but:

Updated PR #682 code (Iron-Man stabilization, in-place math, multimodal support).
Different merge hyperparameters: equal weights (1.0 / 1.0) and density 0.7 — closer to PR #682's worked example than the M4-orig 0.55/0.45/0.53 search.

This isolates "what changed when the merger code was updated AND hyperparams were rebalanced" — not a clean ablation of the code change alone, but the closest fair retest given the original LRP signal is reused.

Sources

Role	Model
Base	`Qwen/Qwen3.5-4B`
Source A	`Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2`
Source B	`Crownelius/Crow-4B-Opus-4.6-Distill-Heretic_Qwen3.5`

Merge config:

merge_method: lrp
base_model:
  model: Qwen/Qwen3.5-4B
parameters:
  density: 0.7
models:
  - model: Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2
    parameters:
      weight: 1.0
      lrp_scores: jackrong-v2_lrp_scores.safetensors
  - model: Crownelius/Crow-4B-Opus-4.6-Distill-Heretic_Qwen3.5
    parameters:
      weight: 1.0
      lrp_scores: crow-4b_lrp_scores.safetensors

LRP scores reused from the original M4 run — multimodal-prefixed form (model.language_model.X), 427 keys, 19.4 GB each. Available under lrp/ in M4-ex-LRP.

Phase 1+2 comparison (Q6_K)

#	Recipe	Merger	Importance	HumanEval pass@1	MBPP pass@1
floor	Qwen/Qwen3.5-4B (no merge)	—	—	60.37% 🥇	45.80%
src A	Jackrong-v2 (Claude-4.6-Opus reasoning distill)	—	—	60.37%	45.00%
src B	Crow-4B (Opus-4.6 distill, heretic)	—	—	54.88%	48.20%
M1	Vanilla DARE-TIES	dare_ties_merge.py	none	51.22%	47.00%
M2	OMv2 recipe (OBIM-lite + DAREx-q + EMR election)	dare_ties_merge.py	none	52.44%	49.40%
M3	OMv2 + Fisher	dare_ties_merge.py	Fisher	57.93%	48.80%
M4	ex-LRP (mergekit PR #682 — original)	mergekit PR #682	LRP	51.22%	49.40%
M4-v2 (this)	ex-LRP (mergekit PR #682 — turbo, w=1/1, d=0.7)	mergekit PR #682 turbo	LRP	55.49%	52.20% 🥇
M5	OMv2 + LRP	dare_ties_merge.py	LRP	53.05%	51.40%

Eval methodology: llama-server (--reasoning-format deepseek --reasoning-budget 8192 --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0 -c 32768) → lm_eval local-completions against raw /v1/completions, temperature 0, max_gen_toks=2048. All variants scored under identical conditions on the same hardware (RTX 3090).

Delta vs M4-orig: HE +4.27 pp, MBPP +2.80 pp. The "turbo" branch + rebalanced hyperparams beat the original PR #682 run by a clear margin on both benchmarks. M4-v2 takes the MBPP lead of the whole study (52.20%) while staying competitive on HumanEval.

Patches required for multimodal Qwen3.5

Running PR #682 (turbo head) against Qwen3_5ForConditionalGeneration required the same set of patches as the M4-orig run, with a couple additions surfaced by the new code path:

mergekit/architecture/base.py — import torch for pydantic forward-ref model_rebuild() (PretrainedConfig schema references torch.dtype).
mergekit/architecture/auto.py — layer-aware optional for hybrid linear-attn / full-attn templates.
mergekit/merge_methods/lrp.py — base-passthrough fallback for tensors without LRP scores (vision tower, MTP heads).
mergekit/sparsify.py — added build_mask(importance, density) helper used by the turbo LRPMergeTask.
mergekit/common.py — added .get() and __contains__ on ImmutableMap (turbo make_task calls params.get("weight", 1.0)).
mergekit/config.py — str allowed in ParameterSetting union (so lrp_scores: "<path>" validates).

Floor reference: the top three rows are the base model and the two source fine-tunes evaluated under the same methodology. Hard finding: no merge beats Qwen3.5-4B base on HumanEval (best merge M3 = 57.93%, base = 60.37%). On MBPP, M4-v2 (52.20%) and M5 (51.40%) clearly beat both sources (best source: Crow at 48.20%) — merging does add MBPP capability, but borrows HumanEval ground.