# TASK: Empirical Pruning of Qwen/Qwen3.5-0.8B-Base to 330-450M Parameters ((Not approximately, but exactly. It must be one of these two, or have a parameter within these two ranges.))

## 0. Ground Truth — Do Not Assume Numbers, Compute Them
Qwen/Qwen3.5-0.8B-Base is a MULTIMODAL model (architecture: Qwen3_5ForConditionalGeneration),
with config split into `vision_config` and `text_config`. This task prunes the TEXT backbone only;
the vision tower is dropped entirely.

Real text_config fields (verified from HF config.json):
- hidden_size: 1024
- intermediate_size: 3584   (NOT 2816)
- num_hidden_layers: 24
- layer_types: ["linear_attention"]*3 + ["full_attention"] repeated 6x (24 entries)
- full_attention_interval: 4
- head_dim: 256 (full attention)
- linear_key_head_dim: 128, linear_conv_kernel_dim: 4
- linear_num_key_heads: 16, linear_num_value_heads: 32 (verify against live config — sources disagree,
  model card text says 16/16, raw config.json says 16 key / 32 value heads; trust the live config.json
  you load at runtime, not any cached number in this document)
- vocab / tied embedding: 248,320 tokens × 1024 = ~254.3M params — this is FIXED regardless of
  how many layers you keep, since it's tied to the LM head.

## 1. Feasibility Check (run BEFORE deciding any target layer count)
Load the real model and measure actual per-layer parameter counts. Do not estimate by hand or trust
any prior document's math — prior estimates for this exact model have been off by 100M+.

```python
from transformers import AutoModelForCausalLM, AutoConfig
import torch

config = AutoConfig.from_pretrained("Qwen/Qwen3.5-0.8B-Base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-0.8B-Base", trust_remote_code=True, torch_dtype=torch.bfloat16
)
# Isolate the text backbone only (drop vision tower)
text_model = model.model.language_model if hasattr(model.model, "language_model") else model.model

embed_params = sum(p.numel() for p in text_model.embed_tokens.parameters())
layer_params = [sum(p.numel() for p in layer.parameters()) for layer in text_model.layers]

print(f"Embedding (tied) params: {embed_params/1e6:.1f}M")
for i, lp in enumerate(layer_params):
    print(f"Layer {i} ({config.text_config.layer_types[i]}): {lp/1e6:.2f}M")

# Compute how many layers fit a given total budget
target_min, target_max = 330_000_000, 350_000_000
budget = target_max - embed_params
cum = 0
for i, lp in enumerate(layer_params):
    cum += lp
    if cum > budget:
        print(f"With hidden_size unchanged, only ~{i} layers fit under {target_max/1e6:.0f}M total.")
        break

```

Run this first. If the number of layers that fit is too small to preserve at least one complete
hybrid block (3 linear_attention + 1 full_attention = 4 layers minimum for the architecture to
still contain a full-attention layer at all), the 330-350M target is not reachable via depth
pruning alone with hidden_size=1024 fixed. In that case go to Step 2b (width pruning) instead of
Step 2a (depth-only).

## 2a. Depth-Only Pruning (if Step 1 shows ≥4-8 layers fit the budget)

Keep the first N layers, where N is a multiple of 4 (to preserve complete hybrid blocks) closest
to (but not exceeding) the layer count computed in Step 1. Do not use activation-norm-based layer
selection — codefuse-ai's F2LLM-v2 paper (arXiv:2603.19223) reports this underperforms simply
keeping the first N layers for their pruning pipeline; apply the same simple-truncation strategy
here.

```python
config.text_config.num_hidden_layers = N
config.text_config.layer_types = config.text_config.layer_types[:N]

```

Transfer weights for layers[:N] as-is. Transfer embed_tokens, final norm, and tied lm_head intact
(do not slice — hidden_size is unchanged in this path).

## 2b. Width + Depth Pruning (if Step 1 shows depth pruning alone can't reach 330-350M)

Follow F2LLM-v2's actual 3-dimension approach (arXiv:2603.19223, Section 3.3):

1. Prune num_hidden_layers by keeping first N layers (same as 2a).
2. Additionally prune hidden_size and intermediate_size using activation-norm-based row/column
selection on a calibration set — rank neurons/dimensions by mean L2 activation norm, keep the
top-scoring indices, slice all weight matrices consistently (q/k/v/gate/up/down projections,
plus norms) to match the new hidden_size.
3. Note: reducing hidden_size on this architecture also changes the GatedDeltaNet recurrent state
width (key/value head dims are defined relative to hidden_size in the live config — re-derive
linear_key_head_dim / linear_num_key_heads / linear_num_value_heads consistently after pruning;
do not assume they scale linearly without checking the actual attention module's forward pass).
4. Re-run the Step 1 measurement script against the pruned checkpoint to confirm the actual total
before proceeding to distillation — do not trust the arithmetic prediction.

## 3. Knowledge Distillation

Teacher: frozen Qwen/Qwen3.5-0.8B-Base text backbone (bfloat16, eval mode).
Student: the pruned checkpoint from Step 2a or 2b.
Loss: standard next-token cross-entropy + MSE between teacher and student final hidden states
(sequence representation), matching F2LLM-v2's knowledge distillation ablation (Table 4), which
shows a consistent, measurable performance drop when distillation is omitted — do not skip it.
Optimizer: AdamW, cosine schedule with warmup — tune lr/steps empirically; no external source
gives a validated hyperparameter set for this specific pruned architecture, so treat any lr/steps
figures as a starting point requiring empirical validation, not a fixed target.

## 4. Validation

Assert final total parameter count is inside [330_000_000, 350_000_000] using the Step 1 script
against the final checkpoint (not a predicted number). If it falls outside the range, adjust N
(depth) or the width-pruning target dimension and re-measure — do not adjust the target range to
match whatever number you got.

## 5. Commit

Break each step into phases and make atomic commits.
Making atomic commits makes it easier to track history and makes progress tracking much better.
Subsequently, you must write the progress to PROGRESS.md and track it from there; checkmarks are added to the boxes as each task is completed.