---
base_model: siddharthmb/2026.AP.cpt_dense
datasets:
- allenai/tulu-3-sft-mixture
---

# 2026.AP.sft_dense

Dense-baseline SFT run: initialized from the cpt_dense model (init_from models/cpt_dense) and fine-tuned on tulu-3-sft-mixture, as the dense control arm.

Part of a replication of Apple's **Instruction-Following Pruning for Large Language Models** ([arXiv:2501.02086](https://arxiv.org/abs/2501.02086)) — "AP" = apple-paper-replicate, package `ifpruning` in [Sid-MB/mats_exploration](https://github.com/Sid-MB/mats_exploration) under `code/apple-paper-replicate/` (branch `introspection-causal-test`, merged to main at `a76965b`).

## Architecture

- LLM: Qwen2.5-3B (Qwen2ForCausalLM, 36 layers, d_ffn 11008)
- Predictor backbone: Qwen2.5-0.5B (Qwen2Model) + 2-layer MLP head (`head.pt`) producing per-layer FFN importance scores `[36, 11008]`; per-row SoftTopK selects t_ffn=1536 of 11008 FFN units (~1B activated params). Dense-baseline runs train the same LLM without masking.

## Contents

- `checkpoints/step_1000/pytorch_model_fsdp_0/` — FSDP2 SHARDED_STATE_DICT model weights (llm + predictor_backbone + head)
- `checkpoints/step_1000/optimizer_0/` — optimizer state (for exact training resumption)
- `checkpoints/step_1000/random_states_*.pkl`, `scheduler.bin` — RNG/scheduler state

## Performance

**No evaluations were run on this checkpoint.** The project reached "scaffold + smoke test + this training grid" before being paused (see `code/mats_exploration/everything we learned.md`); `logs/eval/` is empty and no eval_results directory exists. The only training-quality signal is the loss curves in the wandb runs below.

## Reproduction

From `code/` in the mats_exploration repo (paths as of June 2026; `IFP_ROOT=/nlp/scr/siddharth/apple-paper-replicate` set in `slurm/_common.sh`):

```bash
sbatch apple-paper-replicate/slurm/train.sbatch apple-paper-replicate/configs/presets/sft_dense.yaml
```

which runs (8 GPUs, accelerate FSDP2 full-shard bf16, `SHARDED_STATE_DICT`):

```bash
srun uv run accelerate launch --config_file apple-paper-replicate/configs/accelerate_fsdp8.yaml \
    -m ifpruning.train --config apple-paper-replicate/configs/presets/sft_dense.yaml \
    --ckpt-root $IFP_ROOT/ckpts --out-root $IFP_ROOT/models
```

Preset: `configs/presets/sft_dense.yaml`. Data: allenai/tulu-3-sft-mixture. Seed 0. Slurm job 15878289 (jagupard39, 8 GPUs, afterok:15878286).

## Weights & Biases

- https://wandb.ai/siddharth-stanford/ifpruning-sft/runs/v3objrks

## Cluster paths (Stanford NLP)

- Training log: `/nlp/scr2/siddharth/code/mats_exploration/code/apple-paper-replicate/logs/train/sft_dense_15878289.out`
- Original checkpoint dir: `/nlp/scr2/siddharth/apple-paper-replicate/ckpts/sft_dense` (deleted after this upload was verified; this repo is now the only copy)
- Code, configs, paper PDF, research notes: `/nlp/scr2/siddharth/code/mats_exploration/code/apple-paper-replicate/`
- Research note with the full run grid: `.../research-notes/2026-06-12_setup-and-smoke.md`