[Thesis]CEFR Steering: PPLM
Collection
PPLM baseline checkpoints for my thesis: "Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation". • 8 items • Updated
Llama-3.1-8B CEFR 1-Layer MLP Steering Head (Ordinal Universal)
This repository contains a 1-Layer Linear Classifier Head designed to steer the hidden representations of meta-llama/Llama-3.1-8B-Instruct towards specific CEFR proficiency levels (A1-C2).
🧠 Architectural Rationale
Unlike deeper multi-layer MLPs, this 1-layer head maps hidden representations ($d=4096$) directly to CEFR logits ($d=6$) using a single linear layer with an input dropout of 0.35. By eliminating ReLU activations and LayerNorm scaling, this architecture ensures:
- Strict Gradient Continuity: Eliminates vanishing/exploding gradient boundaries.
- Stable Backpropagation Landscape: The continuous linear gradient vector provides stable, non-saturating paths for real-time decoding-time token perturbation (PPLM).
📊 Training Specifications & Performance
- Loss Function: Custom Ordinal Weighted Loss (Weighted Cross-Entropy + MSE Ordinal Distance Penalty)
- Optimizer: AdamW (Learning Rate: 3e-4, Weight Decay: 0.01)
- Epochs: 50
- Batch Size: 128
- Peak Validation Strict Accuracy: 85.16%
- Peak Validation Adjacent Accuracy: 97.32%