---
license: apache-2.0
base_model: Qwen/Qwen3.5-0.8B
pipeline_tag: feature-extraction
library_name: transformers
language:
  - multilingual
tags:
  - multilingual
  - qwen3.5
  - classification-backbone
  - text-classification
  - knowledge-distillation
  - model-compression
  - edge-ai
---

# Qwen3.5 Standalone 4L Classification Base

This repository publishes a **headless, multilingual-input, classification-ready
four-layer Qwen3.5 text backbone**. The root model has no task labels and no
classification head. It is a compact starting point for user-trained
single-label or multilabel classifiers, not a chat model or a ready-made
universal classifier.

The repository follows one integrated 1-1-1 layout:

1. **Repository root:** task-agnostic four-layer base trained on unlabeled
   general text.
2. **`models/semeval-propaganda/`:** ready-to-use SemEval-derived propaganda
   classifier preserved as a separate task-specific lineage.
3. **`distillation/`:** reusable 24L→8L→6L→4L platform for user-owned data.

## Model at a glance

| Item | Root model |
|---|---|
| Upstream | [`Qwen/Qwen3.5-0.8B`](https://huggingface.co/Qwen/Qwen3.5-0.8B) |
| Runtime class | `Qwen3_5TextModel` in Transformers 5.13.0 |
| Text layers | 4: `linear_attention → full_attention → linear_attention → full_attention` |
| Hidden / FFN | 1,024 / 3,584 |
| Vocabulary | trimmed Qwen3.5 tokenizer, 128,000 entries |
| Root parameters | 210,889,024 |
| Root BF16 weights | 421,783,648 bytes (402.29 MiB) |
| Output | hidden states; no LM head, task head, labels, or thresholds |
| Task-free KD data | 4,096 unlabeled WikiText-103 raw rows |
| Downstream evidence | one English 14-label SemEval-derived transfer probe |

## Load the headless base

```python
from transformers import AutoModel, AutoTokenizer

model_id = "mp-juuuns/qwen35-standalone4l-classification-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
backbone = AutoModel.from_pretrained(model_id, trust_remote_code=False)
```

The published root weight file contains no `score.weight`. A fresh task head and
labeled fine-tuning are required before outputs have class meaning.

## Make your own classifier

```bash
git clone https://huggingface.co/mp-juuuns/qwen35-standalone4l-classification-base
cd qwen35-standalone4l-classification-base/distillation
python -m pip install -e .

qwen35-distill finetune \
  --checkpoint .. \
  --train examples/multilabel_train.jsonl \
  --labels examples/labels.json \
  --mode multilabel \
  --output my-classifier-4l
```

For a full task-specific compression run, fine-tune a 24-layer teacher, then
repeat `materialize-classifier` and `distill-classifier` for `24to8`, `8to6`,
and `6to4`. Exact JSONL formats, split rules, manifests, and commands are in
[`distillation/README.md`](distillation/README.md).

## How the root weights were made

The text backbone was retained from the upstream-derived 4L model. The 128k
tokenizer was selected from the verified English-focused classifier lineage,
and embedding rows were remapped by token string rather than by row position.
This release is a vocabulary-remapped artifact, not a new training run.
Students were trained sequentially through 24→8→6→4 using
hidden-boundary and final-representation matching on unlabeled text.

The task-free training set was exactly 4,096 deterministically shuffled,
non-empty rows from the first locally cached WikiText-103 raw training shard.
No SemEval examples, labels, logits, thresholds, classification heads, or
evidence/span objectives were read while training the root weights.

| KD stage | Selected source layers | Mean total loss | Mean interface loss | Mean final loss |
|---|---|---:|---:|---:|
| 24→8 | `0,4,6,11,13,16,20,23` | 0.176227 | 0.215628 | 0.166264 |
| 8→6 | `0,1,3,4,6,7` | 0.033821 | 0.044861 | 0.031147 |
| 6→4 | `0,2,3,5` | 0.047020 | 0.070484 | 0.043120 |

These are training-objective values, not downstream quality scores. Exact
parameter counts, BF16 file sizes, hashes, and stage values are in
[`benchmark/compression_ladder.csv`](benchmark/compression_ladder.csv) and
[`provenance/task_agnostic_stages/`](provenance/task_agnostic_stages/).

![Task-free 24L to 4L compression ladder](benchmark/figures/04_compression_ladder.png)

The chart reports the actual unsharded safetensors parameter counts and BF16
file sizes. It does not imply that downstream quality scales monotonically with
layer count.

## Why 128k vocabulary trimming?

The original root used 248,320 input-token rows. This update retains 128,000
token IDs and remaps each retained token to the corresponding row of the
original embedding. The goal is to reduce edge storage and memory for the
English-focused classification deployment path; it is not a claim that the
model remains equally suitable for every language represented by the upstream
tokenizer.

| Quantity | 248k root | 128k root | Change |
|---|---:|---:|---:|
| Vocabulary entries | 248,320 | 128,000 | −48.4% |
| Parameters | 334,096,704 | 210,889,024 | −36.8% |
| BF16 weight bytes | 668,198,976 | 421,783,648 | −36.9% |

![Vocabulary size comparison](benchmark/figures/v128k_vocabulary_size.png)

The remapping manifest is `v128k_remap_manifest.json`. It records the source
hashes, token-string mapping hash, output hash, and parameter count.

## Quality and efficiency evidence

The new root is headless, so its downstream quality is **NR** until a new task
head is attached and evaluated. Existing evidence is shown for context:

- 248k task-free root transfer probe: Macro-F1 mean `0.587476` across three
  seeds on the previously opened SemEval-derived test.
- 128k specialized classifier lineage: Macro-F1 mean `0.587568` across three
  seeds on the same opened task.
- These two values are **not a controlled vocabulary ablation**: the models
  differ in task-specific head/training lineage. They must not be interpreted
  as proof that vocabulary trimming preserves quality.

![Quality and storage evidence](benchmark/figures/v128k_quality_storage.png)

The direct efficiency result of this release is the storage/parameter
reduction above. Existing CUDA resource measurements also show the 128k
specialized lineage at lower peak allocation than the 248k task-free lineage,
but that comparison is confounded by the separate classifier lineage and is
therefore descriptive only. Raw inputs and limitations are in
`benchmark/figures/v128k_quality_efficiency.csv` and
`benchmark/figures/manifest.json`.

## Downstream benchmark contract

Because the root is headless, it cannot be scored as a classifier without
adaptation. Its task-appropriate benchmark is therefore a **controlled transfer
probe**:

- attach a fresh randomly initialized 14-label classification head;
- train all parameters for five epochs on the same SemEval-derived train split;
- select one global threshold on the epoch-5 calibration split;
- evaluate once per seed on the already-opened 55-article test split;
- compare against the same four-layer structural copy without task-free KD.

This derived task measures article-level presence of 14 propaganda techniques.
It is not the official SemEval span- or fragment-level task. Results are
exploratory because the public test split had already been opened.

### Frozen protocol

| Setting | Value |
|---|---|
| Seeds | 41, 42, 43 |
| Train / calibration / test articles | 260 / 56 / 55 |
| Test windows | 434 |
| Epochs | 5, no early stopping |
| Input windows | max length 256, overlap stride 128 |
| Article aggregation | label-wise maximum probability |
| Batch | micro 1, gradient accumulation 32, effective 32 |
| Optimizer | AdamW, LR 2e-5, weight decay 0.01, gradient clip 1.0 |
| Precision / device | BF16 / NVIDIA GeForce RTX 5070 Ti |
| Threshold selection | epoch-5 calibration Macro-F1, then Micro-F1 |

### Overall results by seed

| Initialization | Seed | Threshold | Macro-F1 | Micro-F1 | Exact match | Predicted positive rate |
|---|---:|---:|---:|---:|---:|---:|
| Task-agnostic KD base | 41 | 0.15 | 0.58831 | 0.64681 | 0.05455 | 0.57143 |
| Task-agnostic KD base | 42 | 0.30 | 0.60284 | 0.66552 | 0.03636 | 0.40909 |
| Task-agnostic KD base | 43 | 0.10 | 0.57128 | 0.64653 | 0.00000 | 0.51558 |
| Structural copy, no task-free KD | 41 | 0.05 | 0.51694 | 0.57740 | 0.00000 | 0.71299 |
| Structural copy, no task-free KD | 42 | 0.20 | 0.53494 | 0.60834 | 0.00000 | 0.62078 |
| Structural copy, no task-free KD | 43 | 0.20 | 0.50238 | 0.58564 | 0.00000 | 0.59610 |

### Three-seed summary

Values are mean ± sample SD across the three training seeds. The SD is not a
confidence interval.

| Initialization | Macro-F1 | Micro-F1 | Exact match | Predicted positive rate |
|---|---:|---:|---:|---:|
| **Task-agnostic KD base** | **0.58748 ± 0.01579** | **0.65295 ± 0.01088** | **0.03030 ± 0.02777** | 0.49870 ± 0.08248 |
| Structural copy, no task-free KD | 0.51809 ± 0.01631 | 0.59046 ± 0.01603 | 0.00000 ± 0.00000 | 0.64329 ± 0.06161 |
| Existing specialized 4L, separate lineage | 0.58757 ± 0.00717 | see nested reports | see nested reports | see nested reports |

The task-agnostic KD initialization improved same-seed mean Macro-F1 by
`+0.06939` over the structural-copy control. This is an observed comparison
under this protocol, not a significance or universal-superiority claim.

![Three-seed Macro-F1 and Micro-F1 comparison](benchmark/figures/01_overall_transfer_quality.png)

Bars are three-seed means and whiskers are sample SD. The specialized 4L point
is a separate historical lineage rather than a fine-tuned child of the root
base.

### Task-agnostic base: per-label test results

Precision, recall, and F1 are three-seed means. Support is the fixed number of
positive test articles, not a three-seed sum.

| Label | Support | Precision | Recall | F1 |
|---|---:|---:|---:|---:|
| `Appeal_to_Authority` | 11 | 0.3959 | 0.5758 | 0.4657 |
| `Appeal_to_fear-prejudice` | 24 | 0.5540 | 0.7361 | 0.6278 |
| `Bandwagon,Reductio_ad_hitlerum` | 7 | 0.4852 | 0.8095 | 0.6007 |
| `Black-and-White_Fallacy` | 12 | 0.3668 | 0.4722 | 0.4054 |
| `Causal_Oversimplification` | 18 | 0.5429 | 0.7963 | 0.6447 |
| `Doubt` | 22 | 0.5224 | 0.9848 | 0.6802 |
| `Exaggeration,Minimisation` | 22 | 0.4621 | 0.7727 | 0.5727 |
| `Flag-Waving` | 17 | 0.5643 | 0.8235 | 0.6465 |
| `Loaded_Language` | 45 | 0.8649 | 0.9852 | 0.9209 |
| `Name_Calling,Labeling` | 33 | 0.7381 | 0.9091 | 0.8125 |
| `Repetition` | 23 | 0.5197 | 0.7536 | 0.6148 |
| `Slogans` | 14 | 0.5161 | 0.7381 | 0.5835 |
| `Thought-terminating_Cliches` | 7 | 0.3492 | 0.3333 | 0.3190 |
| `Whataboutism,Straw_Men,Red_Herring` | 10 | 0.2762 | 0.5000 | 0.3304 |

Per-seed values and sample SDs for all labels and all arms are available in
[`benchmark/per_label_metrics.csv`](benchmark/per_label_metrics.csv) and
[`benchmark/per_label_summary.csv`](benchmark/per_label_summary.csv).

![Per-label precision recall and F1](benchmark/figures/02_per_label_transfer_metrics.png)

Labels are ordered by their three-seed mean F1. Support remains visible because
several classes have only 7–14 positive test articles.

### Efficiency and resource measurements

These numbers belong to the fresh-head transfer checkpoints, not a standalone
headless-root latency benchmark.

| Measure | Task-agnostic transfer model |
|---|---:|
| Parameters with 14-label head | 334,111,040 |
| Test inference | 0.08131 ± 0.00243 s/article |
| Test batch | 55 articles / 434 windows |
| Peak CUDA allocated | 3,380,975,616 bytes (3.15 GiB) |
| Total train-and-evaluate wall time | 1,591.69 s mean (26.53 min) |

The timing is specific to the recorded RTX 5070 Ti, BF16, batch-1,
sliding-window protocol and includes article aggregation. It is not a claim
about CPU, mobile, UNO Q, or other deployment performance. Full seed-level
resource fields are in
[`benchmark/resource_metrics.csv`](benchmark/resource_metrics.csv).

![Quality efficiency comparison](benchmark/figures/03_quality_efficiency.png)

These plots compare the recorded fresh-head transfer checkpoints only.
Apparent differences must not be generalized to the headless root on untested
hardware.

### Machine-readable benchmark bundle

- [`benchmark/BENCHMARK_CARD.md`](benchmark/BENCHMARK_CARD.md): generated
  benchmark report and interpretation
- [`benchmark/summary.json`](benchmark/summary.json): complete aggregate
- [`benchmark/seed_metrics.csv`](benchmark/seed_metrics.csv): overall metrics
  per arm and seed
- [`benchmark/arm_summary.csv`](benchmark/arm_summary.csv): three-seed means
  and sample SDs
- [`benchmark/per_label_metrics.csv`](benchmark/per_label_metrics.csv):
  per-label values per arm and seed
- [`benchmark/per_label_summary.csv`](benchmark/per_label_summary.csv):
  per-label means and sample SDs
- [`benchmark/resource_metrics.csv`](benchmark/resource_metrics.csv):
  parameters, CUDA memory, wall time, and test timing
- [`benchmark/reports/`](benchmark/reports/): normalized full frozen reports,
  including article-level probability vectors
- [`benchmark/figures/`](benchmark/figures/): Matplotlib PNG/PDF figures,
  per-figure source CSVs, and a hash manifest

## Multilingual scope

The root retains the upstream Qwen3.5 tokenizer and multilingual architecture.
The [official Qwen3.5-0.8B model card](https://huggingface.co/Qwen/Qwen3.5-0.8B)
states expanded support for **201 languages and dialects** and reports
multilingual upstream benchmarks such as MMMLU, MMLU-ProX, NOVA-63, INCLUDE,
Global PIQA, PolyMATH, WMT24++, and MAXIFE. This is why this repository carries
the Hugging Face `multilingual` language and model tag.

However, this four-layer root was distilled on English WikiText and its only
downstream transfer probe is English SemEval-derived data. **Tokenizer coverage
and inherited architecture support do not establish retained classification
quality in all 201 languages.** No multilingual downstream score is reported
for this root release. Users should fine-tune and evaluate on each intended
language and domain before making performance claims.

## Separate SemEval model

The ready-to-use propaganda classifier is under
[`models/semeval-propaganda/`](models/semeval-propaganda/). It preserves the
published seed-41/42/43 checkpoints and links to the original repository at
immutable revision
[`e7e0ff1`](https://huggingface.co/mp-juuuns/qwen35-standalone4l-propaganda-classifier/tree/e7e0ff16828052687d2e8dd7849e7a521629cf38).

That specialized model is **not** claimed to descend from this task-free base.
It has its own SemEval task-specific shrink/distillation and fine-tuning history
and a reduced 128k vocabulary. The updated root now uses the token-string-
remapped 128k tokenizer documented above. Its complete reports remain under
[`models/semeval-propaganda/benchmark/`](models/semeval-propaganda/benchmark/).

## Limitations

- A classification head and labeled training are required before root-model
  predictions have task meaning.
- Task-free KD used only 4,096 rows from one English WikiText shard.
- Only one downstream task and one hardware/software setting were measured.
- The 55-article test split was previously opened; results are exploratory.
- Per-label support ranges from 7 to 45 articles and rare-label estimates are
  unstable.
- Three seeds describe run-to-run variation; they are not a confidence
  interval.
- Multilingual input support has not been validated as multilingual downstream
  classification quality for this four-layer root.
- Default layer maps are documented structural choices, not universal optima.
- Long-context, calibration, robustness, fairness, and production safety have
  not been established.
- Do not use either model as a fact checker, safety oracle, or autonomous
  decision maker.

## License and attribution

The released code and model artifacts use Apache-2.0, subject to upstream model
and data terms. Qwen3.5 is attributed to Qwen. Transformers, PyTorch,
Hugging Face Hub, WikiText, and SemEval remain the work of their respective
authors. No WikiText or SemEval source records are redistributed.

The WikiText page currently has a license wording discrepancy: metadata lists
CC BY-SA 3.0 and GFDL, while prose says CC BY-SA 4.0. Users should inspect
[`Salesforce/wikitext`](https://huggingface.co/datasets/Salesforce/wikitext)
directly.