Feature Extraction
Transformers
Safetensors
GGUF
English
Korean
qwen3_5_text
qwen3.5
backbone
headless
classification-backbone
knowledge-distillation
model-compression
vocabulary-pruning
korean
edge-ai
conversational
Instructions to use mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless") model = AutoModel.from_pretrained("mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0 # Run inference directly in the terminal: llama cli -hf mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0 # Run inference directly in the terminal: llama cli -hf mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0 # Run inference directly in the terminal: ./llama-cli -hf mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0 # Run inference directly in the terminal: ./build/bin/llama-cli -hf mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
Use Docker
docker model run hf.co/mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
- LM Studio
- Jan
- Ollama
How to use mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless with Ollama:
ollama run hf.co/mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
- Unsloth Desktop
- Pi
How to use mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless with Docker Model Runner:
docker model run hf.co/mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
- Lemonade
How to use mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
Run and chat with the model
lemonade run user.qwen3.5-4l-vocab40k-en-ko-headless-Q8_0
List all available models
lemonade list
- Hermes Agent
How to use mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "mp-juuuns/qwen3.5-4l-vocab40k-en-ko-headless:Q8_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Add bilingual 4L and MALINT benchmark evidence
Browse files- .gitattributes +3 -0
- README.md +168 -8
- benchmark/figures/README.md +22 -1
- benchmark/figures/cross_task/05_why_4l_edge_tradeoff.csv +34 -0
- benchmark/figures/cross_task/05_why_4l_edge_tradeoff.pdf +0 -0
- benchmark/figures/cross_task/05_why_4l_edge_tradeoff.png +3 -0
- benchmark/figures/cross_task/06_malint_4l_quality.csv +21 -0
- benchmark/figures/cross_task/06_malint_4l_quality.pdf +0 -0
- benchmark/figures/cross_task/06_malint_4l_quality.png +3 -0
- benchmark/figures/cross_task/07_malint_quality_resource.csv +15 -0
- benchmark/figures/cross_task/07_malint_quality_resource.pdf +0 -0
- benchmark/figures/cross_task/07_malint_quality_resource.png +3 -0
- benchmark/figures/cross_task/README.md +16 -0
- benchmark/figures/cross_task/manifest.json +158 -0
- benchmark/figures/cross_task/source_summary.json +972 -0
.gitattributes
CHANGED
|
@@ -1,3 +1,6 @@
|
|
| 1 |
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 2 |
benchmark/figures/02_per_label_transfer_metrics.png filter=lfs diff=lfs merge=lfs -text
|
| 3 |
benchmark/figures/04_compression_ladder.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 2 |
benchmark/figures/02_per_label_transfer_metrics.png filter=lfs diff=lfs merge=lfs -text
|
| 3 |
benchmark/figures/04_compression_ladder.png filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
benchmark/figures/cross_task/05_why_4l_edge_tradeoff.png filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
benchmark/figures/cross_task/06_malint_4l_quality.png filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
benchmark/figures/cross_task/07_malint_quality_resource.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -10,9 +10,11 @@ tags:
|
|
| 10 |
- qwen3.5
|
| 11 |
- classification-backbone
|
| 12 |
- text-classification
|
|
|
|
| 13 |
- knowledge-distillation
|
| 14 |
- model-compression
|
| 15 |
- edge-ai
|
|
|
|
| 16 |
---
|
| 17 |
|
| 18 |
# Qwen3.5 Standalone 4L Classification Base
|
|
@@ -31,6 +33,26 @@ The repository follows one integrated 1-1-1 layout:
|
|
| 31 |
classifier preserved as a separate task-specific lineage.
|
| 32 |
3. **`distillation/`:** reusable 24L→8L→6L→4L platform for user-owned data.
|
| 33 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
## Model at a glance
|
| 35 |
|
| 36 |
| Item | Root model |
|
|
@@ -44,7 +66,7 @@ The repository follows one integrated 1-1-1 layout:
|
|
| 44 |
| Root BF16 weights | 668,198,976 bytes (637.24 MiB) |
|
| 45 |
| Output | hidden states; no LM head, task head, labels, or thresholds |
|
| 46 |
| Task-free KD data | 4,096 unlabeled WikiText-103 raw rows |
|
| 47 |
-
| Downstream evidence |
|
| 48 |
|
| 49 |
## Load the headless base
|
| 50 |
|
|
@@ -107,6 +129,126 @@ The chart reports the actual unsharded safetensors parameter counts and BF16
|
|
| 107 |
file sizes. It does not imply that downstream quality scales monotonically with
|
| 108 |
layer count.
|
| 109 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 110 |
## Downstream benchmark contract
|
| 111 |
|
| 112 |
Because the root is headless, it cannot be scored as a classifier without
|
|
@@ -245,6 +387,10 @@ hardware.
|
|
| 245 |
including article-level probability vectors
|
| 246 |
- [`benchmark/figures/`](benchmark/figures/): Matplotlib PNG/PDF figures,
|
| 247 |
per-figure source CSVs, and a hash manifest
|
|
|
|
|
|
|
|
|
|
|
|
|
| 248 |
|
| 249 |
## Multilingual scope
|
| 250 |
|
|
@@ -255,12 +401,13 @@ multilingual upstream benchmarks such as MMMLU, MMLU-ProX, NOVA-63, INCLUDE,
|
|
| 255 |
Global PIQA, PolyMATH, WMT24++, and MAXIFE. This is why this repository carries
|
| 256 |
the Hugging Face `multilingual` language and model tag.
|
| 257 |
|
| 258 |
-
However, this four-layer root was distilled on English WikiText and its
|
| 259 |
-
downstream
|
| 260 |
-
|
| 261 |
-
|
| 262 |
-
|
| 263 |
-
|
|
|
|
| 264 |
|
| 265 |
## Separate SemEval model
|
| 266 |
|
|
@@ -281,12 +428,19 @@ tokenizer. Its complete reports remain under
|
|
| 281 |
- A classification head and labeled training are required before root-model
|
| 282 |
predictions have task meaning.
|
| 283 |
- Task-free KD used only 4,096 rows from one English WikiText shard.
|
| 284 |
-
- Only
|
|
|
|
|
|
|
|
|
|
| 285 |
- The 55-article test split was previously opened; results are exploratory.
|
| 286 |
- Per-label support ranges from 7 to 45 articles and rare-label estimates are
|
| 287 |
unstable.
|
| 288 |
- Three seeds describe run-to-run variation; they are not a confidence
|
| 289 |
interval.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 290 |
- Multilingual input support has not been validated as multilingual downstream
|
| 291 |
classification quality for this four-layer root.
|
| 292 |
- Default layer maps are documented structural choices, not universal optima.
|
|
@@ -302,6 +456,12 @@ and data terms. Qwen3.5 is attributed to Qwen. Transformers, PyTorch,
|
|
| 302 |
Hugging Face Hub, WikiText, and SemEval remain the work of their respective
|
| 303 |
authors. No WikiText or SemEval source records are redistributed.
|
| 304 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 305 |
The WikiText page currently has a license wording discrepancy: metadata lists
|
| 306 |
CC BY-SA 3.0 and GFDL, while prose says CC BY-SA 4.0. Users should inspect
|
| 307 |
[`Salesforce/wikitext`](https://huggingface.co/datasets/Salesforce/wikitext)
|
|
|
|
| 10 |
- qwen3.5
|
| 11 |
- classification-backbone
|
| 12 |
- text-classification
|
| 13 |
+
- multilabel-classification
|
| 14 |
- knowledge-distillation
|
| 15 |
- model-compression
|
| 16 |
- edge-ai
|
| 17 |
+
- malint
|
| 18 |
---
|
| 19 |
|
| 20 |
# Qwen3.5 Standalone 4L Classification Base
|
|
|
|
| 33 |
classifier preserved as a separate task-specific lineage.
|
| 34 |
3. **`distillation/`:** reusable 24L→8L→6L→4L platform for user-owned data.
|
| 35 |
|
| 36 |
+
## 한국어 요약
|
| 37 |
+
|
| 38 |
+
이 저장소의 루트 모델은 Qwen3.5-0.8B의 text backbone을 **4개 layer로 축소한
|
| 39 |
+
분류 전용 기반 모델**이다. LM head와 task label, 분류 head가 없으므로 채팅 모델이나
|
| 40 |
+
즉시 사용할 수 있는 범용 분류기가 아니다. 사용자가 자신의 label과 데이터로 새 head를
|
| 41 |
+
붙여 학습하는 출발점이다.
|
| 42 |
+
|
| 43 |
+
4L을 선택한 근거는 한 숫자가 아니라 세 범주의 측정이다. 첫째, full-vocabulary
|
| 44 |
+
task-free backbone은 24L의 752.39M parameter/1,435.11 MiB에서 4L의
|
| 45 |
+
334.10M/637.24 MiB로 줄었다. 둘째, 동일 V128k/F16 UNO Q 깊이 비교에서 역사적
|
| 46 |
+
4L proxy는 8L 대비 artifact `-27.74%`, PP128 latency `-49.84%`, peak RSS
|
| 47 |
+
`-21.81%`, core-seconds `-48.96%`를 기록했다. 셋째, 별도 task 적응 결과에서
|
| 48 |
+
4L endpoint가 propaganda와 MALINT 모두에서 실제 분류 가능성을 보였다.
|
| 49 |
+
|
| 50 |
+
다만 이 근거는 “4L이 항상 8L/24L보다 정확하다”는 뜻이 아니다. UNO Q의 4L은
|
| 51 |
+
historical early-path proxy이고, propaganda 24L/8L/4L endpoint는 계보와 vocabulary가
|
| 52 |
+
달라 순수 depth ablation이 아니다. MALINT의 Qwen student 세 모델은 모두 4L이므로,
|
| 53 |
+
MALINT는 **4L 내부 pooling/KD interface 개선**을 검증할 뿐 4L-vs-8L/24L 우월성을
|
| 54 |
+
검증하지 않는다. 아래 표와 그림은 이 경계를 수치와 함께 명시한다.
|
| 55 |
+
|
| 56 |
## Model at a glance
|
| 57 |
|
| 58 |
| Item | Root model |
|
|
|
|
| 66 |
| Root BF16 weights | 668,198,976 bytes (637.24 MiB) |
|
| 67 |
| Output | hidden states; no LM head, task head, labels, or thresholds |
|
| 68 |
| Task-free KD data | 4,096 unlabeled WikiText-103 raw rows |
|
| 69 |
+
| Downstream evidence | English 14-label propaganda transfer and 5-label MALINT adaptation studies under separate protocols |
|
| 70 |
|
| 71 |
## Load the headless base
|
| 72 |
|
|
|
|
| 129 |
file sizes. It does not imply that downstream quality scales monotonically with
|
| 130 |
layer count.
|
| 131 |
|
| 132 |
+
## Why four layers? / 왜 4L인가?
|
| 133 |
+
|
| 134 |
+
The choice is an **edge-oriented operating point**, not a universal optimum.
|
| 135 |
+
The task-free full-vocabulary root has 334,096,704 parameters and a 637.24 MiB
|
| 136 |
+
BF16 weight file, versus 752,393,024 parameters and 1,435.11 MiB for the 24L
|
| 137 |
+
text teacher: a `55.59%` reduction in both parameter count and raw BF16 bytes.
|
| 138 |
+
|
| 139 |
+
A separate controlled Arduino UNO Q chain held vocabulary (`128k`), precision
|
| 140 |
+
(`F16`), runtime, and benchmark protocol fixed while moving 8L→6L→4L. Values
|
| 141 |
+
below normalize the 8L arm to 100; lower is better.
|
| 142 |
+
|
| 143 |
+
| Controlled UNO Q arm | Artifact | PP128 latency | Peak RSS | Core-seconds |
|
| 144 |
+
|---|---:|---:|---:|---:|
|
| 145 |
+
| 8L | 100.00 | 100.00 | 100.00 | 100.00 |
|
| 146 |
+
| 6L | 85.59 | 76.37 | 87.51 | 76.17 |
|
| 147 |
+
| Historical 4L proxy | **72.26** | **50.16** | **78.19** | **51.04** |
|
| 148 |
+
|
| 149 |
+
Thus the matched 8L→4L route reduced artifact size by `27.74%`, PP128 latency
|
| 150 |
+
by `49.84%`, peak RSS by `21.81%`, and core-seconds by `48.96%`. CPU occupancy
|
| 151 |
+
remained near 3.6–3.7 saturated cores; the benefit was finishing sooner, not
|
| 152 |
+
lower active occupancy. The 4L point is a historical early-path proxy, not the
|
| 153 |
+
final common-head quality model.
|
| 154 |
+
|
| 155 |
+
The earlier English 14-label propaganda benchmark supplies task evidence under
|
| 156 |
+
one common five-epoch protocol. Its endpoints are operationally relevant but
|
| 157 |
+
not a pure depth study because lineage and vocabulary differ.
|
| 158 |
+
|
| 159 |
+
| Propaganda endpoint | Macro-F1 | Micro-F1 | Subset exact match | Test s/article | Peak CUDA allocated | Parameters |
|
| 160 |
+
|---|---:|---:|---:|---:|---:|---:|
|
| 161 |
+
| Qwen3.5 24L | 0.56426 ± 0.00626 | 0.62648 ± 0.01399 | 0.03030 ± 0.02777 | 0.58636 ± 0.00598 | 7,635,339,264 B | 752.41M |
|
| 162 |
+
| Qwen3.5 C8Q 8L | 0.53669 ± 0.01726 | 0.62388 ± 0.01362 | 0.00606 ± 0.01050 | 0.20666 ± 0.00207 | 4,257,894,400 B | 420.33M |
|
| 163 |
+
| Qwen3.5 final 4L | **0.58757 ± 0.00717** | **0.64798 ± 0.00187** | 0.00606 ± 0.01050 | **0.08544 ± 0.00060** | **2,143,655,936 B** | **210.90M** |
|
| 164 |
+
|
| 165 |
+
These results show that a compact four-layer endpoint can be useful; they do
|
| 166 |
+
not prove that reducing depth caused the quality difference. The headless root
|
| 167 |
+
also differs from the specialized 4L endpoint: the root keeps the full 248,320
|
| 168 |
+
token vocabulary and has 334.10M parameters.
|
| 169 |
+
|
| 170 |
+

|
| 171 |
+
|
| 172 |
+
한국어로 요약하면, 4L 선택은 동일 조건 edge chain에서 확인된 latency·memory·artifact
|
| 173 |
+
감소와 별도 task에서 확인된 분류 가능성을 함께 본 결과다. 그러나 propaganda endpoint의
|
| 174 |
+
계보/vocabulary 차이 때문에 4L의 품질 우위를 depth 하나의 인과 효과로 해석하면 안 된다.
|
| 175 |
+
|
| 176 |
+
## MALINT adaptation evidence / MALINT 적응 근거
|
| 177 |
+
|
| 178 |
+
MALINT is an English five-label malicious-intent multilabel task covering
|
| 179 |
+
changing political views (CPV), promoting social stereotypes/antagonisms
|
| 180 |
+
(PSSA), undermining international organizations/alliances (UIOA), promoting
|
| 181 |
+
anti-scientific views (PASV), and undermining public institutions (UCPI). No
|
| 182 |
+
MALINT row is redistributed here.
|
| 183 |
+
|
| 184 |
+
The strongest changed-model evidence is a frozen **3 repeats × 3 folds** matched
|
| 185 |
+
study over the combined 1,100-row development corpus. Each repeat reconstructs
|
| 186 |
+
exact out-of-fold predictions for all 1,100 rows at threshold `0.30`. There was
|
| 187 |
+
no fold checkpoint selection, threshold tuning, or official held-out access.
|
| 188 |
+
|
| 189 |
+
| Arm | Weighted-F1 | Macro-F1 | Micro-F1 | Subset exact match | Label-wise accuracy |
|
| 190 |
+
|---|---:|---:|---:|---:|---:|
|
| 191 |
+
| Rightmost token + hard labels (4L) | 0.52218 ± 0.01497 | 0.51096 ± 0.01628 | 0.53203 ± 0.01076 | 0.64030 ± 0.00367 | 0.87467 ± 0.00121 |
|
| 192 |
+
| Mean pooling + hard labels (4L) | 0.61180 ± 0.00886 | 0.60551 ± 0.00753 | 0.61769 ± 0.00746 | **0.66970 ± 0.00517** | **0.88764 ± 0.00175** |
|
| 193 |
+
| **Mean pooling + KD (4L)** | **0.63465 ± 0.01197** | **0.62818 ± 0.01423** | **0.63896 ± 0.00985** | 0.65576 ± 0.00319 | 0.87576 ± 0.00204 |
|
| 194 |
+
| Fresh RoBERTa teacher | 0.60468 ± 0.01580 | 0.59557 ± 0.01637 | 0.60462 ± 0.01777 | 0.62000 ± 0.00596 | 0.85079 ± 0.01087 |
|
| 195 |
+
|
| 196 |
+
Values are mean ± sample SD across three repeated OOF reconstructions. Subset
|
| 197 |
+
exact match requires all five labels for a row to match. Label-wise accuracy is
|
| 198 |
+
the mean over all row-label decisions and is imbalance-sensitive.
|
| 199 |
+
|
| 200 |
+
Mean+KD improved weighted F1 over rightmost-hard by `+0.11247`, with positive
|
| 201 |
+
deltas in `9/9` folds and corrected interval `[+0.07160, +0.15925]`. Mean
|
| 202 |
+
pooling alone improved it by `+0.08962` in `9/9` folds. The incremental KD
|
| 203 |
+
contrast was `+0.02285` in `8/9` folds, but its interval
|
| 204 |
+
`[-0.00155, +0.05059]` crosses zero. Therefore the strongest isolated evidence
|
| 205 |
+
is for masked mean pooling; independent KD and teacher superiority are not
|
| 206 |
+
established.
|
| 207 |
+
|
| 208 |
+

|
| 209 |
+
|
| 210 |
+
All three Qwen student arms above are four-layer models. This is evidence that
|
| 211 |
+
a 4L backbone can support a materially better document interface under this
|
| 212 |
+
internal protocol. It is **not** a MALINT 4L-vs-8L/24L depth ablation. It is
|
| 213 |
+
also post-selection development evidence, not untouched confirmation or SOTA.
|
| 214 |
+
The older validation-selected rightmost 4L has an official fixed-threshold test
|
| 215 |
+
weighted F1 of `0.53749` with row-bootstrap interval `[0.49102, 0.58403]`; the
|
| 216 |
+
changed Mean+KD artifact did not reopen that test, so the two protocols must not
|
| 217 |
+
be compared as if they were one held-out leaderboard.
|
| 218 |
+
|
| 219 |
+
### Quality–resource trade-off / 품질–자원 절충
|
| 220 |
+
|
| 221 |
+
A separate label-free same-input resource run used an RTX 5070 Ti, BF16,
|
| 222 |
+
sequence length 256, fresh processes, three counterbalanced rounds per batch,
|
| 223 |
+
and CUDA-resident token IDs/masks. Timing is model forward only.
|
| 224 |
+
|
| 225 |
+
| Batch | Rightmost ms/example | Mean+KD ms/example | Latency change | Rightmost peak allocated | Mean+KD peak allocated | Peak change |
|
| 226 |
+
|---:|---:|---:|---:|---:|---:|---:|
|
| 227 |
+
| 1 | 18.47463 | 19.31878 | **+4.569%** | 1,373,150,208 B | 1,374,199,296 B | **+0.076%** |
|
| 228 |
+
| 4 | 4.58975 | 4.67101 | **+1.770%** | 1,454,388,736 B | 1,455,437,824 B | **+0.072%** |
|
| 229 |
+
|
| 230 |
+
Mean+KD adds exactly five parameters. It is **not faster**; the supported claim
|
| 231 |
+
is higher internal F1 at a small measured forward-latency and peak-allocation
|
| 232 |
+
cost. Ambient `nvidia-smi` utilization is not reported as model occupancy.
|
| 233 |
+
|
| 234 |
+

|
| 235 |
+
|
| 236 |
+
한국어 해석: MALINT에서 확인된 핵심은 “4L보다 더 깊은 모델을 이겼다”가 아니라,
|
| 237 |
+
**깊이를 4L로 고정한 상태에서 문서 전체 mean pooling interface가 rightmost-token
|
| 238 |
+
interface보다 반복 내부평가 F1을 높였다**는 것이다. Mean+KD는 batch 1/4에서 각각
|
| 239 |
+
`4.569%`/`1.770%` 느렸고 peak allocation도 소폭 증가했다. 따라서 빠른 모델이
|
| 240 |
+
아니라, 거의 같은 크기·memory에서 제한된 latency 비용으로 내부 품질을 높인 적응
|
| 241 |
+
결과로 설명해야 한다.
|
| 242 |
+
|
| 243 |
+
### Cross-task boundary / task 간 비교 경계
|
| 244 |
+
|
| 245 |
+
Propaganda and MALINT have different labels, split histories, thresholds,
|
| 246 |
+
metrics, and evaluation protocols. Their F1 values are not pooled or ranked
|
| 247 |
+
against each other. Together they support only a bounded statement: this 4L
|
| 248 |
+
platform has been adapted successfully to two English multilabel settings,
|
| 249 |
+
while generalization, multilingual quality, pure depth superiority, and SOTA
|
| 250 |
+
remain unproven.
|
| 251 |
+
|
| 252 |
## Downstream benchmark contract
|
| 253 |
|
| 254 |
Because the root is headless, it cannot be scored as a classifier without
|
|
|
|
| 387 |
including article-level probability vectors
|
| 388 |
- [`benchmark/figures/`](benchmark/figures/): Matplotlib PNG/PDF figures,
|
| 389 |
per-figure source CSVs, and a hash manifest
|
| 390 |
+
- [`benchmark/figures/cross_task/manifest.json`](benchmark/figures/cross_task/manifest.json):
|
| 391 |
+
immutable input/output hashes for figures 05–07
|
| 392 |
+
- [`benchmark/figures/cross_task/source_summary.json`](benchmark/figures/cross_task/source_summary.json):
|
| 393 |
+
aggregate-only source values and claim boundaries for figures 05–07
|
| 394 |
|
| 395 |
## Multilingual scope
|
| 396 |
|
|
|
|
| 401 |
Global PIQA, PolyMATH, WMT24++, and MAXIFE. This is why this repository carries
|
| 402 |
the Hugging Face `multilingual` language and model tag.
|
| 403 |
|
| 404 |
+
However, this four-layer root was distilled on English WikiText and its current
|
| 405 |
+
downstream evidence is limited to two English multilabel settings:
|
| 406 |
+
SemEval-derived propaganda and MALINT. **Tokenizer coverage and inherited
|
| 407 |
+
architecture support do not establish retained classification quality in all
|
| 408 |
+
201 languages.** No multilingual downstream score is reported for this root
|
| 409 |
+
release. Users should fine-tune and evaluate on each intended language and
|
| 410 |
+
domain before making performance claims.
|
| 411 |
|
| 412 |
## Separate SemEval model
|
| 413 |
|
|
|
|
| 428 |
- A classification head and labeled training are required before root-model
|
| 429 |
predictions have task meaning.
|
| 430 |
- Task-free KD used only 4,096 rows from one English WikiText shard.
|
| 431 |
+
- Only two English multilabel downstream settings and a limited set of
|
| 432 |
+
hardware/software conditions were measured.
|
| 433 |
+
- Cross-task evidence now covers two English multilabel settings, but they use
|
| 434 |
+
different protocols and cannot be pooled into one score.
|
| 435 |
- The 55-article test split was previously opened; results are exploratory.
|
| 436 |
- Per-label support ranges from 7 to 45 articles and rare-label estimates are
|
| 437 |
unstable.
|
| 438 |
- Three seeds describe run-to-run variation; they are not a confidence
|
| 439 |
interval.
|
| 440 |
+
- The changed MALINT model has post-selection repeated-development evidence but
|
| 441 |
+
no newly opened official held-out score; MALINT is not a depth ablation.
|
| 442 |
+
- MALINT forward latency excludes tokenization, loading, disk, host-to-device
|
| 443 |
+
transfer, CPU, Mac, Arduino, and end-to-end application costs.
|
| 444 |
- Multilingual input support has not been validated as multilingual downstream
|
| 445 |
classification quality for this four-layer root.
|
| 446 |
- Default layer maps are documented structural choices, not universal optima.
|
|
|
|
| 456 |
Hugging Face Hub, WikiText, and SemEval remain the work of their respective
|
| 457 |
authors. No WikiText or SemEval source records are redistributed.
|
| 458 |
|
| 459 |
+
No MALINT row is redistributed. The MALINT task/data attribution follows
|
| 460 |
+
Modzelewski et al., *MALicious INTent Dataset and Inoculating LLMs for Enhanced
|
| 461 |
+
Disinformation Detection*, EACL 2026
|
| 462 |
+
([ACL Anthology](https://aclanthology.org/2026.eacl-long.144/)); the official
|
| 463 |
+
dataset repository states CC BY 4.0 for the dataset.
|
| 464 |
+
|
| 465 |
The WikiText page currently has a license wording discrepancy: metadata lists
|
| 466 |
CC BY-SA 3.0 and GFDL, while prose says CC BY-SA 4.0. Users should inspect
|
| 467 |
[`Salesforce/wikitext`](https://huggingface.co/datasets/Salesforce/wikitext)
|
benchmark/figures/README.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
| 1 |
# Matplotlib benchmark figures
|
| 2 |
|
| 3 |
-
|
|
|
|
|
|
|
| 4 |
PNG files are intended for the Hugging Face model card; PDF files are vector
|
| 5 |
exports for reports. Each plot has a compact source CSV beside it.
|
| 6 |
|
|
@@ -8,6 +10,25 @@ exports for reports. Each plot has a compact source CSV beside it.
|
|
| 8 |
2. Per-label precision/recall/F1 for the task-agnostic base.
|
| 9 |
3. Macro-F1 against article inference time and peak CUDA allocation.
|
| 10 |
4. Parameters and BF16 size across the 24L→8L→6L→4L task-free ladder.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
The 55-article test split was previously opened. Error bars are not confidence
|
| 13 |
intervals. The specialized checkpoint is a separate reference lineage.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Matplotlib benchmark figures
|
| 2 |
|
| 3 |
+
Figures 01–04 are deterministically rendered from the frozen v2 task-agnostic
|
| 4 |
+
transfer bundle. Figures 05–07 are recomputed from frozen propaganda, UNO Q,
|
| 5 |
+
and MALINT aggregate records.
|
| 6 |
PNG files are intended for the Hugging Face model card; PDF files are vector
|
| 7 |
exports for reports. Each plot has a compact source CSV beside it.
|
| 8 |
|
|
|
|
| 10 |
2. Per-label precision/recall/F1 for the task-agnostic base.
|
| 11 |
3. Macro-F1 against article inference time and peak CUDA allocation.
|
| 12 |
4. Parameters and BF16 size across the 24L→8L→6L→4L task-free ladder.
|
| 13 |
+
5. Controlled UNO Q depth resources plus descriptive propaganda 24L/8L/4L
|
| 14 |
+
quality and resources.
|
| 15 |
+
6. MALINT repeated-OOF weighted/macro/micro F1, subset exact match, and
|
| 16 |
+
label-wise accuracy.
|
| 17 |
+
7. MALINT internal weighted-F1 gain beside measured CUDA-forward latency and
|
| 18 |
+
peak-allocation changes.
|
| 19 |
|
| 20 |
The 55-article test split was previously opened. Error bars are not confidence
|
| 21 |
intervals. The specialized checkpoint is a separate reference lineage.
|
| 22 |
+
|
| 23 |
+
`manifest.json` and `source_summary.json` cover figures 01–04.
|
| 24 |
+
`cross_task/manifest.json` and `cross_task/source_summary.json` cover figures
|
| 25 |
+
05–07 and bind every frozen input by SHA-256. The latter bundle is aggregate
|
| 26 |
+
only and contains no raw text, IDs, row labels, logits, probabilities, or
|
| 27 |
+
predictions.
|
| 28 |
+
|
| 29 |
+
## 한국어
|
| 30 |
+
|
| 31 |
+
01–04는 기존 task-free base/propaganda transfer 결과이고, 05–07은 4L 선택 근거와
|
| 32 |
+
MALINT 내부 반복평가를 추가한다. MALINT의 Qwen student는 모두 4L이므로 06은
|
| 33 |
+
4L-vs-8L/24L depth 비교가 아니다. 07은 Mean+KD가 더 빠르지 않으며 제한된 latency와
|
| 34 |
+
peak CUDA allocation 비용이 있음을 함께 표시한다.
|
benchmark/figures/cross_task/05_why_4l_edge_tradeoff.csv
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
scope,model,layers,metric,value,sample_sd,unit,normalized_reference,normalized_value,n,claim_boundary
|
| 2 |
+
uno_q_controlled_v128k_f16_depth_chain,8L,8,artifact_mib,100.0,,percent_of_8l,8L=100,100.0,3,4L is the historical early-path proxy
|
| 3 |
+
uno_q_controlled_v128k_f16_depth_chain,6L,6,artifact_mib,85.59176355424236,,percent_of_8l,8L=100,85.59176355424236,3,4L is the historical early-path proxy
|
| 4 |
+
uno_q_controlled_v128k_f16_depth_chain,4L proxy,4,artifact_mib,72.2607217304372,,percent_of_8l,8L=100,72.2607217304372,3,4L is the historical early-path proxy
|
| 5 |
+
uno_q_controlled_v128k_f16_depth_chain,8L,8,latency_pp128_ms,100.0,,percent_of_8l,8L=100,100.0,3,4L is the historical early-path proxy
|
| 6 |
+
uno_q_controlled_v128k_f16_depth_chain,6L,6,latency_pp128_ms,76.36565126547553,,percent_of_8l,8L=100,76.36565126547553,3,4L is the historical early-path proxy
|
| 7 |
+
uno_q_controlled_v128k_f16_depth_chain,4L proxy,4,latency_pp128_ms,50.16216836129361,,percent_of_8l,8L=100,50.16216836129361,3,4L is the historical early-path proxy
|
| 8 |
+
uno_q_controlled_v128k_f16_depth_chain,8L,8,peak_rss_mib,100.0,,percent_of_8l,8L=100,100.0,3,4L is the historical early-path proxy
|
| 9 |
+
uno_q_controlled_v128k_f16_depth_chain,6L,6,peak_rss_mib,87.50983221571457,,percent_of_8l,8L=100,87.50983221571457,3,4L is the historical early-path proxy
|
| 10 |
+
uno_q_controlled_v128k_f16_depth_chain,4L proxy,4,peak_rss_mib,78.19200407435702,,percent_of_8l,8L=100,78.19200407435702,3,4L is the historical early-path proxy
|
| 11 |
+
uno_q_controlled_v128k_f16_depth_chain,8L,8,mean_core_seconds_per_round,100.0,,percent_of_8l,8L=100,100.0,3,4L is the historical early-path proxy
|
| 12 |
+
uno_q_controlled_v128k_f16_depth_chain,6L,6,mean_core_seconds_per_round,76.16702179915283,,percent_of_8l,8L=100,76.16702179915283,3,4L is the historical early-path proxy
|
| 13 |
+
uno_q_controlled_v128k_f16_depth_chain,4L proxy,4,mean_core_seconds_per_round,51.03561371028788,,percent_of_8l,8L=100,51.03561371028788,3,4L is the historical early-path proxy
|
| 14 |
+
propaganda_operational_endpoint_quality,24L,24,macro_f1,0.5642638295960175,0.006258972649035587,score,,,3,same task protocol; lineage/vocabulary differ
|
| 15 |
+
propaganda_operational_endpoint_quality,24L,24,micro_f1,0.6264844890231888,0.01398932248989461,score,,,3,same task protocol; lineage/vocabulary differ
|
| 16 |
+
propaganda_operational_endpoint_quality,24L,24,subset_exact_match,0.0303030303030303,0.027773186030035393,score,,,3,same task protocol; lineage/vocabulary differ
|
| 17 |
+
propaganda_operational_endpoint_resource,24L,24,test_seconds_per_article,0.586362602903109,0.005979023788731576,seconds_per_article,24L=100,100.0,3,same task protocol; lineage/vocabulary differ
|
| 18 |
+
propaganda_operational_endpoint_resource,24L,24,peak_cuda_allocated_bytes,7635339264.0,0.0,bytes,24L=100,100.0,3,same task protocol; lineage/vocabulary differ
|
| 19 |
+
propaganda_operational_endpoint_resource,24L,24,parameters,752407360,,count,24L=100,100.0,3,same task protocol; lineage/vocabulary differ
|
| 20 |
+
propaganda_operational_endpoint_resource,24L,24,total_wall_seconds,12842.063612925334,94.80225434811064,seconds,24L=100,99.99999999999999,3,same task protocol; lineage/vocabulary differ
|
| 21 |
+
propaganda_operational_endpoint_quality,8L,8,macro_f1,0.5366873350531403,0.017264532146217884,score,,,3,same task protocol; lineage/vocabulary differ
|
| 22 |
+
propaganda_operational_endpoint_quality,8L,8,micro_f1,0.6238830315003392,0.013623163834245318,score,,,3,same task protocol; lineage/vocabulary differ
|
| 23 |
+
propaganda_operational_endpoint_quality,8L,8,subset_exact_match,0.006060606060606061,0.01049727762162956,score,,,3,same task protocol; lineage/vocabulary differ
|
| 24 |
+
propaganda_operational_endpoint_resource,8L,8,test_seconds_per_article,0.206657676709195,0.002071265185964238,seconds_per_article,24L=100,35.24400698237286,3,same task protocol; lineage/vocabulary differ
|
| 25 |
+
propaganda_operational_endpoint_resource,8L,8,peak_cuda_allocated_bytes,4257894400.0,3632373.815194686,bytes,24L=100,55.765621575921635,3,same task protocol; lineage/vocabulary differ
|
| 26 |
+
propaganda_operational_endpoint_resource,8L,8,parameters,420332480,,count,24L=100,55.865014398583234,3,same task protocol; lineage/vocabulary differ
|
| 27 |
+
propaganda_operational_endpoint_resource,8L,8,total_wall_seconds,4291.582853937677,44.49609583978631,seconds,24L=100,33.418171590571056,3,same task protocol; lineage/vocabulary differ
|
| 28 |
+
propaganda_operational_endpoint_quality,4L,4,macro_f1,0.5875679559555907,0.007165753447433348,score,,,3,same task protocol; lineage/vocabulary differ
|
| 29 |
+
propaganda_operational_endpoint_quality,4L,4,micro_f1,0.647978512363484,0.0018739735696546695,score,,,3,same task protocol; lineage/vocabulary differ
|
| 30 |
+
propaganda_operational_endpoint_quality,4L,4,subset_exact_match,0.006060606060606061,0.01049727762162956,score,,,3,same task protocol; lineage/vocabulary differ
|
| 31 |
+
propaganda_operational_endpoint_resource,4L,4,test_seconds_per_article,0.08543555729062946,0.0005980667826383295,seconds_per_article,24L=100,14.570430799582711,3,same task protocol; lineage/vocabulary differ
|
| 32 |
+
propaganda_operational_endpoint_resource,4L,4,peak_cuda_allocated_bytes,2143655936.0,0.0,bytes,24L=100,28.075451029493376,3,same task protocol; lineage/vocabulary differ
|
| 33 |
+
propaganda_operational_endpoint_resource,4L,4,parameters,210903360,,count,24L=100,28.030475406301182,3,same task protocol; lineage/vocabulary differ
|
| 34 |
+
propaganda_operational_endpoint_resource,4L,4,total_wall_seconds,1643.6388439506602,9.226529522850198,seconds,24L=100,12.798868573555138,3,same task protocol; lineage/vocabulary differ
|
benchmark/figures/cross_task/05_why_4l_edge_tradeoff.pdf
ADDED
|
Binary file (27.2 kB). View file
|
|
|
benchmark/figures/cross_task/05_why_4l_edge_tradeoff.png
ADDED
|
Git LFS Details
|
benchmark/figures/cross_task/06_malint_4l_quality.csv
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
arm,arm_label,role,layers,metric,mean,sample_sd,n_repeats,threshold
|
| 2 |
+
rightmost_hard,Rightmost + hard,qwen_4l_student,4,weighted_f1,0.522182806412052,0.014974460565445307,3,0.3
|
| 3 |
+
rightmost_hard,Rightmost + hard,qwen_4l_student,4,macro_f1,0.5109624469249431,0.016275524962473233,3,0.3
|
| 4 |
+
rightmost_hard,Rightmost + hard,qwen_4l_student,4,micro_f1,0.5320256865023298,0.010757300158510568,3,0.3
|
| 5 |
+
rightmost_hard,Rightmost + hard,qwen_4l_student,4,subset_exact_match,0.6403030303030303,0.003674047167570365,3,0.3
|
| 6 |
+
rightmost_hard,Rightmost + hard,qwen_4l_student,4,label_accuracy,0.8746666666666667,0.0012106051124507965,3,0.3
|
| 7 |
+
mean_hard,Mean pool + hard,qwen_4l_student,4,weighted_f1,0.6118049035859879,0.008860176348616246,3,0.3
|
| 8 |
+
mean_hard,Mean pool + hard,qwen_4l_student,4,macro_f1,0.6055064323204021,0.007533985719586246,3,0.3
|
| 9 |
+
mean_hard,Mean pool + hard,qwen_4l_student,4,micro_f1,0.6176900341883843,0.007461802499370809,3,0.3
|
| 10 |
+
mean_hard,Mean pool + hard,qwen_4l_student,4,subset_exact_match,0.6696969696969697,0.005169309730070291,3,0.3
|
| 11 |
+
mean_hard,Mean pool + hard,qwen_4l_student,4,label_accuracy,0.8876363636363637,0.0017533910474532954,3,0.3
|
| 12 |
+
mean_kd,Mean pool + KD,qwen_4l_student,4,weighted_f1,0.6346533875063399,0.011966996735426847,3,0.3
|
| 13 |
+
mean_kd,Mean pool + KD,qwen_4l_student,4,macro_f1,0.6281768437686408,0.014225316120600004,3,0.3
|
| 14 |
+
mean_kd,Mean pool + KD,qwen_4l_student,4,micro_f1,0.6389632888388834,0.009850482866928375,3,0.3
|
| 15 |
+
mean_kd,Mean pool + KD,qwen_4l_student,4,subset_exact_match,0.6557575757575758,0.003192622349349309,3,0.3
|
| 16 |
+
mean_kd,Mean pool + KD,qwen_4l_student,4,label_accuracy,0.8757575757575757,0.0020436018545560413,3,0.3
|
| 17 |
+
teacher,RoBERTa teacher,external_teacher,,weighted_f1,0.6046800630077942,0.01579900095351724,3,0.3
|
| 18 |
+
teacher,RoBERTa teacher,external_teacher,,macro_f1,0.5955721586042237,0.01637303032583919,3,0.3
|
| 19 |
+
teacher,RoBERTa teacher,external_teacher,,micro_f1,0.6046188233688233,0.01776761734134435,3,0.3
|
| 20 |
+
teacher,RoBERTa teacher,external_teacher,,subset_exact_match,0.62,0.005961307749365459,3,0.3
|
| 21 |
+
teacher,RoBERTa teacher,external_teacher,,label_accuracy,0.8507878787878788,0.01087469324477906,3,0.3
|
benchmark/figures/cross_task/06_malint_4l_quality.pdf
ADDED
|
Binary file (26.4 kB). View file
|
|
|
benchmark/figures/cross_task/06_malint_4l_quality.png
ADDED
|
Git LFS Details
|
benchmark/figures/cross_task/07_malint_quality_resource.csv
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
panel,batch,arm,metric,value,sample_sd,unit,ratio_vs_rightmost,percent_change,scope
|
| 2 |
+
quality,,rightmost_hard,weighted_f1,0.522182806412052,0.014974460565445307,score,1.0,0.0,post_selection_internal_repeated_cv
|
| 3 |
+
quality,,mean_kd,weighted_f1,0.6346533875063399,0.011966996735426847,score,1.2153854545060183,21.53854545060183,post_selection_internal_repeated_cv
|
| 4 |
+
resource,1,existing_rightmost,latency_ms_per_example,18.474629627574572,,ms_per_example,1.0,0.0,same_input_cuda_forward_only
|
| 5 |
+
resource,1,existing_rightmost,peak_allocated_bytes,1373150208,,bytes,1.0,0.0,same_input_cuda_forward_only
|
| 6 |
+
resource,1,existing_rightmost,parameter_count,334101824,,count,1.0,0.0,same_input_cuda_forward_only
|
| 7 |
+
resource,1,mean_kd,latency_ms_per_example,19.318781786253957,,ms_per_example,1.0456925078172845,4.569250781728451,same_input_cuda_forward_only
|
| 8 |
+
resource,1,mean_kd,peak_allocated_bytes,1374199296,,bytes,1.0007640009038254,0.07640009038254458,same_input_cuda_forward_only
|
| 9 |
+
resource,1,mean_kd,parameter_count,334101829,,count,1.0000000149654975,1.496549750790166e-06,same_input_cuda_forward_only
|
| 10 |
+
resource,4,existing_rightmost,latency_ms_per_example,4.589751928502863,,ms_per_example,1.0,0.0,same_input_cuda_forward_only
|
| 11 |
+
resource,4,existing_rightmost,peak_allocated_bytes,1454388736,,bytes,1.0,0.0,same_input_cuda_forward_only
|
| 12 |
+
resource,4,existing_rightmost,parameter_count,334101824,,count,1.0,0.0,same_input_cuda_forward_only
|
| 13 |
+
resource,4,mean_kd,latency_ms_per_example,4.671012829289292,,ms_per_example,1.0177048568315403,1.7704856831540328,same_input_cuda_forward_only
|
| 14 |
+
resource,4,mean_kd,peak_allocated_bytes,1455437824,,bytes,1.0007213257185181,0.07213257185181288,same_input_cuda_forward_only
|
| 15 |
+
resource,4,mean_kd,parameter_count,334101829,,count,1.0000000149654975,1.496549750790166e-06,same_input_cuda_forward_only
|
benchmark/figures/cross_task/07_malint_quality_resource.pdf
ADDED
|
Binary file (29.3 kB). View file
|
|
|
benchmark/figures/cross_task/07_malint_quality_resource.png
ADDED
|
Git LFS Details
|
benchmark/figures/cross_task/README.md
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Cross-task Matplotlib figures
|
| 2 |
+
|
| 3 |
+
This directory contains aggregate-only, deterministic plots for the existing
|
| 4 |
+
Qwen3.5 standalone 4L classification-base model card. PNG files are for the
|
| 5 |
+
Hugging Face README; PDF files are vector exports; each figure has a source CSV.
|
| 6 |
+
|
| 7 |
+
1. `05_why_4l_edge_tradeoff`: controlled UNO Q depth-chain resources plus
|
| 8 |
+
descriptive propaganda endpoint quality/resources.
|
| 9 |
+
2. `06_malint_4l_quality`: repeated-development F1 and accuracy-type metrics.
|
| 10 |
+
3. `07_malint_quality_resource`: the primary internal quality contrast and its
|
| 11 |
+
measured CUDA-forward latency/peak-allocation cost.
|
| 12 |
+
|
| 13 |
+
Scientific boundaries are printed inside the figures. In particular, MALINT
|
| 14 |
+
does not compare 4L against 8L or 24L; all Qwen student arms keep four layers.
|
| 15 |
+
No raw text, IDs, row labels, logits, probabilities, or predictions are copied
|
| 16 |
+
into this public bundle.
|
benchmark/figures/cross_task/manifest.json
ADDED
|
@@ -0,0 +1,158 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"figures": [
|
| 3 |
+
"05_why_4l_edge_tradeoff",
|
| 4 |
+
"06_malint_4l_quality",
|
| 5 |
+
"07_malint_quality_resource"
|
| 6 |
+
],
|
| 7 |
+
"files_before_manifest": [
|
| 8 |
+
{
|
| 9 |
+
"bytes": 5442,
|
| 10 |
+
"path": "05_why_4l_edge_tradeoff.csv",
|
| 11 |
+
"sha256": "afd203646513c49140a25ef2a562a064286a4fa5efb495e6a73f83f0106e7933"
|
| 12 |
+
},
|
| 13 |
+
{
|
| 14 |
+
"bytes": 27224,
|
| 15 |
+
"path": "05_why_4l_edge_tradeoff.pdf",
|
| 16 |
+
"sha256": "22f531ea586f9c79ac8f114345cec56e74ea5ed97aa3726f7310018666ae6312"
|
| 17 |
+
},
|
| 18 |
+
{
|
| 19 |
+
"bytes": 178965,
|
| 20 |
+
"path": "05_why_4l_edge_tradeoff.png",
|
| 21 |
+
"sha256": "cdc0a8a98cd13ad35a68b980d2e0750e2424b8932f7d27dc0ecd548fed823c74"
|
| 22 |
+
},
|
| 23 |
+
{
|
| 24 |
+
"bytes": 2118,
|
| 25 |
+
"path": "06_malint_4l_quality.csv",
|
| 26 |
+
"sha256": "25848237b9c77596a8c4b6224fa111f86c816e065804ccfdd6a4fdf6c63ef8d1"
|
| 27 |
+
},
|
| 28 |
+
{
|
| 29 |
+
"bytes": 26351,
|
| 30 |
+
"path": "06_malint_4l_quality.pdf",
|
| 31 |
+
"sha256": "9a1ae78b4e5871f5d5c5321db6c5d1fe0bf349a7869977b2649b44ccea633d04"
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"bytes": 181392,
|
| 35 |
+
"path": "06_malint_4l_quality.png",
|
| 36 |
+
"sha256": "2c5b22415d311ca5fa2b198d55b77f3570b57595d01f0b5031abde59ffc3e9e6"
|
| 37 |
+
},
|
| 38 |
+
{
|
| 39 |
+
"bytes": 1800,
|
| 40 |
+
"path": "07_malint_quality_resource.csv",
|
| 41 |
+
"sha256": "aac7d3400cf8dcb3ac8b6fc317c27ded5343231c93476997011a3bb56b5e4d6c"
|
| 42 |
+
},
|
| 43 |
+
{
|
| 44 |
+
"bytes": 29309,
|
| 45 |
+
"path": "07_malint_quality_resource.pdf",
|
| 46 |
+
"sha256": "498338991976e64038096a12361f63830ced3b9c6eea3e1bdaed8d0e4efac606"
|
| 47 |
+
},
|
| 48 |
+
{
|
| 49 |
+
"bytes": 163562,
|
| 50 |
+
"path": "07_malint_quality_resource.png",
|
| 51 |
+
"sha256": "1cb13cc695f3d59ae9ce75599b86867da624d6e0aae0897e0db8686f6db984ba"
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"bytes": 868,
|
| 55 |
+
"path": "README.md",
|
| 56 |
+
"sha256": "f3e79965f14ec0fd20d11c361d25c3a3abc4c9a20b5dbce9fac0660bb12789f0"
|
| 57 |
+
},
|
| 58 |
+
{
|
| 59 |
+
"bytes": 29752,
|
| 60 |
+
"path": "source_summary.json",
|
| 61 |
+
"sha256": "20ea292f85757440d8641199003c47674deb0c6aacea0ef8f9f5fcd2480bb5d0"
|
| 62 |
+
}
|
| 63 |
+
],
|
| 64 |
+
"inputs": [
|
| 65 |
+
{
|
| 66 |
+
"bytes": 58857,
|
| 67 |
+
"path": "results/malint_qwen35_4l_repeated_cv/20260814-r3x3-terminal-v1/aggregate-v1/aggregate_result.json",
|
| 68 |
+
"role": "malint_repeated_aggregate",
|
| 69 |
+
"sha256": "e3d659c2f5aca645f040d03c57e4effa9eb2616f622aceb503c71236b6b01efa"
|
| 70 |
+
},
|
| 71 |
+
{
|
| 72 |
+
"bytes": 325527,
|
| 73 |
+
"path": "results/malint_qwen35_4l_repeated_cv/20260814-r3x3-terminal-v1/aggregate-v1/aggregate_oof_predictions.npz",
|
| 74 |
+
"role": "malint_repeated_oof",
|
| 75 |
+
"sha256": "a1d7827ebe4beecb49676ca42e0c986c5f4101c394d55c9965161481c347a3f1"
|
| 76 |
+
},
|
| 77 |
+
{
|
| 78 |
+
"bytes": 15407,
|
| 79 |
+
"path": "results/malint_qwen35_4l_resource_benchmark/20260814-matched-bf16-v2/resource_benchmark.json",
|
| 80 |
+
"role": "malint_resource_aggregate",
|
| 81 |
+
"sha256": "5337e44c907a05590c0ea3da6a0760baff22c3f930ed81c069c257060fd2e725"
|
| 82 |
+
},
|
| 83 |
+
{
|
| 84 |
+
"bytes": 3144,
|
| 85 |
+
"path": "results/final_broad_benchmark/20260729-v1/report/quality_matrix.csv",
|
| 86 |
+
"role": "propaganda_quality_matrix",
|
| 87 |
+
"sha256": "c6e316f7081e2a6dcda1ec4e95b57cdcc357ae49ed8b3f5a876c5b51e0df722f"
|
| 88 |
+
},
|
| 89 |
+
{
|
| 90 |
+
"bytes": 115661,
|
| 91 |
+
"path": "results/final_broad_benchmark/20260729-v1/trainable/qwen35-full24l-commonhead/seed41/report.json",
|
| 92 |
+
"role": "propaganda_seed_report_01",
|
| 93 |
+
"sha256": "1c572fb728755dbde0460e6b8eb9a84d078b128f8a42a9148a17c40fc0b27d82"
|
| 94 |
+
},
|
| 95 |
+
{
|
| 96 |
+
"bytes": 116024,
|
| 97 |
+
"path": "results/final_broad_benchmark/20260729-v1/trainable/qwen35-full24l-commonhead/seed42/report.json",
|
| 98 |
+
"role": "propaganda_seed_report_02",
|
| 99 |
+
"sha256": "055edfee92e39928aa9577f239c5b0389f3dda375a96fc69a2c000a0ed6edc36"
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"bytes": 115639,
|
| 103 |
+
"path": "results/final_broad_benchmark/20260729-v1/trainable/qwen35-full24l-commonhead/seed43-retry2/report.json",
|
| 104 |
+
"role": "propaganda_seed_report_03",
|
| 105 |
+
"sha256": "84b2c316556db4aa5fdf7c08433a4808a5fe9ef6f620da6472f4f2f73dee2e4b"
|
| 106 |
+
},
|
| 107 |
+
{
|
| 108 |
+
"bytes": 114924,
|
| 109 |
+
"path": "results/final_broad_benchmark/20260729-v1/trainable/qwen35-c8q8l-commonhead/seed41/report.json",
|
| 110 |
+
"role": "propaganda_seed_report_04",
|
| 111 |
+
"sha256": "9234258589ffb24eaa92859639a66aeb3e55190f8aa82f798d88e791a282bf9c"
|
| 112 |
+
},
|
| 113 |
+
{
|
| 114 |
+
"bytes": 115800,
|
| 115 |
+
"path": "results/final_broad_benchmark/20260729-v1/trainable/qwen35-c8q8l-commonhead/seed42/report.json",
|
| 116 |
+
"role": "propaganda_seed_report_05",
|
| 117 |
+
"sha256": "8161dd1378081de9960136fddab800994529aa3c754674ca41bb4aa71515b3c6"
|
| 118 |
+
},
|
| 119 |
+
{
|
| 120 |
+
"bytes": 116039,
|
| 121 |
+
"path": "results/final_broad_benchmark/20260729-v1/trainable/qwen35-c8q8l-commonhead/seed43/report.json",
|
| 122 |
+
"role": "propaganda_seed_report_06",
|
| 123 |
+
"sha256": "ce0c76a2aa97a5b7c8cdf820af0bd4f8a26bf50d460612ca1db30bd497ca3d9d"
|
| 124 |
+
},
|
| 125 |
+
{
|
| 126 |
+
"bytes": 115985,
|
| 127 |
+
"path": "results/final_broad_benchmark/20260729-v1/trainable/qwen35-standalone4l-commonhead/seed41/report.json",
|
| 128 |
+
"role": "propaganda_seed_report_07",
|
| 129 |
+
"sha256": "ec5314140fff0bdd46e87ee00ad67bee7e61d82c7cfa34add1510dd7d5bdb3c0"
|
| 130 |
+
},
|
| 131 |
+
{
|
| 132 |
+
"bytes": 115972,
|
| 133 |
+
"path": "results/final_broad_benchmark/20260729-v1/trainable/qwen35-standalone4l-commonhead/seed42/report.json",
|
| 134 |
+
"role": "propaganda_seed_report_08",
|
| 135 |
+
"sha256": "e5c45a7dc41bebf4550da7c5763d0759c6d0922da9efda4048564c11c6d2f3af"
|
| 136 |
+
},
|
| 137 |
+
{
|
| 138 |
+
"bytes": 116177,
|
| 139 |
+
"path": "results/final_broad_benchmark/20260729-v1/trainable/qwen35-standalone4l-commonhead/seed43/report.json",
|
| 140 |
+
"role": "propaganda_seed_report_09",
|
| 141 |
+
"sha256": "e05bc2146d720706780fc45a1fb05911d5c2926caa0bcb12ffead946028eef0d"
|
| 142 |
+
},
|
| 143 |
+
{
|
| 144 |
+
"bytes": 827,
|
| 145 |
+
"path": "results/standalone4l_base/20260731-v1/huggingface_release_v3/benchmark/compression_ladder.csv",
|
| 146 |
+
"role": "task_free_compression_ladder",
|
| 147 |
+
"sha256": "2c544c3907c4f592d724f658850b8c6b9634845cb5b50a66ebd2ea6c5d48ce0b"
|
| 148 |
+
},
|
| 149 |
+
{
|
| 150 |
+
"bytes": 13963,
|
| 151 |
+
"path": "results/unoq_lineage_matrix/20260731-v1/analysis/analysis.json",
|
| 152 |
+
"role": "uno_q_depth_analysis",
|
| 153 |
+
"sha256": "c899a19e4433e3a40547a11cfcfc3b0a7b6f00c436aa8c1d742a5dc0399bf649"
|
| 154 |
+
}
|
| 155 |
+
],
|
| 156 |
+
"schema": "qwen35-4l-cross-task-base-card-figures-manifest-v1",
|
| 157 |
+
"status": "complete"
|
| 158 |
+
}
|
benchmark/figures/cross_task/source_summary.json
ADDED
|
@@ -0,0 +1,972 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"claim_boundaries": [
|
| 3 |
+
"The controlled UNO Q 4L point is a historical early-path proxy, not the final common-head quality endpoint.",
|
| 4 |
+
"The propaganda 24L/8L/4L endpoints differ in lineage and vocabulary; their comparison is descriptive, not a pure depth ablation.",
|
| 5 |
+
"All Qwen MALINT student arms are 4L; MALINT supports a 4L interface result, not 4L-versus-8L/24L depth superiority.",
|
| 6 |
+
"The MALINT result is post-selection internal repeated-fold evidence, not untouched held-out confirmation or SOTA evidence.",
|
| 7 |
+
"Mean+KD incurred measured CUDA-forward latency and peak-allocation increases; it is not claimed to be faster.",
|
| 8 |
+
"No ambient GPU utilization value is used as model occupancy."
|
| 9 |
+
],
|
| 10 |
+
"malint_quality": [
|
| 11 |
+
{
|
| 12 |
+
"arm": "rightmost_hard",
|
| 13 |
+
"arm_label": "Rightmost + hard",
|
| 14 |
+
"layers": 4,
|
| 15 |
+
"mean": 0.522182806412052,
|
| 16 |
+
"metric": "weighted_f1",
|
| 17 |
+
"n_repeats": 3,
|
| 18 |
+
"role": "qwen_4l_student",
|
| 19 |
+
"sample_sd": 0.014974460565445307,
|
| 20 |
+
"threshold": 0.3
|
| 21 |
+
},
|
| 22 |
+
{
|
| 23 |
+
"arm": "rightmost_hard",
|
| 24 |
+
"arm_label": "Rightmost + hard",
|
| 25 |
+
"layers": 4,
|
| 26 |
+
"mean": 0.5109624469249431,
|
| 27 |
+
"metric": "macro_f1",
|
| 28 |
+
"n_repeats": 3,
|
| 29 |
+
"role": "qwen_4l_student",
|
| 30 |
+
"sample_sd": 0.016275524962473233,
|
| 31 |
+
"threshold": 0.3
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"arm": "rightmost_hard",
|
| 35 |
+
"arm_label": "Rightmost + hard",
|
| 36 |
+
"layers": 4,
|
| 37 |
+
"mean": 0.5320256865023298,
|
| 38 |
+
"metric": "micro_f1",
|
| 39 |
+
"n_repeats": 3,
|
| 40 |
+
"role": "qwen_4l_student",
|
| 41 |
+
"sample_sd": 0.010757300158510568,
|
| 42 |
+
"threshold": 0.3
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"arm": "rightmost_hard",
|
| 46 |
+
"arm_label": "Rightmost + hard",
|
| 47 |
+
"layers": 4,
|
| 48 |
+
"mean": 0.6403030303030303,
|
| 49 |
+
"metric": "subset_exact_match",
|
| 50 |
+
"n_repeats": 3,
|
| 51 |
+
"role": "qwen_4l_student",
|
| 52 |
+
"sample_sd": 0.003674047167570365,
|
| 53 |
+
"threshold": 0.3
|
| 54 |
+
},
|
| 55 |
+
{
|
| 56 |
+
"arm": "rightmost_hard",
|
| 57 |
+
"arm_label": "Rightmost + hard",
|
| 58 |
+
"layers": 4,
|
| 59 |
+
"mean": 0.8746666666666667,
|
| 60 |
+
"metric": "label_accuracy",
|
| 61 |
+
"n_repeats": 3,
|
| 62 |
+
"role": "qwen_4l_student",
|
| 63 |
+
"sample_sd": 0.0012106051124507965,
|
| 64 |
+
"threshold": 0.3
|
| 65 |
+
},
|
| 66 |
+
{
|
| 67 |
+
"arm": "mean_hard",
|
| 68 |
+
"arm_label": "Mean pool + hard",
|
| 69 |
+
"layers": 4,
|
| 70 |
+
"mean": 0.6118049035859879,
|
| 71 |
+
"metric": "weighted_f1",
|
| 72 |
+
"n_repeats": 3,
|
| 73 |
+
"role": "qwen_4l_student",
|
| 74 |
+
"sample_sd": 0.008860176348616246,
|
| 75 |
+
"threshold": 0.3
|
| 76 |
+
},
|
| 77 |
+
{
|
| 78 |
+
"arm": "mean_hard",
|
| 79 |
+
"arm_label": "Mean pool + hard",
|
| 80 |
+
"layers": 4,
|
| 81 |
+
"mean": 0.6055064323204021,
|
| 82 |
+
"metric": "macro_f1",
|
| 83 |
+
"n_repeats": 3,
|
| 84 |
+
"role": "qwen_4l_student",
|
| 85 |
+
"sample_sd": 0.007533985719586246,
|
| 86 |
+
"threshold": 0.3
|
| 87 |
+
},
|
| 88 |
+
{
|
| 89 |
+
"arm": "mean_hard",
|
| 90 |
+
"arm_label": "Mean pool + hard",
|
| 91 |
+
"layers": 4,
|
| 92 |
+
"mean": 0.6176900341883843,
|
| 93 |
+
"metric": "micro_f1",
|
| 94 |
+
"n_repeats": 3,
|
| 95 |
+
"role": "qwen_4l_student",
|
| 96 |
+
"sample_sd": 0.007461802499370809,
|
| 97 |
+
"threshold": 0.3
|
| 98 |
+
},
|
| 99 |
+
{
|
| 100 |
+
"arm": "mean_hard",
|
| 101 |
+
"arm_label": "Mean pool + hard",
|
| 102 |
+
"layers": 4,
|
| 103 |
+
"mean": 0.6696969696969697,
|
| 104 |
+
"metric": "subset_exact_match",
|
| 105 |
+
"n_repeats": 3,
|
| 106 |
+
"role": "qwen_4l_student",
|
| 107 |
+
"sample_sd": 0.005169309730070291,
|
| 108 |
+
"threshold": 0.3
|
| 109 |
+
},
|
| 110 |
+
{
|
| 111 |
+
"arm": "mean_hard",
|
| 112 |
+
"arm_label": "Mean pool + hard",
|
| 113 |
+
"layers": 4,
|
| 114 |
+
"mean": 0.8876363636363637,
|
| 115 |
+
"metric": "label_accuracy",
|
| 116 |
+
"n_repeats": 3,
|
| 117 |
+
"role": "qwen_4l_student",
|
| 118 |
+
"sample_sd": 0.0017533910474532954,
|
| 119 |
+
"threshold": 0.3
|
| 120 |
+
},
|
| 121 |
+
{
|
| 122 |
+
"arm": "mean_kd",
|
| 123 |
+
"arm_label": "Mean pool + KD",
|
| 124 |
+
"layers": 4,
|
| 125 |
+
"mean": 0.6346533875063399,
|
| 126 |
+
"metric": "weighted_f1",
|
| 127 |
+
"n_repeats": 3,
|
| 128 |
+
"role": "qwen_4l_student",
|
| 129 |
+
"sample_sd": 0.011966996735426847,
|
| 130 |
+
"threshold": 0.3
|
| 131 |
+
},
|
| 132 |
+
{
|
| 133 |
+
"arm": "mean_kd",
|
| 134 |
+
"arm_label": "Mean pool + KD",
|
| 135 |
+
"layers": 4,
|
| 136 |
+
"mean": 0.6281768437686408,
|
| 137 |
+
"metric": "macro_f1",
|
| 138 |
+
"n_repeats": 3,
|
| 139 |
+
"role": "qwen_4l_student",
|
| 140 |
+
"sample_sd": 0.014225316120600004,
|
| 141 |
+
"threshold": 0.3
|
| 142 |
+
},
|
| 143 |
+
{
|
| 144 |
+
"arm": "mean_kd",
|
| 145 |
+
"arm_label": "Mean pool + KD",
|
| 146 |
+
"layers": 4,
|
| 147 |
+
"mean": 0.6389632888388834,
|
| 148 |
+
"metric": "micro_f1",
|
| 149 |
+
"n_repeats": 3,
|
| 150 |
+
"role": "qwen_4l_student",
|
| 151 |
+
"sample_sd": 0.009850482866928375,
|
| 152 |
+
"threshold": 0.3
|
| 153 |
+
},
|
| 154 |
+
{
|
| 155 |
+
"arm": "mean_kd",
|
| 156 |
+
"arm_label": "Mean pool + KD",
|
| 157 |
+
"layers": 4,
|
| 158 |
+
"mean": 0.6557575757575758,
|
| 159 |
+
"metric": "subset_exact_match",
|
| 160 |
+
"n_repeats": 3,
|
| 161 |
+
"role": "qwen_4l_student",
|
| 162 |
+
"sample_sd": 0.003192622349349309,
|
| 163 |
+
"threshold": 0.3
|
| 164 |
+
},
|
| 165 |
+
{
|
| 166 |
+
"arm": "mean_kd",
|
| 167 |
+
"arm_label": "Mean pool + KD",
|
| 168 |
+
"layers": 4,
|
| 169 |
+
"mean": 0.8757575757575757,
|
| 170 |
+
"metric": "label_accuracy",
|
| 171 |
+
"n_repeats": 3,
|
| 172 |
+
"role": "qwen_4l_student",
|
| 173 |
+
"sample_sd": 0.0020436018545560413,
|
| 174 |
+
"threshold": 0.3
|
| 175 |
+
},
|
| 176 |
+
{
|
| 177 |
+
"arm": "teacher",
|
| 178 |
+
"arm_label": "RoBERTa teacher",
|
| 179 |
+
"layers": "",
|
| 180 |
+
"mean": 0.6046800630077942,
|
| 181 |
+
"metric": "weighted_f1",
|
| 182 |
+
"n_repeats": 3,
|
| 183 |
+
"role": "external_teacher",
|
| 184 |
+
"sample_sd": 0.01579900095351724,
|
| 185 |
+
"threshold": 0.3
|
| 186 |
+
},
|
| 187 |
+
{
|
| 188 |
+
"arm": "teacher",
|
| 189 |
+
"arm_label": "RoBERTa teacher",
|
| 190 |
+
"layers": "",
|
| 191 |
+
"mean": 0.5955721586042237,
|
| 192 |
+
"metric": "macro_f1",
|
| 193 |
+
"n_repeats": 3,
|
| 194 |
+
"role": "external_teacher",
|
| 195 |
+
"sample_sd": 0.01637303032583919,
|
| 196 |
+
"threshold": 0.3
|
| 197 |
+
},
|
| 198 |
+
{
|
| 199 |
+
"arm": "teacher",
|
| 200 |
+
"arm_label": "RoBERTa teacher",
|
| 201 |
+
"layers": "",
|
| 202 |
+
"mean": 0.6046188233688233,
|
| 203 |
+
"metric": "micro_f1",
|
| 204 |
+
"n_repeats": 3,
|
| 205 |
+
"role": "external_teacher",
|
| 206 |
+
"sample_sd": 0.01776761734134435,
|
| 207 |
+
"threshold": 0.3
|
| 208 |
+
},
|
| 209 |
+
{
|
| 210 |
+
"arm": "teacher",
|
| 211 |
+
"arm_label": "RoBERTa teacher",
|
| 212 |
+
"layers": "",
|
| 213 |
+
"mean": 0.62,
|
| 214 |
+
"metric": "subset_exact_match",
|
| 215 |
+
"n_repeats": 3,
|
| 216 |
+
"role": "external_teacher",
|
| 217 |
+
"sample_sd": 0.005961307749365459,
|
| 218 |
+
"threshold": 0.3
|
| 219 |
+
},
|
| 220 |
+
{
|
| 221 |
+
"arm": "teacher",
|
| 222 |
+
"arm_label": "RoBERTa teacher",
|
| 223 |
+
"layers": "",
|
| 224 |
+
"mean": 0.8507878787878788,
|
| 225 |
+
"metric": "label_accuracy",
|
| 226 |
+
"n_repeats": 3,
|
| 227 |
+
"role": "external_teacher",
|
| 228 |
+
"sample_sd": 0.01087469324477906,
|
| 229 |
+
"threshold": 0.3
|
| 230 |
+
}
|
| 231 |
+
],
|
| 232 |
+
"malint_quality_contract": {
|
| 233 |
+
"claim_boundary": "Post-selection internal repeated-fold robustness and matched ablation only. This result is not untouched confirmation, official held-out evidence, all-label superiority, or SOTA evidence.",
|
| 234 |
+
"contrasts": {
|
| 235 |
+
"conditional_kd_mean_kd_minus_mean_hard": {
|
| 236 |
+
"corrected_resampled_t_interval": {
|
| 237 |
+
"confidence": 0.95,
|
| 238 |
+
"critical_t": 2.306004135204166,
|
| 239 |
+
"degrees_of_freedom": 8,
|
| 240 |
+
"formula": "sqrt((1/n + mean(n_eval/n_train)) * sample_variance)",
|
| 241 |
+
"interval": [
|
| 242 |
+
-0.0015478418507784523,
|
| 243 |
+
0.05059146195651548
|
| 244 |
+
],
|
| 245 |
+
"mean": 0.024521810052868513,
|
| 246 |
+
"mean_eval_train_ratio": 0.5000006195533516,
|
| 247 |
+
"n": 9,
|
| 248 |
+
"sample_variance": 0.00020913642539219796,
|
| 249 |
+
"standard_error": 0.011305119321192738,
|
| 250 |
+
"variance_correction": 0.6111117306644627
|
| 251 |
+
},
|
| 252 |
+
"fold_deltas": [
|
| 253 |
+
0.04524702943141845,
|
| 254 |
+
0.017956307877355804,
|
| 255 |
+
-0.0053068243793217285,
|
| 256 |
+
0.015864642448092825,
|
| 257 |
+
0.022064206421293142,
|
| 258 |
+
0.03213041489613799,
|
| 259 |
+
0.02962871459506977,
|
| 260 |
+
0.026719118927120977,
|
| 261 |
+
0.03639268025864939
|
| 262 |
+
],
|
| 263 |
+
"fold_mean": 0.024521810052868513,
|
| 264 |
+
"fold_sample_standard_deviation": 0.014461549895920491,
|
| 265 |
+
"name": "conditional_kd_mean_kd_minus_mean_hard",
|
| 266 |
+
"per_label_mean_delta": {
|
| 267 |
+
"CPV": 0.0006438268748058898,
|
| 268 |
+
"PASV": 0.027106289374744647,
|
| 269 |
+
"PSSA": 0.058741661936162225,
|
| 270 |
+
"UCPI": 0.0247139382339762,
|
| 271 |
+
"UIOA": 0.0021463408215049995
|
| 272 |
+
},
|
| 273 |
+
"per_label_repeat_deltas": [
|
| 274 |
+
[
|
| 275 |
+
-0.01842723004694835,
|
| 276 |
+
0.04865531704909942,
|
| 277 |
+
0.015803620583568212,
|
| 278 |
+
-0.013921424268790505,
|
| 279 |
+
0.03679497238444562
|
| 280 |
+
],
|
| 281 |
+
[
|
| 282 |
+
0.025183733463988223,
|
| 283 |
+
0.070038654494613,
|
| 284 |
+
-0.0026296349009481412,
|
| 285 |
+
0.021790722761596548,
|
| 286 |
+
0.0004450534649754756
|
| 287 |
+
],
|
| 288 |
+
[
|
| 289 |
+
-0.004825022792622202,
|
| 290 |
+
0.05753101426477425,
|
| 291 |
+
-0.006734963218105072,
|
| 292 |
+
0.0734495696314279,
|
| 293 |
+
0.0369017888525075
|
| 294 |
+
]
|
| 295 |
+
],
|
| 296 |
+
"positive_fold_tail": {
|
| 297 |
+
"null_probability": 0.5,
|
| 298 |
+
"one_sided_probability_at_least_observed": 0.01953125,
|
| 299 |
+
"positive": 8,
|
| 300 |
+
"total": 9
|
| 301 |
+
},
|
| 302 |
+
"repeat_oof_deltas": [
|
| 303 |
+
0.018545164631776245,
|
| 304 |
+
0.020840853227636957,
|
| 305 |
+
0.029159433901642884
|
| 306 |
+
],
|
| 307 |
+
"repeat_oof_t_interval": {
|
| 308 |
+
"confidence": 0.95,
|
| 309 |
+
"critical_t": 4.302652729749462,
|
| 310 |
+
"degrees_of_freedom": 2,
|
| 311 |
+
"interval": [
|
| 312 |
+
0.00897537793561895,
|
| 313 |
+
0.03672158990508511
|
| 314 |
+
],
|
| 315 |
+
"mean": 0.02284848392035203,
|
| 316 |
+
"n": 3,
|
| 317 |
+
"sample_standard_deviation": 0.005584676694496962,
|
| 318 |
+
"standard_error": 0.0032243145929048505
|
| 319 |
+
}
|
| 320 |
+
},
|
| 321 |
+
"mean_kd_minus_teacher": {
|
| 322 |
+
"corrected_resampled_t_interval": {
|
| 323 |
+
"confidence": 0.95,
|
| 324 |
+
"critical_t": 2.306004135204166,
|
| 325 |
+
"degrees_of_freedom": 8,
|
| 326 |
+
"formula": "sqrt((1/n + mean(n_eval/n_train)) * sample_variance)",
|
| 327 |
+
"interval": [
|
| 328 |
+
-0.03180974553628697,
|
| 329 |
+
0.08853586985785872
|
| 330 |
+
],
|
| 331 |
+
"mean": 0.02836306216078588,
|
| 332 |
+
"mean_eval_train_ratio": 0.5000006195533516,
|
| 333 |
+
"n": 9,
|
| 334 |
+
"sample_variance": 0.0011141913151793519,
|
| 335 |
+
"standard_error": 0.026093972156622066,
|
| 336 |
+
"variance_correction": 0.6111117306644627
|
| 337 |
+
},
|
| 338 |
+
"fold_deltas": [
|
| 339 |
+
-0.03162698747672954,
|
| 340 |
+
0.06379890775996899,
|
| 341 |
+
0.056964918230746675,
|
| 342 |
+
0.04693837159160652,
|
| 343 |
+
0.001345287770746495,
|
| 344 |
+
0.014058020214096523,
|
| 345 |
+
0.00030085368163068615,
|
| 346 |
+
0.05808798200431842,
|
| 347 |
+
0.04540020567068814
|
| 348 |
+
],
|
| 349 |
+
"fold_mean": 0.02836306216078588,
|
| 350 |
+
"fold_sample_standard_deviation": 0.0333795044178213,
|
| 351 |
+
"name": "mean_kd_minus_teacher",
|
| 352 |
+
"per_label_mean_delta": {
|
| 353 |
+
"CPV": 0.03245133981428311,
|
| 354 |
+
"PASV": 0.04215462872506418,
|
| 355 |
+
"PSSA": 0.030550793916265178,
|
| 356 |
+
"UCPI": 0.006157663285133859,
|
| 357 |
+
"UIOA": 0.05170900008133931
|
| 358 |
+
},
|
| 359 |
+
"per_label_repeat_deltas": [
|
| 360 |
+
[
|
| 361 |
+
0.010081541882876188,
|
| 362 |
+
0.09069144184384492,
|
| 363 |
+
0.058604490711147705,
|
| 364 |
+
-0.005178527872112815,
|
| 365 |
+
0.0012970008399624255
|
| 366 |
+
],
|
| 367 |
+
[
|
| 368 |
+
0.05816392692322364,
|
| 369 |
+
-0.013100009776126686,
|
| 370 |
+
0.017979371387083054,
|
| 371 |
+
0.03578643578643581,
|
| 372 |
+
0.01576007056748019
|
| 373 |
+
],
|
| 374 |
+
[
|
| 375 |
+
0.029108550636749497,
|
| 376 |
+
0.014060949681077295,
|
| 377 |
+
0.07854313814578717,
|
| 378 |
+
0.09585597826086956,
|
| 379 |
+
0.0014159184479589637
|
| 380 |
+
]
|
| 381 |
+
],
|
| 382 |
+
"positive_fold_tail": {
|
| 383 |
+
"null_probability": 0.5,
|
| 384 |
+
"one_sided_probability_at_least_observed": 0.01953125,
|
| 385 |
+
"positive": 8,
|
| 386 |
+
"total": 9
|
| 387 |
+
},
|
| 388 |
+
"repeat_oof_deltas": [
|
| 389 |
+
0.032162246558158025,
|
| 390 |
+
0.020416569493259984,
|
| 391 |
+
0.037341157444219175
|
| 392 |
+
],
|
| 393 |
+
"repeat_oof_t_interval": {
|
| 394 |
+
"confidence": 0.95,
|
| 395 |
+
"critical_t": 4.302652729749462,
|
| 396 |
+
"degrees_of_freedom": 2,
|
| 397 |
+
"interval": [
|
| 398 |
+
0.00843082924454409,
|
| 399 |
+
0.05151581975254736
|
| 400 |
+
],
|
| 401 |
+
"mean": 0.029973324498545728,
|
| 402 |
+
"n": 3,
|
| 403 |
+
"sample_standard_deviation": 0.008672021342497449,
|
| 404 |
+
"standard_error": 0.005006793856509082
|
| 405 |
+
}
|
| 406 |
+
},
|
| 407 |
+
"pooling_mean_hard_minus_rightmost_hard": {
|
| 408 |
+
"corrected_resampled_t_interval": {
|
| 409 |
+
"confidence": 0.95,
|
| 410 |
+
"critical_t": 2.306004135204166,
|
| 411 |
+
"degrees_of_freedom": 8,
|
| 412 |
+
"formula": "sqrt((1/n + mean(n_eval/n_train)) * sample_variance)",
|
| 413 |
+
"interval": [
|
| 414 |
+
0.056762831891190316,
|
| 415 |
+
0.12504455147590826
|
| 416 |
+
],
|
| 417 |
+
"mean": 0.0909036916835493,
|
| 418 |
+
"mean_eval_train_ratio": 0.5000006195533516,
|
| 419 |
+
"n": 9,
|
| 420 |
+
"sample_variance": 0.00035868079546522353,
|
| 421 |
+
"standard_error": 0.014805203195932804,
|
| 422 |
+
"variance_correction": 0.6111117306644627
|
| 423 |
+
},
|
| 424 |
+
"fold_deltas": [
|
| 425 |
+
0.06678423059606675,
|
| 426 |
+
0.08178035353463753,
|
| 427 |
+
0.08672762777739151,
|
| 428 |
+
0.09344774812516743,
|
| 429 |
+
0.0878998694863643,
|
| 430 |
+
0.07415570161139795,
|
| 431 |
+
0.11094545718958193,
|
| 432 |
+
0.08708564234043492,
|
| 433 |
+
0.12930659449090132
|
| 434 |
+
],
|
| 435 |
+
"fold_mean": 0.0909036916835493,
|
| 436 |
+
"fold_sample_standard_deviation": 0.01893886996273071,
|
| 437 |
+
"name": "pooling_mean_hard_minus_rightmost_hard",
|
| 438 |
+
"per_label_mean_delta": {
|
| 439 |
+
"CPV": 0.14860808426516173,
|
| 440 |
+
"PASV": 0.09778905085334237,
|
| 441 |
+
"PSSA": 0.07923875253833933,
|
| 442 |
+
"UCPI": 0.055020095369494,
|
| 443 |
+
"UIOA": 0.09206394395095778
|
| 444 |
+
},
|
| 445 |
+
"per_label_repeat_deltas": [
|
| 446 |
+
[
|
| 447 |
+
0.09436936936936935,
|
| 448 |
+
0.07977821265492502,
|
| 449 |
+
0.10417829690385194,
|
| 450 |
+
0.07872402692420255,
|
| 451 |
+
0.03863953001237319
|
| 452 |
+
],
|
| 453 |
+
[
|
| 454 |
+
0.15668016194331985,
|
| 455 |
+
0.04167051934042232,
|
| 456 |
+
0.06934192200557099,
|
| 457 |
+
0.11942791523123497,
|
| 458 |
+
0.06908520415955133
|
| 459 |
+
],
|
| 460 |
+
[
|
| 461 |
+
0.194774721482796,
|
| 462 |
+
0.11626752561967063,
|
| 463 |
+
0.1026716129434504,
|
| 464 |
+
0.09521521040458958,
|
| 465 |
+
0.05733555193655748
|
| 466 |
+
]
|
| 467 |
+
],
|
| 468 |
+
"positive_fold_tail": {
|
| 469 |
+
"null_probability": 0.5,
|
| 470 |
+
"one_sided_probability_at_least_observed": 0.001953125,
|
| 471 |
+
"positive": 9,
|
| 472 |
+
"total": 9
|
| 473 |
+
},
|
| 474 |
+
"repeat_oof_deltas": [
|
| 475 |
+
0.07574451062131737,
|
| 476 |
+
0.08540665760778032,
|
| 477 |
+
0.10771512329271016
|
| 478 |
+
],
|
| 479 |
+
"repeat_oof_t_interval": {
|
| 480 |
+
"confidence": 0.95,
|
| 481 |
+
"critical_t": 4.302652729749462,
|
| 482 |
+
"degrees_of_freedom": 2,
|
| 483 |
+
"interval": [
|
| 484 |
+
0.048890004893968025,
|
| 485 |
+
0.13035418945390387
|
| 486 |
+
],
|
| 487 |
+
"mean": 0.08962209717393595,
|
| 488 |
+
"n": 3,
|
| 489 |
+
"sample_standard_deviation": 0.01639687368671199,
|
| 490 |
+
"standard_error": 0.009466739436891459
|
| 491 |
+
}
|
| 492 |
+
},
|
| 493 |
+
"primary_mean_kd_minus_rightmost_hard": {
|
| 494 |
+
"corrected_resampled_t_interval": {
|
| 495 |
+
"confidence": 0.95,
|
| 496 |
+
"critical_t": 2.306004135204166,
|
| 497 |
+
"degrees_of_freedom": 8,
|
| 498 |
+
"formula": "sqrt((1/n + mean(n_eval/n_train)) * sample_variance)",
|
| 499 |
+
"interval": [
|
| 500 |
+
0.07160480852626178,
|
| 501 |
+
0.15924619494657383
|
| 502 |
+
],
|
| 503 |
+
"mean": 0.1154255017364178,
|
| 504 |
+
"mean_eval_train_ratio": 0.5000006195533516,
|
| 505 |
+
"n": 9,
|
| 506 |
+
"sample_variance": 0.0005909050521203029,
|
| 507 |
+
"standard_error": 0.019002868442938105,
|
| 508 |
+
"variance_correction": 0.6111117306644627
|
| 509 |
+
},
|
| 510 |
+
"fold_deltas": [
|
| 511 |
+
0.1120312600274852,
|
| 512 |
+
0.09973666141199333,
|
| 513 |
+
0.08142080339806979,
|
| 514 |
+
0.10931239057326025,
|
| 515 |
+
0.10996407590765744,
|
| 516 |
+
0.10628611650753594,
|
| 517 |
+
0.1405741717846517,
|
| 518 |
+
0.1138047612675559,
|
| 519 |
+
0.1656992747495507
|
| 520 |
+
],
|
| 521 |
+
"fold_mean": 0.1154255017364178,
|
| 522 |
+
"fold_sample_standard_deviation": 0.024308538666902683,
|
| 523 |
+
"name": "primary_mean_kd_minus_rightmost_hard",
|
| 524 |
+
"per_label_mean_delta": {
|
| 525 |
+
"CPV": 0.14925191113996764,
|
| 526 |
+
"PASV": 0.12489534022808702,
|
| 527 |
+
"PSSA": 0.13798041447450154,
|
| 528 |
+
"UCPI": 0.0797340336034702,
|
| 529 |
+
"UIOA": 0.09421028477246278
|
| 530 |
+
},
|
| 531 |
+
"per_label_repeat_deltas": [
|
| 532 |
+
[
|
| 533 |
+
0.075942139322421,
|
| 534 |
+
0.12843352970402444,
|
| 535 |
+
0.11998191748742015,
|
| 536 |
+
0.06480260265541204,
|
| 537 |
+
0.07543450239681881
|
| 538 |
+
],
|
| 539 |
+
[
|
| 540 |
+
0.18186389540730807,
|
| 541 |
+
0.11170917383503531,
|
| 542 |
+
0.06671228710462285,
|
| 543 |
+
0.14121863799283152,
|
| 544 |
+
0.0695302576245268
|
| 545 |
+
],
|
| 546 |
+
[
|
| 547 |
+
0.1899496986901738,
|
| 548 |
+
0.17379853988444488,
|
| 549 |
+
0.09593664972534532,
|
| 550 |
+
0.16866478003601748,
|
| 551 |
+
0.09423734078906498
|
| 552 |
+
]
|
| 553 |
+
],
|
| 554 |
+
"positive_fold_tail": {
|
| 555 |
+
"null_probability": 0.5,
|
| 556 |
+
"one_sided_probability_at_least_observed": 0.001953125,
|
| 557 |
+
"positive": 9,
|
| 558 |
+
"total": 9
|
| 559 |
+
},
|
| 560 |
+
"repeat_oof_deltas": [
|
| 561 |
+
0.09428967525309362,
|
| 562 |
+
0.10624751083541728,
|
| 563 |
+
0.13687455719435304
|
| 564 |
+
],
|
| 565 |
+
"repeat_oof_t_interval": {
|
| 566 |
+
"confidence": 0.95,
|
| 567 |
+
"critical_t": 4.302652729749462,
|
| 568 |
+
"degrees_of_freedom": 2,
|
| 569 |
+
"interval": [
|
| 570 |
+
0.05790922361228608,
|
| 571 |
+
0.1670319385762899
|
| 572 |
+
],
|
| 573 |
+
"mean": 0.11247058109428798,
|
| 574 |
+
"n": 3,
|
| 575 |
+
"sample_standard_deviation": 0.02196390209122417,
|
| 576 |
+
"standard_error": 0.01268086478482286
|
| 577 |
+
}
|
| 578 |
+
}
|
| 579 |
+
},
|
| 580 |
+
"promotion_gate": {
|
| 581 |
+
"fold_gate_passed": true,
|
| 582 |
+
"observed_positive_fold_deltas": 9,
|
| 583 |
+
"observed_positive_repeat_oof_deltas": 3,
|
| 584 |
+
"passed": true,
|
| 585 |
+
"repeat_gate_passed": true,
|
| 586 |
+
"required_positive_fold_deltas": 7,
|
| 587 |
+
"required_positive_repeat_oof_deltas": 3
|
| 588 |
+
},
|
| 589 |
+
"study": {
|
| 590 |
+
"fold_count": 9,
|
| 591 |
+
"fold_runner_protocol": "malint-qwen35-4l-repeated-matched-cv/1.0",
|
| 592 |
+
"fold_runner_sha256": "c71af81f1182ce860c39c107dd19eff02f1addd8dd646e3eebaadedc4503067c",
|
| 593 |
+
"heldout_split_accessed": false,
|
| 594 |
+
"post_selection_internal_evidence": true,
|
| 595 |
+
"raw_text_written": false,
|
| 596 |
+
"repeat_count": 3,
|
| 597 |
+
"rows_per_repeat": 1100,
|
| 598 |
+
"terminal_evaluation_only": true,
|
| 599 |
+
"threshold": 0.3
|
| 600 |
+
}
|
| 601 |
+
},
|
| 602 |
+
"malint_resources": {
|
| 603 |
+
"all_resource_parity_gates_passed": true,
|
| 604 |
+
"claim_boundary": {
|
| 605 |
+
"development_labels_read": false,
|
| 606 |
+
"heldout_access": false,
|
| 607 |
+
"quality_scored": false,
|
| 608 |
+
"resource_only": true,
|
| 609 |
+
"sota_claim_supported": false,
|
| 610 |
+
"timing_scope": "CUDA model forward with CUDA-resident token IDs and masks; tokenization, loading, disk, and host-to-device transfer excluded"
|
| 611 |
+
},
|
| 612 |
+
"comparisons": {
|
| 613 |
+
"1": {
|
| 614 |
+
"existing_over_mean_kd_throughput_ratio": 1.0456925078172845,
|
| 615 |
+
"mean_kd_minus_existing_parameters": 5,
|
| 616 |
+
"mean_kd_minus_existing_serialized_weight_bytes": -220,
|
| 617 |
+
"mean_kd_over_existing_latency_ratio": 1.0456925078172845,
|
| 618 |
+
"mean_kd_over_existing_peak_allocated_ratio": 1.0007640009038254,
|
| 619 |
+
"paired_round_latency_ratio_summary": {
|
| 620 |
+
"maximum": 1.0527066487704841,
|
| 621 |
+
"mean": 1.0488242542281458,
|
| 622 |
+
"median": 1.0511611345543765,
|
| 623 |
+
"minimum": 1.042604979359577,
|
| 624 |
+
"n": 3,
|
| 625 |
+
"p05": 1.043460594879057,
|
| 626 |
+
"p95": 1.0525520973488733,
|
| 627 |
+
"sample_standard_deviation": 0.005441202851039159
|
| 628 |
+
},
|
| 629 |
+
"paired_round_latency_ratios": [
|
| 630 |
+
1.0511611345543765,
|
| 631 |
+
1.042604979359577,
|
| 632 |
+
1.0527066487704841
|
| 633 |
+
],
|
| 634 |
+
"resource_parity_gate": {
|
| 635 |
+
"four_layers_and_length_256_pass": true,
|
| 636 |
+
"latency_pass": true,
|
| 637 |
+
"latency_ratio": 1.0456925078172845,
|
| 638 |
+
"latency_ratio_limit": 1.05,
|
| 639 |
+
"parameter_pass": true,
|
| 640 |
+
"parameter_relative_difference": 1.4965497464629228e-08,
|
| 641 |
+
"parameter_relative_limit": 0.0001,
|
| 642 |
+
"passed": true,
|
| 643 |
+
"peak_allocated_ratio": 1.0007640009038254,
|
| 644 |
+
"peak_memory_pass": true,
|
| 645 |
+
"peak_memory_ratio_limit": 1.05
|
| 646 |
+
}
|
| 647 |
+
},
|
| 648 |
+
"4": {
|
| 649 |
+
"existing_over_mean_kd_throughput_ratio": 1.0177048568315403,
|
| 650 |
+
"mean_kd_minus_existing_parameters": 5,
|
| 651 |
+
"mean_kd_minus_existing_serialized_weight_bytes": -220,
|
| 652 |
+
"mean_kd_over_existing_latency_ratio": 1.0177048568315403,
|
| 653 |
+
"mean_kd_over_existing_peak_allocated_ratio": 1.0007213257185181,
|
| 654 |
+
"paired_round_latency_ratio_summary": {
|
| 655 |
+
"maximum": 1.0378811577131521,
|
| 656 |
+
"mean": 1.0299336811276214,
|
| 657 |
+
"median": 1.0272510844529572,
|
| 658 |
+
"minimum": 1.0246688012167542,
|
| 659 |
+
"n": 3,
|
| 660 |
+
"p05": 1.0249270295403745,
|
| 661 |
+
"p95": 1.0368181503871325,
|
| 662 |
+
"sample_standard_deviation": 0.007002773360331318
|
| 663 |
+
},
|
| 664 |
+
"paired_round_latency_ratios": [
|
| 665 |
+
1.0272510844529572,
|
| 666 |
+
1.0378811577131521,
|
| 667 |
+
1.0246688012167542
|
| 668 |
+
],
|
| 669 |
+
"resource_parity_gate": {
|
| 670 |
+
"four_layers_and_length_256_pass": true,
|
| 671 |
+
"latency_pass": true,
|
| 672 |
+
"latency_ratio": 1.0177048568315403,
|
| 673 |
+
"latency_ratio_limit": 1.05,
|
| 674 |
+
"parameter_pass": true,
|
| 675 |
+
"parameter_relative_difference": 1.4965497464629228e-08,
|
| 676 |
+
"parameter_relative_limit": 0.0001,
|
| 677 |
+
"passed": true,
|
| 678 |
+
"peak_allocated_ratio": 1.0007213257185181,
|
| 679 |
+
"peak_memory_pass": true,
|
| 680 |
+
"peak_memory_ratio_limit": 1.05
|
| 681 |
+
}
|
| 682 |
+
}
|
| 683 |
+
},
|
| 684 |
+
"rows": [
|
| 685 |
+
{
|
| 686 |
+
"arm": "existing_rightmost",
|
| 687 |
+
"batch": 1,
|
| 688 |
+
"latency_ms_per_example": 18.474629627574572,
|
| 689 |
+
"parameter_count": 334101824,
|
| 690 |
+
"peak_allocated_bytes": 1373150208,
|
| 691 |
+
"worker_count": 3
|
| 692 |
+
},
|
| 693 |
+
{
|
| 694 |
+
"arm": "mean_kd",
|
| 695 |
+
"batch": 1,
|
| 696 |
+
"latency_ms_per_example": 19.318781786253957,
|
| 697 |
+
"parameter_count": 334101829,
|
| 698 |
+
"peak_allocated_bytes": 1374199296,
|
| 699 |
+
"worker_count": 3
|
| 700 |
+
},
|
| 701 |
+
{
|
| 702 |
+
"arm": "existing_rightmost",
|
| 703 |
+
"batch": 4,
|
| 704 |
+
"latency_ms_per_example": 4.589751928502863,
|
| 705 |
+
"parameter_count": 334101824,
|
| 706 |
+
"peak_allocated_bytes": 1454388736,
|
| 707 |
+
"worker_count": 3
|
| 708 |
+
},
|
| 709 |
+
{
|
| 710 |
+
"arm": "mean_kd",
|
| 711 |
+
"batch": 4,
|
| 712 |
+
"latency_ms_per_example": 4.671012829289292,
|
| 713 |
+
"parameter_count": 334101829,
|
| 714 |
+
"peak_allocated_bytes": 1455437824,
|
| 715 |
+
"worker_count": 3
|
| 716 |
+
}
|
| 717 |
+
]
|
| 718 |
+
},
|
| 719 |
+
"privacy": {
|
| 720 |
+
"aggregate_only": true,
|
| 721 |
+
"contains_article_ids": false,
|
| 722 |
+
"contains_logits_probabilities_or_predictions": false,
|
| 723 |
+
"contains_raw_text": false,
|
| 724 |
+
"contains_row_labels": false
|
| 725 |
+
},
|
| 726 |
+
"propaganda_operational_endpoints": [
|
| 727 |
+
{
|
| 728 |
+
"arm": "qwen35-full24l-commonhead",
|
| 729 |
+
"label": "24L",
|
| 730 |
+
"layers": 24,
|
| 731 |
+
"lineage_note": "same five-epoch common-head protocol",
|
| 732 |
+
"macro_f1": {
|
| 733 |
+
"mean": 0.5642638295960175,
|
| 734 |
+
"n": 3,
|
| 735 |
+
"sample_sd": 0.006258972649035587
|
| 736 |
+
},
|
| 737 |
+
"micro_f1": {
|
| 738 |
+
"mean": 0.6264844890231888,
|
| 739 |
+
"n": 3,
|
| 740 |
+
"sample_sd": 0.01398932248989461
|
| 741 |
+
},
|
| 742 |
+
"parameters": 752407360,
|
| 743 |
+
"peak_cuda_allocated_bytes": {
|
| 744 |
+
"mean": 7635339264.0,
|
| 745 |
+
"n": 3,
|
| 746 |
+
"sample_sd": 0.0
|
| 747 |
+
},
|
| 748 |
+
"subset_exact_match": {
|
| 749 |
+
"mean": 0.0303030303030303,
|
| 750 |
+
"n": 3,
|
| 751 |
+
"sample_sd": 0.027773186030035393
|
| 752 |
+
},
|
| 753 |
+
"test_articles": 55,
|
| 754 |
+
"test_seconds_per_article": {
|
| 755 |
+
"mean": 0.586362602903109,
|
| 756 |
+
"n": 3,
|
| 757 |
+
"sample_sd": 0.005979023788731576
|
| 758 |
+
},
|
| 759 |
+
"total_wall_seconds": {
|
| 760 |
+
"mean": 12842.063612925334,
|
| 761 |
+
"n": 3,
|
| 762 |
+
"sample_sd": 94.80225434811064
|
| 763 |
+
}
|
| 764 |
+
},
|
| 765 |
+
{
|
| 766 |
+
"arm": "qwen35-c8q8l-commonhead",
|
| 767 |
+
"label": "8L",
|
| 768 |
+
"layers": 8,
|
| 769 |
+
"lineage_note": "same five-epoch common-head protocol",
|
| 770 |
+
"macro_f1": {
|
| 771 |
+
"mean": 0.5366873350531403,
|
| 772 |
+
"n": 3,
|
| 773 |
+
"sample_sd": 0.017264532146217884
|
| 774 |
+
},
|
| 775 |
+
"micro_f1": {
|
| 776 |
+
"mean": 0.6238830315003392,
|
| 777 |
+
"n": 3,
|
| 778 |
+
"sample_sd": 0.013623163834245318
|
| 779 |
+
},
|
| 780 |
+
"parameters": 420332480,
|
| 781 |
+
"peak_cuda_allocated_bytes": {
|
| 782 |
+
"mean": 4257894400.0,
|
| 783 |
+
"n": 3,
|
| 784 |
+
"sample_sd": 3632373.815194686
|
| 785 |
+
},
|
| 786 |
+
"subset_exact_match": {
|
| 787 |
+
"mean": 0.006060606060606061,
|
| 788 |
+
"n": 3,
|
| 789 |
+
"sample_sd": 0.01049727762162956
|
| 790 |
+
},
|
| 791 |
+
"test_articles": 55,
|
| 792 |
+
"test_seconds_per_article": {
|
| 793 |
+
"mean": 0.206657676709195,
|
| 794 |
+
"n": 3,
|
| 795 |
+
"sample_sd": 0.002071265185964238
|
| 796 |
+
},
|
| 797 |
+
"total_wall_seconds": {
|
| 798 |
+
"mean": 4291.582853937677,
|
| 799 |
+
"n": 3,
|
| 800 |
+
"sample_sd": 44.49609583978631
|
| 801 |
+
}
|
| 802 |
+
},
|
| 803 |
+
{
|
| 804 |
+
"arm": "qwen35-standalone4l-commonhead",
|
| 805 |
+
"label": "4L",
|
| 806 |
+
"layers": 4,
|
| 807 |
+
"lineage_note": "same five-epoch common-head protocol",
|
| 808 |
+
"macro_f1": {
|
| 809 |
+
"mean": 0.5875679559555907,
|
| 810 |
+
"n": 3,
|
| 811 |
+
"sample_sd": 0.007165753447433348
|
| 812 |
+
},
|
| 813 |
+
"micro_f1": {
|
| 814 |
+
"mean": 0.647978512363484,
|
| 815 |
+
"n": 3,
|
| 816 |
+
"sample_sd": 0.0018739735696546695
|
| 817 |
+
},
|
| 818 |
+
"parameters": 210903360,
|
| 819 |
+
"peak_cuda_allocated_bytes": {
|
| 820 |
+
"mean": 2143655936.0,
|
| 821 |
+
"n": 3,
|
| 822 |
+
"sample_sd": 0.0
|
| 823 |
+
},
|
| 824 |
+
"subset_exact_match": {
|
| 825 |
+
"mean": 0.006060606060606061,
|
| 826 |
+
"n": 3,
|
| 827 |
+
"sample_sd": 0.01049727762162956
|
| 828 |
+
},
|
| 829 |
+
"test_articles": 55,
|
| 830 |
+
"test_seconds_per_article": {
|
| 831 |
+
"mean": 0.08543555729062946,
|
| 832 |
+
"n": 3,
|
| 833 |
+
"sample_sd": 0.0005980667826383295
|
| 834 |
+
},
|
| 835 |
+
"total_wall_seconds": {
|
| 836 |
+
"mean": 1643.6388439506602,
|
| 837 |
+
"n": 3,
|
| 838 |
+
"sample_sd": 9.226529522850198
|
| 839 |
+
}
|
| 840 |
+
}
|
| 841 |
+
],
|
| 842 |
+
"schema": "qwen35-4l-cross-task-base-card-figures-v1",
|
| 843 |
+
"status": "complete",
|
| 844 |
+
"task_free_compression_ladder": [
|
| 845 |
+
{
|
| 846 |
+
"layers": 24,
|
| 847 |
+
"name": "24L upstream text teacher",
|
| 848 |
+
"parameters": 752393024,
|
| 849 |
+
"weight_bytes": 1504825632,
|
| 850 |
+
"weight_mib": 1435.1135559082031
|
| 851 |
+
},
|
| 852 |
+
{
|
| 853 |
+
"layers": 8,
|
| 854 |
+
"name": "8L task-free KD",
|
| 855 |
+
"parameters": 420318144,
|
| 856 |
+
"weight_bytes": 840647880,
|
| 857 |
+
"weight_mib": 801.7042922973633
|
| 858 |
+
},
|
| 859 |
+
{
|
| 860 |
+
"layers": 6,
|
| 861 |
+
"name": "6L task-free KD",
|
| 862 |
+
"parameters": 377207424,
|
| 863 |
+
"weight_bytes": 754423424,
|
| 864 |
+
"weight_mib": 719.4742431640625
|
| 865 |
+
},
|
| 866 |
+
{
|
| 867 |
+
"layers": 4,
|
| 868 |
+
"name": "4L task-free KD root",
|
| 869 |
+
"parameters": 334096704,
|
| 870 |
+
"weight_bytes": 668198976,
|
| 871 |
+
"weight_mib": 637.2442016601562
|
| 872 |
+
}
|
| 873 |
+
],
|
| 874 |
+
"uno_q_controlled_depth": {
|
| 875 |
+
"boundary": "Matched V128k/F16 runtime comparison. The 4L arm is the historical early-path proxy, not the final common-head quality model.",
|
| 876 |
+
"claim_boundaries": [
|
| 877 |
+
"Only layer_effect_primary is used for same-vocabulary F16 depth-effect estimates.",
|
| 878 |
+
"Only vocabulary_effect_primary is used for exact-remap vocabulary-effect estimates.",
|
| 879 |
+
"Operational and cross-generation groups are descriptive because precision, heads, layer maps, training lineages, or architectures differ.",
|
| 880 |
+
"llama-bench embedding-mode latency measures backbone/classifier compute but does not validate evidence-span quality.",
|
| 881 |
+
"CPU occupancy is instantaneous saturation; core-seconds is a compute-duration proxy, not measured energy.",
|
| 882 |
+
"No power meter was used, so energy consumption is not reported.",
|
| 883 |
+
"Temperature is device- and ambient-specific and is descriptive, not a universal hardware constant."
|
| 884 |
+
],
|
| 885 |
+
"rows": [
|
| 886 |
+
{
|
| 887 |
+
"layers": 8,
|
| 888 |
+
"metric": "artifact_mib",
|
| 889 |
+
"metric_label": "Artifact size",
|
| 890 |
+
"model": "8L",
|
| 891 |
+
"normalized_8l_percent": 100.0
|
| 892 |
+
},
|
| 893 |
+
{
|
| 894 |
+
"layers": 6,
|
| 895 |
+
"metric": "artifact_mib",
|
| 896 |
+
"metric_label": "Artifact size",
|
| 897 |
+
"model": "6L",
|
| 898 |
+
"normalized_8l_percent": 85.59176355424236
|
| 899 |
+
},
|
| 900 |
+
{
|
| 901 |
+
"layers": 4,
|
| 902 |
+
"metric": "artifact_mib",
|
| 903 |
+
"metric_label": "Artifact size",
|
| 904 |
+
"model": "4L proxy",
|
| 905 |
+
"normalized_8l_percent": 72.2607217304372
|
| 906 |
+
},
|
| 907 |
+
{
|
| 908 |
+
"layers": 8,
|
| 909 |
+
"metric": "latency_pp128_ms",
|
| 910 |
+
"metric_label": "PP128 latency",
|
| 911 |
+
"model": "8L",
|
| 912 |
+
"normalized_8l_percent": 100.0
|
| 913 |
+
},
|
| 914 |
+
{
|
| 915 |
+
"layers": 6,
|
| 916 |
+
"metric": "latency_pp128_ms",
|
| 917 |
+
"metric_label": "PP128 latency",
|
| 918 |
+
"model": "6L",
|
| 919 |
+
"normalized_8l_percent": 76.36565126547553
|
| 920 |
+
},
|
| 921 |
+
{
|
| 922 |
+
"layers": 4,
|
| 923 |
+
"metric": "latency_pp128_ms",
|
| 924 |
+
"metric_label": "PP128 latency",
|
| 925 |
+
"model": "4L proxy",
|
| 926 |
+
"normalized_8l_percent": 50.16216836129361
|
| 927 |
+
},
|
| 928 |
+
{
|
| 929 |
+
"layers": 8,
|
| 930 |
+
"metric": "peak_rss_mib",
|
| 931 |
+
"metric_label": "Peak RSS",
|
| 932 |
+
"model": "8L",
|
| 933 |
+
"normalized_8l_percent": 100.0
|
| 934 |
+
},
|
| 935 |
+
{
|
| 936 |
+
"layers": 6,
|
| 937 |
+
"metric": "peak_rss_mib",
|
| 938 |
+
"metric_label": "Peak RSS",
|
| 939 |
+
"model": "6L",
|
| 940 |
+
"normalized_8l_percent": 87.50983221571457
|
| 941 |
+
},
|
| 942 |
+
{
|
| 943 |
+
"layers": 4,
|
| 944 |
+
"metric": "peak_rss_mib",
|
| 945 |
+
"metric_label": "Peak RSS",
|
| 946 |
+
"model": "4L proxy",
|
| 947 |
+
"normalized_8l_percent": 78.19200407435702
|
| 948 |
+
},
|
| 949 |
+
{
|
| 950 |
+
"layers": 8,
|
| 951 |
+
"metric": "mean_core_seconds_per_round",
|
| 952 |
+
"metric_label": "Core-seconds",
|
| 953 |
+
"model": "8L",
|
| 954 |
+
"normalized_8l_percent": 100.0
|
| 955 |
+
},
|
| 956 |
+
{
|
| 957 |
+
"layers": 6,
|
| 958 |
+
"metric": "mean_core_seconds_per_round",
|
| 959 |
+
"metric_label": "Core-seconds",
|
| 960 |
+
"model": "6L",
|
| 961 |
+
"normalized_8l_percent": 76.16702179915283
|
| 962 |
+
},
|
| 963 |
+
{
|
| 964 |
+
"layers": 4,
|
| 965 |
+
"metric": "mean_core_seconds_per_round",
|
| 966 |
+
"metric_label": "Core-seconds",
|
| 967 |
+
"model": "4L proxy",
|
| 968 |
+
"normalized_8l_percent": 51.03561371028788
|
| 969 |
+
}
|
| 970 |
+
]
|
| 971 |
+
}
|
| 972 |
+
}
|