Distilled Flan-T5-Large transcript formatter + model card

README.md

@@ -0,0 +1,67 @@
+---
+license: apache-2.0
+language: [en]
+base_model: google/flan-t5-large
+pipeline_tag: text2text-generation
+library_name: transformers
+tags:
+  - speech-to-text
+  - transcript-formatting
+  - asr-post-processing
+  - dictation
+  - knowledge-distillation
+---
+
+# Flan-T5-Large Transcript Formatter (distilled, < 1 GB target)
+
+Raw speech-to-text transcript **in** → clean, formatted transcript **out**
+(punctuation, casing, filler/disfluency removal, ITN, homophones, proper nouns,
+URLs/emails, and layout). The model takes the **raw transcript as its only input**
+(no system prompt) and emits the formatted transcript.
+
+## Distillation
+
+- **Teacher:** GPT-OSS-120B → Phase-1 distilled **GPT-OSS-20B** formatter, which
+  reproduces the L0–L5 + RL curriculum behaviour at ~99.5% adjusted accuracy.
+- **Student:** `google/flan-t5-large` (783M, encoder–decoder).
+- **Method:** off-policy / sequence-level knowledge distillation (prompt
+  distillation). Trained on **(raw → formatted)** pairs using the curriculum
+  **gold** targets (cleaner than propagating the teacher's residual errors).
+
+## Data
+
+Sampled from the layered curriculum (L0–L5 + RL), de-duplicated latest-layer-wins,
+with the per-layer val splits held out and made disjoint from train:
+**18,129 train / 2,713 validation** pairs, 21 formatting categories.
+
+## Training
+
+| | |
+|---|---|
+| Hardware | 1× RTX 5090 (32 GB), bf16 |
+| Optimizer | Adafactor, lr 1e-4, warmup_ratio 0.03 |
+| Batch | 8 × grad-accum 2 = effective 16 |
+| Seq length | max 640 tokens (covers the longest pair, 552 tok; zero truncation) |
+| Regularization | gradient checkpointing; early stopping (patience 3), best-checkpoint by val loss |
+| Result | best **val loss 0.005784 @ step 2250 (~epoch 2.0)**; early-stopped at step 3000 |
+
+![loss curves](loss_curves.png)
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+tok = AutoTokenizer.from_pretrained("Akash-Sakala/flan-t5-large-transcript-formatter")
+model = AutoModelForSeq2SeqLM.from_pretrained("Akash-Sakala/flan-t5-large-transcript-formatter")
+raw = "i went to the the store yesterday it was closed so i couldnt get the milk"
+ids = tok(raw, return_tensors="pt")
+print(tok.decode(model.generate(**ids, max_new_tokens=640, num_beams=1)[0], skip_special_tokens=True))
+```
+
+## Limitations
+
+A 783M student shows a capacity step-down from the 20B teacher on the hardest
+**L4 layout** cases (long email/list blocks); T5 was pretrained mostly at 512
+tokens. For an int8, CPU-only deployment (< 1 GB) convert with CTranslate2.
+
+License Apache-2.0 (matches the `openai/gpt-oss` base and the curriculum dataset).

config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "T5ForConditionalGeneration"
+  ],
+  "classifier_dropout": 0.0,
+  "d_ff": 2816,
+  "d_kv": 64,
+  "d_model": 1024,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "gelu_new",
+  "dropout_rate": 0.1,
+  "dtype": "float32",
+  "eos_token_id": 1,
+  "feed_forward_proj": "gated-gelu",
+  "initializer_factor": 1.0,
+  "is_decoder": false,
+  "is_encoder_decoder": true,
+  "is_gated_act": true,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "t5",
+  "n_positions": 512,
+  "num_decoder_layers": 24,
+  "num_heads": 16,
+  "num_layers": 24,
+  "output_past": true,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "scale_decoder_outputs": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.12.1",
+  "use_cache": false,
+  "vocab_size": 32128
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "decoder_start_token_id": 0,
+  "eos_token_id": [
+    1
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.12.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:75d1892934ce91095cd368e07318b678c776f781b35aeaa895857b61b9464c9d
+size 3132668808

tokenizer_config.json

@@ -0,0 +1,114 @@
+{
+  "backend": "tokenizers",
+  "eos_token": "</s>",
+  "extra_ids": 100,
+  "extra_special_tokens": [
+    "<extra_id_0>",
+    "<extra_id_1>",
+    "<extra_id_2>",
+    "<extra_id_3>",
+    "<extra_id_4>",
+    "<extra_id_5>",
+    "<extra_id_6>",
+    "<extra_id_7>",
+    "<extra_id_8>",
+    "<extra_id_9>",
+    "<extra_id_10>",
+    "<extra_id_11>",
+    "<extra_id_12>",
+    "<extra_id_13>",
+    "<extra_id_14>",
+    "<extra_id_15>",
+    "<extra_id_16>",
+    "<extra_id_17>",
+    "<extra_id_18>",
+    "<extra_id_19>",
+    "<extra_id_20>",
+    "<extra_id_21>",
+    "<extra_id_22>",
+    "<extra_id_23>",
+    "<extra_id_24>",
+    "<extra_id_25>",
+    "<extra_id_26>",
+    "<extra_id_27>",
+    "<extra_id_28>",
+    "<extra_id_29>",
+    "<extra_id_30>",
+    "<extra_id_31>",
+    "<extra_id_32>",
+    "<extra_id_33>",
+    "<extra_id_34>",
+    "<extra_id_35>",
+    "<extra_id_36>",
+    "<extra_id_37>",
+    "<extra_id_38>",
+    "<extra_id_39>",
+    "<extra_id_40>",
+    "<extra_id_41>",
+    "<extra_id_42>",
+    "<extra_id_43>",
+    "<extra_id_44>",
+    "<extra_id_45>",
+    "<extra_id_46>",
+    "<extra_id_47>",
+    "<extra_id_48>",
+    "<extra_id_49>",
+    "<extra_id_50>",
+    "<extra_id_51>",
+    "<extra_id_52>",
+    "<extra_id_53>",
+    "<extra_id_54>",
+    "<extra_id_55>",
+    "<extra_id_56>",
+    "<extra_id_57>",
+    "<extra_id_58>",
+    "<extra_id_59>",
+    "<extra_id_60>",
+    "<extra_id_61>",
+    "<extra_id_62>",
+    "<extra_id_63>",
+    "<extra_id_64>",
+    "<extra_id_65>",
+    "<extra_id_66>",
+    "<extra_id_67>",
+    "<extra_id_68>",
+    "<extra_id_69>",
+    "<extra_id_70>",
+    "<extra_id_71>",
+    "<extra_id_72>",
+    "<extra_id_73>",
+    "<extra_id_74>",
+    "<extra_id_75>",
+    "<extra_id_76>",
+    "<extra_id_77>",
+    "<extra_id_78>",
+    "<extra_id_79>",
+    "<extra_id_80>",
+    "<extra_id_81>",
+    "<extra_id_82>",
+    "<extra_id_83>",
+    "<extra_id_84>",
+    "<extra_id_85>",
+    "<extra_id_86>",
+    "<extra_id_87>",
+    "<extra_id_88>",
+    "<extra_id_89>",
+    "<extra_id_90>",
+    "<extra_id_91>",
+    "<extra_id_92>",
+    "<extra_id_93>",
+    "<extra_id_94>",
+    "<extra_id_95>",
+    "<extra_id_96>",
+    "<extra_id_97>",
+    "<extra_id_98>",
+    "<extra_id_99>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "T5Tokenizer",
+  "unk_token": "<unk>"
+}