Update Colab long knowledge probe

README.md

@@ -168,15 +168,15 @@ Prompt format used by the project-side inference code:
 <|im_start|><|object_ref_start|>YOUR_PROMPT_HERE<|im_end|>
 ```
 
-### Colab T4 Quick Generation
+### Colab T4 Long Knowledge Probe
 
 A ready-to-run Colab notebook is available in the project repo:
 
-https://github.com/LLM-OS-Models/KoHRM-text/blob/main/notebooks/KoHRM_Text_1_4B_Colab_T4_Smoke_Test.ipynb
+https://github.com/LLM-OS-Models/KoHRM-text/blob/main/notebooks/KoHRM_Text_1_4B_Colab_T4_Long_Knowledge_Probe.ipynb
 
-The notebook downloads the latest public files and runs training-aligned probes on a Colab T4. Korean law/wiki/finance probes use Korean prompts. Terminal, tool-call, and coding probes use English prompts because that is closer to the current training mix for those behaviors.
+The notebook downloads the latest public files and runs long-form generation prompts that match the current pretraining data style. It is intended to inspect knowledge signal, Korean fluency, repetition, and runtime correctness after pretraining-stage checkpoints.
 
-This notebook is a pretraining-checkpoint probe, not a final chat/SFT benchmark. Strict JSON-only, command-only, and code-only failures before SFT/LoRA/RL should be interpreted as post-training readiness signals, not as final model quality.
+This is not a final chat/SFT benchmark. It intentionally avoids format-constrained SFT-style tests because the public checkpoint is still a pretraining-stage model and has not been behavior-aligned by SFT/LoRA/RL.
 
 It intentionally avoids `transformers`, `AutoTokenizer`, and `AutoModelForCausalLM`. Instead, it uses:
 
@@ -231,23 +231,51 @@ spec.loader.exec_module(kohrm)
 model, tokenizer, cfg = kohrm.load_kohrm(repo_dir, max_gpu_memory_gib=14.0)
 
 settings = dict(
-    max_seq_len=512,
-    temperature=0.0,
-    top_p=1.0,
-    repetition_penalty=1.20,
-    no_repeat_ngram_size=4,
+    max_seq_len=1536,
+    temperature=0.65,
+    top_p=0.92,
+    repetition_penalty=1.05,
+    no_repeat_ngram_size=0,
     condition="direct",
 )
 
 prompts = {
-    "ko_finance": "환율 변동이 개인 투자에 미치는 영향과 대비 전략을 한국어로 4문장 이내로 설명하세요. 같은 표현을 반복하지 마세요.",
-    "en_terminal": "Return one bash command only. No explanation. Task: find the 10 largest files under the current directory, excluding .git, sorted by size descending.",
+    "finance": "환율 변동이 개인 투자에 미치는 영향과 대비 전략은 무엇인가요?",
+    "kowiki_style": """다음은 한국어 위키백과 문서 원문 일부입니다. 백과사전식 한국어, 고유명사, 날짜, 기술/사회/문화 지식을 그대로 학습하십시오.
+
+[문서명]
+훈민정음
+
+[부분]
+1/1""",
+    "legal_style": """다음은 대한민국 법령/자치법규 원문 일부입니다. 법률 한국어, 조문 구조, 번호 체계, 기관명, 시행일자 표현을 그대로 학습하십시오.
+
+[자료종류]
+law
+
+[문서명]
+형법
+
+[경로]
+kr/형법/법률.md
+
+[부분]
+1/1""",
 }
 
 for name, prompt in prompts.items():
     print("=" * 80)
     print(name)
-    print(kohrm.generate_from_loaded(model, tokenizer, cfg, prompt, max_new_tokens=96, **settings))
+    output = kohrm.generate_from_loaded(
+        model,
+        tokenizer,
+        cfg,
+        prompt,
+        max_new_tokens=384,
+        min_new_tokens=160,
+        **settings,
+    )
+    print(output)
 ```
 
 Expected result:
@@ -257,7 +285,7 @@ Expected result:
 - The helper should load the 1.38B public `model.safetensors` export.
 - On Colab T4, generation runs in fp16 through PyTorch scaled-dot-product attention.
 - First generation can take a few minutes because it downloads and loads the full weight file.
-- This is a rolling pretraining checkpoint. If JSON-only, command-only, or Korean repetition behavior is weak, compare later checkpoints with the same notebook before drawing final conclusions.
+- This is a rolling pretraining checkpoint. Compare later checkpoints with the same long prompts before drawing final conclusions.
 
 Prompt format used by the helper, matching upstream `InferenceCheckpoint.tokenize_prompt()`:
 
@@ -505,15 +533,15 @@ schedule: H2L3 recurrent computation
 <|im_start|><|object_ref_start|>여기에_프롬프트를_넣습니다<|im_end|>
 ```
 
-### Colab T4 빠른 생성
+### Colab T4 긴 지식 생성 확인
 
 바로 실행할 수 있는 Colab 노트북은 project repo에 있습니다.
 
-https://github.com/LLM-OS-Models/KoHRM-text/blob/main/notebooks/KoHRM_Text_1_4B_Colab_T4_Smoke_Test.ipynb
+https://github.com/LLM-OS-Models/KoHRM-text/blob/main/notebooks/KoHRM_Text_1_4B_Colab_T4_Long_Knowledge_Probe.ipynb
 
-이 노트북은 Colab T4에서 최신 공개 파일을 다운로드하고 학습 포맷에 맞춘 probe를 실행합니다. 한국어 법률/wiki/금융은 한국어 prompt로, 터미널/툴콜/코딩은 현재 학습 mix에 더 가까운 영어 prompt로 확인합니다.
+이 노트북은 Colab T4에서 최신 공개 파일을 다운로드하고 현재 사전학습 데이터와 같은 스타��의 긴 생성 prompt를 실행합니다. 목적은 pretraining stage checkpoint의 지식 신호, 한국어 유창성, 반복 여부, 공개 `model.safetensors` runtime 동작을 직접 확인하는 것입니다.
 
-이 노트북은 pretraining checkpoint 확인용이지, 최종 chat/SFT benchmark가 아닙니다. SFT/LoRA/RL 전 단계에서 JSON-only, command-only, code-only가 실패하면 이것은 최종 품질 판정이 아니라 post-training에서 고쳐야 할 readiness signal로 봐야 합니다.
+이 노트북은 최종 chat/SFT benchmark가 아닙니다. 공개 checkpoint는 아직 SFT/LoRA/RL로 행동 정렬을 끝낸 모델이 아니므로, 포맷 준수 중심의 SFT식 과제는 의도적으로 제외했습니다.
 
 일부 Colab 환경에서 `transformers`가 `torchvision::nms` import 오류를 내거나 custom architecture를 못 찾는 문제가 생길 수 있으므로, 이 노트북은 `AutoTokenizer`와 `AutoModelForCausalLM`을 쓰지 않습니다. 대신 아래 경로를 사용합니다.
 
@@ -568,23 +596,51 @@ spec.loader.exec_module(kohrm)
 model, tokenizer, cfg = kohrm.load_kohrm(repo_dir, max_gpu_memory_gib=14.0)
 
 settings = dict(
-    max_seq_len=512,
-    temperature=0.0,
-    top_p=1.0,
-    repetition_penalty=1.20,
-    no_repeat_ngram_size=4,
+    max_seq_len=1536,
+    temperature=0.65,
+    top_p=0.92,
+    repetition_penalty=1.05,
+    no_repeat_ngram_size=0,
     condition="direct",
 )
 
 prompts = {
-    "ko_finance": "환율 변동이 개인 투자에 미치는 영향과 대비 전략을 한국어로 4문장 이내로 설명하세요. 같은 표현을 반복하지 마세요.",
-    "en_terminal": "Return one bash command only. No explanation. Task: find the 10 largest files under the current directory, excluding .git, sorted by size descending.",
+    "finance": "환율 변동이 개인 투자에 미치는 영향과 대비 전략은 무엇인가요?",
+    "kowiki_style": """다음은 한국어 위키백과 문서 원문 일부입니다. 백과사전식 한국어, 고유명사, 날짜, 기술/사회/문화 지식을 그대로 학습하십시오.
+
+[문서명]
+훈민정음
+
+[부분]
+1/1""",
+    "legal_style": """다음은 대한민국 법령/자치법규 원문 일부입니다. 법률 한국어, 조문 구조, 번호 체계, 기관명, 시행일자 표현을 그대로 학습하십시오.
+
+[자료종류]
+law
+
+[문서명]
+형법
+
+[경로]
+kr/형법/법률.md
+
+[부분]
+1/1""",
 }
 
 for name, prompt in prompts.items():
     print("=" * 80)
     print(name)
-    print(kohrm.generate_from_loaded(model, tokenizer, cfg, prompt, max_new_tokens=96, **settings))
+    output = kohrm.generate_from_loaded(
+        model,
+        tokenizer,
+        cfg,
+        prompt,
+        max_new_tokens=384,
+        min_new_tokens=160,
+        **settings,
+    )
+    print(output)
 ```
 
 정상 결과:
@@ -594,7 +650,7 @@ for name, prompt in prompts.items():
 - helper가 1.38B 공개 `model.safetensors` 변환본을 로드합니다.
 - Colab T4에서는 fp16 PyTorch scaled-dot-product attention으로 생성합니다.
 - 첫 실행은 2.8 GiB급 weight 다운로드와 로드 때문에 몇 분 걸릴 수 있습니다.
-- 현재 repo는 rolling pretraining checkpoint입니다. JSON-only, command-only, 한국어 반복 억제가 약하게 보이면 같은 노트북으로 이후 checkpoint와 비교해야 합니다.
+- 현재 repo는 rolling pretraining checkpoint입니다. 같은 긴 prompt로 이후 checkpoint와 비교해서 지식, 문체, 반복 여부를 봐야 합니다.
 
 helper가 쓰는 prompt 형식은 upstream `InferenceCheckpoint.tokenize_prompt()`와 맞춥니다.
 

kohrm_colab_generate.py

@@ -2,8 +2,8 @@
 
 This file intentionally avoids `transformers` and FlashAttention. It loads the
 public `model.safetensors` export and runs HRM-Text generation with PyTorch
-scaled-dot-product attention. It is built for smoke generation tests on Colab
-T4 and small CUDA machines.
+scaled-dot-product attention. It is built for long pretraining-checkpoint
+knowledge probes on Colab T4 and small CUDA machines.
 """
 
 from __future__ import annotations
@@ -298,11 +298,14 @@ def _sample_next(
     seen_ids: list[int] | None = None,
     repetition_penalty: float = 1.0,
     no_repeat_ngram_size: int = 0,
+    blocked_ids: set[int] | None = None,
 ) -> int:
     logits = logits.float()
     seen_ids = seen_ids or []
     logits = _apply_repetition_penalty(logits, seen_ids, repetition_penalty)
     logits = _apply_no_repeat_ngram(logits, seen_ids, no_repeat_ngram_size)
+    if blocked_ids:
+        logits[..., list(blocked_ids)] = -torch.inf
     if temperature <= 0:
         return int(torch.argmax(logits, dim=-1).item())
     probs = torch.softmax(logits / temperature, dim=-1)
@@ -325,6 +328,7 @@ def generate_from_loaded(
     prompt: str,
     *,
     max_new_tokens: int = 64,
+    min_new_tokens: int = 0,
     max_seq_len: int = 512,
     temperature: float = 0.0,
     top_p: float = 0.9,
@@ -355,9 +359,17 @@ def generate_from_loaded(
     stop_ids = {int(x) for x in stop_ids if x is not None}
     out_ids: list[int] = []
     seen_ids = list(input_ids)
-    next_id = _sample_next(logits, temperature, top_p, seen_ids, repetition_penalty, no_repeat_ngram_size)
+    next_id = _sample_next(
+        logits,
+        temperature,
+        top_p,
+        seen_ids,
+        repetition_penalty,
+        no_repeat_ngram_size,
+        blocked_ids=stop_ids if min_new_tokens > 0 else None,
+    )
     for _ in range(max_new_tokens):
-        if next_id in stop_ids:
+        if next_id in stop_ids and len(out_ids) >= min_new_tokens:
             break
         out_ids.append(next_id)
         seen_ids.append(next_id)
@@ -365,7 +377,15 @@ def generate_from_loaded(
         pos = torch.tensor([[cache_pos]], device=dev, dtype=torch.long)
         logits = model(token, pos, caches=caches, cache_pos=cache_pos)[:, -1, :]
         cache_pos += 1
-        next_id = _sample_next(logits, temperature, top_p, seen_ids, repetition_penalty, no_repeat_ngram_size)
+        next_id = _sample_next(
+            logits,
+            temperature,
+            top_p,
+            seen_ids,
+            repetition_penalty,
+            no_repeat_ngram_size,
+            blocked_ids=stop_ids if len(out_ids) < min_new_tokens else None,
+        )
 
     return tokenizer.decode(out_ids, skip_special_tokens=True).strip()
 
@@ -376,6 +396,7 @@ def generate_text(
     prompt: str,
     *,
     max_new_tokens: int = 64,
+    min_new_tokens: int = 0,
     max_seq_len: int = 512,
     temperature: float = 0.0,
     top_p: float = 0.9,
@@ -392,6 +413,7 @@ def generate_text(
         cfg,
         prompt,
         max_new_tokens=max_new_tokens,
+        min_new_tokens=min_new_tokens,
         max_seq_len=max_seq_len,
         temperature=temperature,
         top_p=top_p,
@@ -403,21 +425,23 @@ def generate_text(
 
 
 def main() -> None:
-    parser = argparse.ArgumentParser(description="Run a small KoHRM-Text generation test without transformers.")
+    parser = argparse.ArgumentParser(description="Run a KoHRM-Text long generation probe without transformers.")
     parser.add_argument("repo_dir", type=Path, help="Directory containing config.json, tokenizer.json, and model.safetensors")
     parser.add_argument(
         "--prompt",
         default=(
-            "Return one bash command only. Task: find the 10 largest files under "
-            "the current directory, excluding .git, sorted by size descending."
+            "다음은 한국어 위키백과 문서 원문 일부입니다. 백과사전식 한국어, "
+            "고유명사, 날짜, 기술/사회/문화 지식을 그대로 학습하십시오.\n\n"
+            "[문서명]\n훈민정음\n\n[부분]\n1/1"
         ),
     )
-    parser.add_argument("--max-new-tokens", type=int, default=64)
-    parser.add_argument("--max-seq-len", type=int, default=512)
-    parser.add_argument("--temperature", type=float, default=0.0)
-    parser.add_argument("--top-p", type=float, default=0.9)
-    parser.add_argument("--repetition-penalty", type=float, default=1.18)
-    parser.add_argument("--no-repeat-ngram-size", type=int, default=4)
+    parser.add_argument("--max-new-tokens", type=int, default=384)
+    parser.add_argument("--min-new-tokens", type=int, default=160)
+    parser.add_argument("--max-seq-len", type=int, default=1536)
+    parser.add_argument("--temperature", type=float, default=0.65)
+    parser.add_argument("--top-p", type=float, default=0.92)
+    parser.add_argument("--repetition-penalty", type=float, default=1.05)
+    parser.add_argument("--no-repeat-ngram-size", type=int, default=0)
     parser.add_argument(
         "--condition",
         default="direct",
@@ -434,6 +458,7 @@ def main() -> None:
         args.repo_dir,
         args.prompt,
         max_new_tokens=args.max_new_tokens,
+        min_new_tokens=args.min_new_tokens,
         max_seq_len=args.max_seq_len,
         temperature=args.temperature,
         top_p=args.top_p,

notebooks/KoHRM_Text_1_4B_Colab_T4_Long_Knowledge_Probe.ipynb

@@ -0,0 +1,480 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# KoHRM-Text-1.4B Colab T4 Long Knowledge Probe\n",
+    "\n",
+    "This notebook checks a pretraining checkpoint's long-form knowledge signal after PT-style training by generating long responses from training-format instructions.\n",
+    "\n",
+    "It follows the same training data layout used by `scripts/prepare_sft_data.py`:\n",
+    "\n",
+    "```text\n",
+    "instruction text -> response text\n",
+    "```\n",
+    "\n",
+    "The runtime wraps each instruction exactly in the HRM-Text control format:\n",
+    "\n",
+    "```text\n",
+    "<|im_start|><|object_ref_start|>instruction<|im_end|>\n",
+    "```\n",
+    "\n",
+    "`<|object_ref_start|>` is the `direct` condition. The model then generates the response until `<|box_end|>` or the token budget. The prompts below are intentionally close to the actual Korean legal raw corpus, Korean wiki raw corpus, BCAI finance QA, and terminal conversation builders used in pretraining.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Install Dependencies\n",
+    "\n",
+    "This path intentionally avoids `transformers`, `AutoTokenizer`, and `AutoModelForCausalLM`. The current public export is a custom HRM-Text architecture, so the notebook uses the lightweight project helper instead.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip -q install -U huggingface_hub hf_transfer safetensors\n",
+    "!pip -q install --force-reinstall -q \"tokenizers>=0.22.0,<0.23.1\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Runtime Settings\n",
+    "\n",
+    "`MAX_SEQ_LEN=1536` and long generation are intended for T4 knowledge probing. If Colab runs out of memory, lower `MAX_SEQ_LEN` to `1024` and `DEFAULT_MAX_NEW_TOKENS` to `256`.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import json\n",
+    "import gc\n",
+    "import importlib.util\n",
+    "import subprocess\n",
+    "import sys\n",
+    "from pathlib import Path\n",
+    "\n",
+    "os.environ[\"HF_HUB_ENABLE_HF_TRANSFER\"] = \"1\"\n",
+    "\n",
+    "REPO_ID = \"LLM-OS-Models/KoHRM-Text-1.4B\"\n",
+    "REVISION = \"main\"\n",
+    "LOCAL_DIR = Path(\"/content/KoHRM-Text-1.4B\")\n",
+    "HELPER_PATH = LOCAL_DIR / \"kohrm_colab_generate.py\"\n",
+    "\n",
+    "MAX_SEQ_LEN = 1536\n",
+    "DEFAULT_MAX_NEW_TOKENS = 384\n",
+    "DEFAULT_MIN_NEW_TOKENS = 160\n",
+    "\n",
+    "LONG_TEXT_SETTINGS = {\n",
+    "    \"max_seq_len\": MAX_SEQ_LEN,\n",
+    "    \"temperature\": 0.65,\n",
+    "    \"top_p\": 0.92,\n",
+    "    \"repetition_penalty\": 1.05,\n",
+    "    \"no_repeat_ngram_size\": 0,\n",
+    "    \"condition\": \"direct\",\n",
+    "}\n",
+    "\n",
+    "print(\"repo:\", REPO_ID)\n",
+    "print(\"revision:\", REVISION)\n",
+    "print(\"local_dir:\", LOCAL_DIR)\n",
+    "print(\"max_seq_len:\", MAX_SEQ_LEN)\n",
+    "print(\"default max/min new tokens:\", DEFAULT_MAX_NEW_TOKENS, DEFAULT_MIN_NEW_TOKENS)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Download Latest Public Checkpoint\n",
+    "\n",
+    "The notebook downloads only the files needed for public `model.safetensors` inference and the helper runtime. If the helper is missing from the model repo, it falls back to the GitHub repository.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from huggingface_hub import snapshot_download\n",
+    "\n",
+    "LOCAL_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "patterns = [\n",
+    "    \"README.md\",\n",
+    "    \"config.json\",\n",
+    "    \"tokenizer.json\",\n",
+    "    \"tokenizer_config.json\",\n",
+    "    \"special_tokens_map.json\",\n",
+    "    \"model.safetensors\",\n",
+    "    \"kohrm_colab_generate.py\",\n",
+    "    \"notebooks/kohrm_colab_generate.py\",\n",
+    "]\n",
+    "\n",
+    "snapshot_download(\n",
+    "    repo_id=REPO_ID,\n",
+    "    repo_type=\"model\",\n",
+    "    revision=REVISION,\n",
+    "    local_dir=str(LOCAL_DIR),\n",
+    "    local_dir_use_symlinks=False,\n",
+    "    allow_patterns=patterns,\n",
+    ")\n",
+    "\n",
+    "nested_helper = LOCAL_DIR / \"notebooks\" / \"kohrm_colab_generate.py\"\n",
+    "if not HELPER_PATH.exists() and nested_helper.exists():\n",
+    "    HELPER_PATH.write_text(nested_helper.read_text(encoding=\"utf-8\"), encoding=\"utf-8\")\n",
+    "\n",
+    "if not HELPER_PATH.exists():\n",
+    "    repo = Path(\"/content/KoHRM-text\")\n",
+    "    if not repo.exists():\n",
+    "        subprocess.run(\n",
+    "            [\"git\", \"clone\", \"--depth\", \"1\", \"https://github.com/LLM-OS-Models/KoHRM-text\", str(repo)],\n",
+    "            check=True,\n",
+    "        )\n",
+    "    HELPER_PATH.write_text((repo / \"notebooks\" / \"kohrm_colab_generate.py\").read_text(encoding=\"utf-8\"), encoding=\"utf-8\")\n",
+    "\n",
+    "for name in [\"config.json\", \"tokenizer.json\", \"model.safetensors\", \"README.md\", \"kohrm_colab_generate.py\"]:\n",
+    "    p = LOCAL_DIR / name\n",
+    "    print(name, \"OK\" if p.exists() else \"MISSING\", p)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Inspect Config and Training Wrapper\n",
+    "\n",
+    "This cell checks the model shape and special tokens. The prompt wrapper shown here is the exact wrapper used for all probes below.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "spec = importlib.util.spec_from_file_location(\"kohrm_colab_generate\", HELPER_PATH)\n",
+    "kohrm = importlib.util.module_from_spec(spec)\n",
+    "sys.modules[\"kohrm_colab_generate\"] = kohrm\n",
+    "spec.loader.exec_module(kohrm)\n",
+    "\n",
+    "config = json.loads((LOCAL_DIR / \"config.json\").read_text(encoding=\"utf-8\"))\n",
+    "print(json.dumps({\n",
+    "    \"model_type\": config.get(\"model_type\"),\n",
+    "    \"architectures\": config.get(\"architectures\"),\n",
+    "    \"vocab_size\": config.get(\"vocab_size\"),\n",
+    "    \"hidden_size\": config.get(\"hidden_size\"),\n",
+    "    \"num_hidden_layers\": config.get(\"num_hidden_layers\"),\n",
+    "    \"num_attention_heads\": config.get(\"num_attention_heads\"),\n",
+    "    \"H_cycles\": config.get(\"H_cycles\"),\n",
+    "    \"L_cycles\": config.get(\"L_cycles\"),\n",
+    "    \"max_position_embeddings\": config.get(\"max_position_embeddings\"),\n",
+    "    \"prefix_lm\": config.get(\"prefix_lm\"),\n",
+    "}, indent=2, ensure_ascii=False))\n",
+    "\n",
+    "from tokenizers import Tokenizer\n",
+    "raw_tok = Tokenizer.from_file(str(LOCAL_DIR / \"tokenizer.json\"))\n",
+    "for token in [\"<|im_start|>\", \"<|object_ref_start|>\", \"<|object_ref_end|>\", \"<|quad_start|>\", \"<|quad_end|>\", \"<|im_end|>\", \"<|box_end|>\"]:\n",
+    "    print(f\"{token:22s}\", raw_tok.token_to_id(token))\n",
+    "\n",
+    "example_instruction = \"비씨카드는 어떤 회사인가요?\"\n",
+    "print(\"direct condition token:\", kohrm.condition_to_tokens(\"direct\"))\n",
+    "print(\"wrapped prompt:\", kohrm.format_kohrm_prompt(example_instruction, condition=\"direct\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Load Model Once\n",
+    "\n",
+    "Loading can take a few minutes on Colab. The helper uses PyTorch scaled-dot-product attention and a static KV cache. It is slower than the training-time FlashAttention path, but it is enough for long text inspection.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "\n",
+    "gc.collect()\n",
+    "if torch.cuda.is_available():\n",
+    "    torch.cuda.empty_cache()\n",
+    "    print(torch.cuda.get_device_name(0))\n",
+    "\n",
+    "model, tokenizer, cfg = kohrm.load_kohrm(LOCAL_DIR, max_gpu_memory_gib=14.0)\n",
+    "print(\"loaded dtype:\", next(model.parameters()).dtype)\n",
+    "print(\"loaded device:\", next(model.parameters()).device)\n",
+    "if torch.cuda.is_available():\n",
+    "    free, total = torch.cuda.mem_get_info()\n",
+    "    print(f\"GPU memory free/total GiB after load: {free / 2**30:.2f}/{total / 2**30:.2f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. PT-Format Long Knowledge Prompts\n",
+    "\n",
+    "These long-response probes use prompts shaped like the data that went into pretraining:\n",
+    "\n",
+    "- BCAI Finance Kor: plain Korean finance QA instruction.\n",
+    "- Korean wiki raw corpus: `다음은 한국어 위키백과 문서 원문 일부입니다...` instruction.\n",
+    "- Korean legal raw corpus: `다음은 대한민국 법령/자치법규 원문 일부입니다...` instruction.\n",
+    "- Terminal conversation corpus: `다음 터미널/코딩 작업 대화 맥락에서...` instruction.\n",
+    "\n",
+    "The notebook prints output length and the raw generated text. Judge the text manually: domain terms, Korean fluency, factual continuity, repetition, and whether it collapses to one-token answers.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "KNOWLEDGE_PROMPTS = [\n",
+    "    {\n",
+    "        \"name\": \"finance_bc_card_company\",\n",
+    "        \"source_format\": \"BCAI Finance Kor plain QA\",\n",
+    "        \"prompt\": \"비씨카드는 어떤 회사인가요?\",\n",
+    "        \"max_new_tokens\": 420,\n",
+    "        \"min_new_tokens\": 180,\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"finance_exchange_investment\",\n",
+    "        \"source_format\": \"BCAI Finance Kor plain QA\",\n",
+    "        \"prompt\": \"환율 변동이 개인 투���에 미치는 영향과 대비 전략은 무엇인가요?\",\n",
+    "        \"max_new_tokens\": 420,\n",
+    "        \"min_new_tokens\": 180,\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"kowiki_hunminjeongeum_raw_style\",\n",
+    "        \"source_format\": \"kowiki raw instruction style\",\n",
+    "        \"prompt\": \"\"\"다음은 한국어 위키백과 문서 원문 일부입니다. 백과사전식 한국어, 고유명사, 날짜, 기술/사회/문화 지식을 그대로 학습하십시오.\n",
+    "\n",
+    "[문서명]\n",
+    "훈민정음\n",
+    "\n",
+    "[부분]\n",
+    "1/1\"\"\",\n",
+    "        \"max_new_tokens\": 520,\n",
+    "        \"min_new_tokens\": 220,\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"kowiki_jimmy_carter_raw_style\",\n",
+    "        \"source_format\": \"kowiki raw instruction style\",\n",
+    "        \"prompt\": \"\"\"다음은 한국어 위키백과 문서 원문 일부입니다. 백과사전식 한국어, 고유명사, 날짜, 기술/사회/문화 지식을 그대로 학습하십시오.\n",
+    "\n",
+    "[문서명]\n",
+    "지미 카터\n",
+    "\n",
+    "[부분]\n",
+    "1/3\"\"\",\n",
+    "        \"max_new_tokens\": 520,\n",
+    "        \"min_new_tokens\": 220,\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"legal_criminal_code_raw_style\",\n",
+    "        \"source_format\": \"korean legal raw instruction style\",\n",
+    "        \"prompt\": \"\"\"다음은 대한민국 법령/자치법규 원문 일부입니다. 법률 한국어, 조문 구조, 번호 체계, 기관명, 시행일자 표현을 그대로 학습하십시오.\n",
+    "\n",
+    "[자료종류]\n",
+    "law\n",
+    "\n",
+    "[문서명]\n",
+    "형법\n",
+    "\n",
+    "[경로]\n",
+    "kr/형법/법률.md\n",
+    "\n",
+    "[부분]\n",
+    "1/1\"\"\",\n",
+    "        \"max_new_tokens\": 520,\n",
+    "        \"min_new_tokens\": 220,\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"legal_restoration_raw_style\",\n",
+    "        \"source_format\": \"korean legal raw instruction style\",\n",
+    "        \"prompt\": \"\"\"다음은 대한민국 법령/자치법규 원문 일부입니다. 법률 한국어, 조문 구조, 번호 체계, 기관명, 시행일자 표현을 그대로 학습하십시오.\n",
+    "\n",
+    "[자료종류]\n",
+    "law\n",
+    "\n",
+    "[문서명]\n",
+    "10ㆍ27법난 피해자의 명예회복 등에 관한 법률\n",
+    "\n",
+    "[경로]\n",
+    "kr/10ㆍ27법난피해자의명예회복등에관한법률/법률.md\n",
+    "\n",
+    "[부분]\n",
+    "1/1\"\"\",\n",
+    "        \"max_new_tokens\": 520,\n",
+    "        \"min_new_tokens\": 220,\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"terminal_conversation_style_long\",\n",
+    "        \"source_format\": \"local terminal conversation instruction style\",\n",
+    "        \"prompt\": \"\"\"다음 터미널/코딩 작업 대화 맥락에서 assistant가 이어서 수행할 분석, 계획, 명령 JSON 또는 최종 응답을 작성하십시오.\n",
+    "\n",
+    "[user]\n",
+    "현재 디렉터리에서 용량이 큰 파일을 찾아 디스크 정리 후보를 보고 싶습니다. 숨김 폴더와 일반 폴더를 모두 확인하되, 결과는 사람이 읽기 쉽게 정리해 주세요.\"\"\",\n",
+    "        \"max_new_tokens\": 420,\n",
+    "        \"min_new_tokens\": 160,\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "print(\"probe count:\", len(KNOWLEDGE_PROMPTS))\n",
+    "for item in KNOWLEDGE_PROMPTS:\n",
+    "    print(\"-\", item[\"name\"], \"|\", item[\"source_format\"], \"| max/min\", item[\"max_new_tokens\"], item[\"min_new_tokens\"])\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Run Long Generations\n",
+    "\n",
+    "The goal is to inspect the generated text itself after PT-style training: length, Korean fluency, domain terms, factual continuity, and repetition.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def count_tokens(text):\n",
+    "    return len(tokenizer.encode(text, add_special_tokens=False).ids)\n",
+    "\n",
+    "RUN_ONLY = None  # Example: {\"finance_bc_card_company\", \"kowiki_hunminjeongeum_raw_style\"}\n",
+    "\n",
+    "results = []\n",
+    "for case in KNOWLEDGE_PROMPTS:\n",
+    "    if RUN_ONLY is not None and case[\"name\"] not in RUN_ONLY:\n",
+    "        continue\n",
+    "\n",
+    "    prompt = case[\"prompt\"]\n",
+    "    max_new = case.get(\"max_new_tokens\", DEFAULT_MAX_NEW_TOKENS)\n",
+    "    min_new = case.get(\"min_new_tokens\", DEFAULT_MIN_NEW_TOKENS)\n",
+    "\n",
+    "    print(\"=\" * 100)\n",
+    "    print(\"case:\", case[\"name\"])\n",
+    "    print(\"source_format:\", case[\"source_format\"])\n",
+    "    print(\"prompt_chars:\", len(prompt), \"prompt_tokens:\", count_tokens(kohrm.format_kohrm_prompt(prompt, condition=\"direct\")))\n",
+    "    print(\"max_new_tokens:\", max_new, \"min_new_tokens:\", min_new)\n",
+    "    print(\"--- prompt ---\")\n",
+    "    print(prompt)\n",
+    "    print(\"--- output ---\")\n",
+    "\n",
+    "    output = kohrm.generate_from_loaded(\n",
+    "        model,\n",
+    "        tokenizer,\n",
+    "        cfg,\n",
+    "        prompt,\n",
+    "        max_new_tokens=max_new,\n",
+    "        min_new_tokens=min_new,\n",
+    "        **LONG_TEXT_SETTINGS,\n",
+    "    )\n",
+    "    out_tokens = count_tokens(output)\n",
+    "    results.append({\n",
+    "        \"case\": case[\"name\"],\n",
+    "        \"source_format\": case[\"source_format\"],\n",
+    "        \"output_chars\": len(output),\n",
+    "        \"output_tokens\": out_tokens,\n",
+    "        \"output\": output,\n",
+    "    })\n",
+    "    print(output)\n",
+    "    print(\"--- output stats ---\")\n",
+    "    print(\"chars:\", len(output), \"tokens:\", out_tokens)\n",
+    "\n",
+    "print(\"=\" * 100)\n",
+    "print(json.dumps([{k: r[k] for k in [\"case\", \"source_format\", \"output_chars\", \"output_tokens\"]} for r in results], indent=2, ensure_ascii=False))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 8. Optional Decode Settings Sweep\n",
+    "\n",
+    "Use this only if long output collapses to short tokens or repeats. It keeps the same training-format prompts and changes only decoding parameters.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "SWEEP_CASE_NAME = \"finance_exchange_investment\"\n",
+    "SWEEP_PROMPT = next(item[\"prompt\"] for item in KNOWLEDGE_PROMPTS if item[\"name\"] == SWEEP_CASE_NAME)\n",
+    "\n",
+    "SWEEP_SETTINGS = [\n",
+    "    {\"name\": \"deterministic_minlen\", \"temperature\": 0.0, \"top_p\": 1.0, \"repetition_penalty\": 1.02, \"no_repeat_ngram_size\": 0},\n",
+    "    {\"name\": \"sample_balanced\", \"temperature\": 0.65, \"top_p\": 0.92, \"repetition_penalty\": 1.05, \"no_repeat_ngram_size\": 0},\n",
+    "    {\"name\": \"sample_more_diverse\", \"temperature\": 0.85, \"top_p\": 0.95, \"repetition_penalty\": 1.08, \"no_repeat_ngram_size\": 0},\n",
+    "]\n",
+    "\n",
+    "for settings in SWEEP_SETTINGS:\n",
+    "    run_settings = dict(LONG_TEXT_SETTINGS)\n",
+    "    run_settings.update({k: v for k, v in settings.items() if k != \"name\"})\n",
+    "    print(\"=\" * 100)\n",
+    "    print(\"decode:\", settings[\"name\"])\n",
+    "    output = kohrm.generate_from_loaded(\n",
+    "        model,\n",
+    "        tokenizer,\n",
+    "        cfg,\n",
+    "        SWEEP_PROMPT,\n",
+    "        max_new_tokens=320,\n",
+    "        min_new_tokens=120,\n",
+    "        **run_settings,\n",
+    "    )\n",
+    "    print(output)\n",
+    "    print(\"chars:\", len(output), \"tokens:\", count_tokens(output))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 9. How To Read Results\n",
+    "\n",
+    "For this PT knowledge probe, useful signs are:\n",
+    "\n",
+    "- The output is long enough to inspect, not `Yes`, `B`, `0`, or one short fragment.\n",
+    "- Korean legal/wiki/finance outputs contain domain terms and coherent Korean paragraphs.\n",
+    "- Raw-style prompts may produce article/law-like continuations because that is how raw corpora were converted for PT.\n",
+    "- Terminal conversation style may produce analysis plus commands, because that is what the local terminal conversation data used as the response target.\n",
+    "\n",
+    "If every long prompt still collapses to one-token or unrelated boilerplate despite `min_new_tokens`, the next thing to debug is the public converted weight/runtime compatibility, not prompt wording.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

notebooks/KoHRM_Text_1_4B_Colab_T4_Smoke_Test.ipynb

@@ -4,24 +4,23 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# KoHRM-Text-1.4B Colab T4 Pretraining Checkpoint Probe\n",
+    "# KoHRM-Text-1.4B Colab T4 Long Knowledge Probe\n",
     "\n",
-    "This notebook loads the latest public KoHRM-Text-1.4B checkpoint without `transformers`, so it avoids the Colab `torchvision::nms` / custom `HrmTextConfig` import failure.\n",
+    "This notebook checks a pretraining checkpoint's long-form knowledge signal after PT-style training by generating long responses from training-format instructions.\n",
     "\n",
-    "Important: this is a **rolling pretraining checkpoint probe**, not a final SFT/chat benchmark. KoHRM is currently trained with HRM-Text style single-stage instruction pretraining. It has not yet gone through the later behavior SFT/LoRA/RL pass, so strict JSON-only, command-only, grounded summary, and code-correctness tasks should be interpreted as post-training readiness checks.\n",
-    "\n",
-    "Training format:\n",
+    "It follows the same training data layout used by `scripts/prepare_sft_data.py`:\n",
     "\n",
     "```text\n",
-    "<|im_start|><condition_token>instruction<|im_end|>response<|box_end|>\n",
+    "instruction text -> response text\n",
     "```\n",
     "\n",
-    "The helper follows upstream `InferenceCheckpoint.tokenize_prompt()`: `<boq><condition_tokens><instruction><eoq>`, then stops generation on `<|box_end|>`. Use `direct` / `<|object_ref_start|>` for answer-only completions.\n",
+    "The runtime wraps each instruction exactly in the HRM-Text control format:\n",
     "\n",
-    "The notebook separates:\n",
+    "```text\n",
+    "<|im_start|><|object_ref_start|>instruction<|im_end|>\n",
+    "```\n",
     "\n",
-    "- pretraining-distribution probes: simple continuation/QA tasks that check whether the checkpoint is learning the data distribution.\n",
-    "- strict post-training probes: JSON-only, command-only, and code-only tasks that will usually need SFT/LoRA/RL before they become reliable."
+    "`<|object_ref_start|>` is the `direct` condition. The model then generates the response until `<|box_end|>` or the token budget. The prompts below are intentionally close to the actual Korean legal raw corpus, Korean wiki raw corpus, BCAI finance QA, and terminal conversation builders used in pretraining.\n"
    ]
   },
   {
@@ -30,7 +29,7 @@
    "source": [
     "## 1. Install Dependencies\n",
     "\n",
-    "The runtime intentionally does not import `transformers`. `tokenizers` is pinned below `0.23.1` to avoid conflicts with Colab images that already contain `transformers 5.x`."
+    "This path intentionally avoids `transformers`, `AutoTokenizer`, and `AutoModelForCausalLM`. The current public export is a custom HRM-Text architecture, so the notebook uses the lightweight project helper instead.\n"
    ]
   },
   {
@@ -49,7 +48,7 @@
    "source": [
     "## 2. Runtime Settings\n",
     "\n",
-    "`MAX_SEQ_LEN=512` is the safest T4 default. Raise it to `768` only if the first load leaves enough free VRAM."
+    "`MAX_SEQ_LEN=1536` and long generation are intended for T4 knowledge probing. If Colab runs out of memory, lower `MAX_SEQ_LEN` to `1024` and `DEFAULT_MAX_NEW_TOKENS` to `256`.\n"
    ]
   },
   {
@@ -72,21 +71,25 @@
     "REVISION = \"main\"\n",
     "LOCAL_DIR = Path(\"/content/KoHRM-Text-1.4B\")\n",
     "HELPER_PATH = LOCAL_DIR / \"kohrm_colab_generate.py\"\n",
-    "MAX_SEQ_LEN = 512\n",
     "\n",
-    "STRICT_SETTINGS = {\n",
+    "MAX_SEQ_LEN = 1536\n",
+    "DEFAULT_MAX_NEW_TOKENS = 384\n",
+    "DEFAULT_MIN_NEW_TOKENS = 160\n",
+    "\n",
+    "LONG_TEXT_SETTINGS = {\n",
     "    \"max_seq_len\": MAX_SEQ_LEN,\n",
-    "    \"temperature\": 0.0,\n",
-    "    \"top_p\": 1.0,\n",
-    "    \"repetition_penalty\": 1.20,\n",
-    "    \"no_repeat_ngram_size\": 4,\n",
+    "    \"temperature\": 0.65,\n",
+    "    \"top_p\": 0.92,\n",
+    "    \"repetition_penalty\": 1.05,\n",
+    "    \"no_repeat_ngram_size\": 0,\n",
     "    \"condition\": \"direct\",\n",
     "}\n",
     "\n",
     "print(\"repo:\", REPO_ID)\n",
     "print(\"revision:\", REVISION)\n",
     "print(\"local_dir:\", LOCAL_DIR)\n",
-    "print(\"max_seq_len:\", MAX_SEQ_LEN)"
+    "print(\"max_seq_len:\", MAX_SEQ_LEN)\n",
+    "print(\"default max/min new tokens:\", DEFAULT_MAX_NEW_TOKENS, DEFAULT_MIN_NEW_TOKENS)"
    ]
   },
   {
@@ -95,7 +98,7 @@
    "source": [
     "## 3. Download Latest Public Checkpoint\n",
     "\n",
-    "The public model repo is expected to contain `config.json`, `tokenizer.json`, `model.safetensors`, and the model card. If the lightweight helper is not present in the model repo, the notebook clones the GitHub repo and copies the helper from `notebooks/`."
+    "The notebook downloads only the files needed for public `model.safetensors` inference and the helper runtime. If the helper is missing from the model repo, it falls back to the GitHub repository.\n"
    ]
   },
   {
@@ -141,17 +144,17 @@
     "    HELPER_PATH.write_text((repo / \"notebooks\" / \"kohrm_colab_generate.py\").read_text(encoding=\"utf-8\"), encoding=\"utf-8\")\n",
     "\n",
     "for name in [\"config.json\", \"tokenizer.json\", \"model.safetensors\", \"README.md\", \"kohrm_colab_generate.py\"]:\n",
-    "    path = LOCAL_DIR / name\n",
-    "    print(f\"{name}: {path.exists()} {path.stat().st_size / 2**20:.2f} MiB\" if path.exists() else f\"{name}: missing\")"
+    "    p = LOCAL_DIR / name\n",
+    "    print(name, \"OK\" if p.exists() else \"MISSING\", p)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 4. Inspect Config and Prompt Format\n",
+    "## 4. Inspect Config and Training Wrapper\n",
     "\n",
-    "Do not use `AutoTokenizer` or `AutoModelForCausalLM` here. The current public export uses a custom HRM-Text architecture, and the helper below loads it directly from `safetensors`."
+    "This cell checks the model shape and special tokens. The prompt wrapper shown here is the exact wrapper used for all probes below.\n"
    ]
   },
   {
@@ -162,6 +165,7 @@
    "source": [
     "spec = importlib.util.spec_from_file_location(\"kohrm_colab_generate\", HELPER_PATH)\n",
     "kohrm = importlib.util.module_from_spec(spec)\n",
+    "sys.modules[\"kohrm_colab_generate\"] = kohrm\n",
     "spec.loader.exec_module(kohrm)\n",
     "\n",
     "config = json.loads((LOCAL_DIR / \"config.json\").read_text(encoding=\"utf-8\"))\n",
@@ -183,9 +187,9 @@
     "for token in [\"<|im_start|>\", \"<|object_ref_start|>\", \"<|object_ref_end|>\", \"<|quad_start|>\", \"<|quad_end|>\", \"<|im_end|>\", \"<|box_end|>\"]:\n",
     "    print(f\"{token:22s}\", raw_tok.token_to_id(token))\n",
     "\n",
+    "example_instruction = \"비씨카드는 어떤 회사인가요?\"\n",
     "print(\"direct condition token:\", kohrm.condition_to_tokens(\"direct\"))\n",
-    "example = kohrm.format_kohrm_prompt(\"Return one bash command only.\", condition=\"direct\")\n",
-    "print(\"wrapped prompt:\", example)"
+    "print(\"wrapped prompt:\", kohrm.format_kohrm_prompt(example_instruction, condition=\"direct\"))"
    ]
   },
   {
@@ -194,7 +198,7 @@
    "source": [
     "## 5. Load Model Once\n",
     "\n",
-    "On a T4, loading can take a few minutes. The helper uses PyTorch scaled-dot-product attention and a static KV cache. It is slower than the training-time FlashAttention path, but it is portable enough for Colab smoke tests."
+    "Loading can take a few minutes on Colab. The helper uses PyTorch scaled-dot-product attention and a static KV cache. It is slower than the training-time FlashAttention path, but it is enough for long text inspection.\n"
    ]
   },
   {
@@ -222,11 +226,16 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 6. Probe Cases\n",
+    "## 6. PT-Format Long Knowledge Prompts\n",
     "\n",
-    "The first block is intentionally easier and closer to pretraining distribution. The second block is stricter and should be read as a post-training target, not as a final failure verdict for the current unsupervised/SFT-free checkpoint.\n",
+    "These long-response probes use prompts shaped like the data that went into pretraining:\n",
     "\n",
-    "Prompts here are deliberately plain. Over-complicated meta-prompts are not a good fix for a pretraining checkpoint; behavior such as JSON-only, command-only, and exact code generation belongs in the later SFT/LoRA/RL pass."
+    "- BCAI Finance Kor: plain Korean finance QA instruction.\n",
+    "- Korean wiki raw corpus: `다음은 한국어 위키백과 문서 원문 일부입니다...` instruction.\n",
+    "- Korean legal raw corpus: `다음은 대한민국 법령/자치법규 원문 일부입니다...` instruction.\n",
+    "- Terminal conversation corpus: `다음 터미널/코딩 작업 대화 맥락에서...` instruction.\n",
+    "\n",
+    "The notebook prints output length and the raw generated text. Judge the text manually: domain terms, Korean fluency, factual continuity, repetition, and whether it collapses to one-token answers.\n"
    ]
   },
   {
@@ -235,257 +244,172 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import ast\n",
-    "import json\n",
-    "import re\n",
-    "\n",
-    "PRETRAINING_PROBES = [\n",
+    "KNOWLEDGE_PROMPTS = [\n",
     "    {\n",
-    "        \"name\": \"ko_finance_plain_qa\",\n",
-    "        \"phase\": \"pretraining_probe\",\n",
-    "        \"lang\": \"ko\",\n",
-    "        \"expect\": \"ko_contains_terms\",\n",
-    "        \"required_terms\": [\"환율\", \"투자\"],\n",
-    "        \"max_new_tokens\": 96,\n",
-    "        \"prompt\": \"환율 변동이 개인 투자에 미치는 영향을 간단히 설명하세요.\",\n",
+    "        \"name\": \"finance_bc_card_company\",\n",
+    "        \"source_format\": \"BCAI Finance Kor plain QA\",\n",
+    "        \"prompt\": \"비씨카드는 어떤 회사인가요?\",\n",
+    "        \"max_new_tokens\": 420,\n",
+    "        \"min_new_tokens\": 180,\n",
     "    },\n",
     "    {\n",
-    "        \"name\": \"ko_legal_plain_extraction\",\n",
-    "        \"phase\": \"pretraining_probe\",\n",
-    "        \"lang\": \"ko\",\n",
-    "        \"expect\": \"ko_contains_terms\",\n",
-    "        \"required_terms\": [\"홍보대사\", \"무보수\"],\n",
-    "        \"max_new_tokens\": 96,\n",
-    "        \"prompt\": \"\"\"다음 조문에서 핵심 내용을 한 문장으로 말하세요.\n",
-    "\n",
-    "제5조 (보상)\n",
-    "① 홍보대사는 무보수 명예직으로 한다.\n",
-    "② 군수는 홍보대사가 임무 수행을 위하여 활동하는 경우 예산의 범위 안에서 홍보활동에 직접 소요되는 실 경비로 숙식비, 차량운행 경비, 기타 비용과 격려금품을 지급할 수 있다.\"\"\",\n",
+    "        \"name\": \"finance_exchange_investment\",\n",
+    "        \"source_format\": \"BCAI Finance Kor plain QA\",\n",
+    "        \"prompt\": \"환율 변동이 개인 투자에 미치는 영향과 대비 전략은 무엇인가요?\",\n",
+    "        \"max_new_tokens\": 420,\n",
+    "        \"min_new_tokens\": 180,\n",
     "    },\n",
     "    {\n",
-    "        \"name\": \"ko_wiki_plain_qa\",\n",
-    "        \"phase\": \"pretraining_probe\",\n",
-    "        \"lang\": \"ko\",\n",
-    "        \"expect\": \"ko_contains_terms\",\n",
-    "        \"required_terms\": [\"훈민정음\", \"세종\"],\n",
-    "        \"max_new_tokens\": 96,\n",
-    "        \"prompt\": \"훈민정음은 누가 만들었고 어떤 목적이 있었나요?\",\n",
-    "    },\n",
-    "    {\n",
-    "        \"name\": \"en_terminal_intent_completion\",\n",
-    "        \"phase\": \"pretraining_probe\",\n",
-    "        \"lang\": \"en\",\n",
-    "        \"expect\": \"mentions_shell_concept\",\n",
-    "        \"required_terms\": [\"find\", \"sort\"],\n",
-    "        \"max_new_tokens\": 96,\n",
-    "        \"prompt\": \"In bash, to list the largest files under the current directory, you can use\",\n",
-    "    },\n",
-    "]\n",
-    "\n",
-    "STRICT_POSTTRAINING_PROBES = [\n",
-    "    {\n",
-    "        \"name\": \"ko_legal_json_direct\",\n",
-    "        \"phase\": \"posttraining_strict_probe\",\n",
-    "        \"lang\": \"ko\",\n",
-    "        \"expect\": \"strict_json_keys\",\n",
-    "        \"required_keys\": [\"조문명\", \"적용 대상\", \"핵심 의무\"],\n",
-    "        \"required_terms\": [\"제5조\", \"홍보대사\", \"무보수\", \"실 경비\"],\n",
-    "        \"max_new_tokens\": 128,\n",
-    "        \"prompt\": \"\"\"다음 한국 법령/행정규칙 발췌문에서 조문명, 적용 대상, 핵심 의무를 JSON 객체 하나로만 추출하세요. JSON 밖의 설명은 쓰지 마세요.\n",
+    "        \"name\": \"kowiki_hunminjeongeum_raw_style\",\n",
+    "        \"source_format\": \"kowiki raw instruction style\",\n",
+    "        \"prompt\": \"\"\"다음은 한국어 위키백과 문서 원문 일부입니다. 백과사전식 한국어, 고유명사, 날짜, 기술/사회/문화 지식을 그대로 학습하십시오.\n",
     "\n",
     "[문서명]\n",
-    "진안군 홍보대사 운영 조례\n",
+    "훈민정음\n",
     "\n",
-    "[조문]\n",
-    "제5조 (보상)\n",
-    "① 홍보대사는 무보수 명예직으로 한다.\n",
-    "② 군수는 홍보대사가 임무 수행을 위하여 활동하는 경우 예산의 범위 안에서 홍보활동에 직접 소요되는 실 경비로 숙식비, 차량운행 경비, 기타 비용과 격려금품을 지급할 수 있다.\"\"\",\n",
-    "    },\n",
-    "    {\n",
-    "        \"name\": \"ko_wiki_grounded_summary\",\n",
-    "        \"phase\": \"posttraining_strict_probe\",\n",
-    "        \"lang\": \"ko\",\n",
-    "        \"expect\": \"ko_grounded_summary\",\n",
-    "        \"required_terms\": [\"훈민정음\", \"세종\"],\n",
-    "        \"forbidden_terms\": [\"ganjang\", \"A:\", \"<br>\", \"</br>\"],\n",
-    "        \"max_new_tokens\": 96,\n",
-    "        \"prompt\": \"\"\"다음 글만 근거로 핵심 내용을 한국어로 3문장 이내로 요약하세요. 글에 없는 사실은 추가하지 마세요.\n",
-    "\n",
-    "[글]\n",
-    "훈민정음은 조선 세종이 창제한 문자 체계이다. 창제 목적은 백성이 자신의 뜻을 쉽게 글로 표현하도록 돕는 데 있었다. 자음은 발음 기관의 모양을 본떠 만들었고, 모음은 하늘, 땅, 사람의 원리를 바탕으로 구성되었다.\"\"\",\n",
+    "[부분]\n",
+    "1/1\"\"\",\n",
+    "        \"max_new_tokens\": 520,\n",
+    "        \"min_new_tokens\": 220,\n",
     "    },\n",
     "    {\n",
-    "        \"name\": \"ko_finance_short\",\n",
-    "        \"phase\": \"posttraining_strict_probe\",\n",
-    "        \"lang\": \"ko\",\n",
-    "        \"expect\": \"ko_grounded_summary\",\n",
-    "        \"required_terms\": [\"환율\", \"투자\"],\n",
-    "        \"forbidden_terms\": [\"<br>\", \"</br>\", \"A:\", \"B:\", \"C:\"],\n",
-    "        \"max_new_tokens\": 96,\n",
-    "        \"prompt\": \"환율 변동이 개인 투자에 미치는 영향과 대비 전략을 한국어로 4문장 이내로 설명하세요. 같은 표현을 반복하지 마세��.\",\n",
+    "        \"name\": \"kowiki_jimmy_carter_raw_style\",\n",
+    "        \"source_format\": \"kowiki raw instruction style\",\n",
+    "        \"prompt\": \"\"\"다음은 한국어 위키백과 문서 원문 일부입니다. 백과사전식 한국어, 고유명사, 날짜, 기술/사회/문화 지식을 그대로 학습하십시오.\n",
+    "\n",
+    "[문서명]\n",
+    "지미 카터\n",
+    "\n",
+    "[부분]\n",
+    "1/3\"\"\",\n",
+    "        \"max_new_tokens\": 520,\n",
+    "        \"min_new_tokens\": 220,\n",
     "    },\n",
     "    {\n",
-    "        \"name\": \"en_terminal_command_only\",\n",
-    "        \"phase\": \"posttraining_strict_probe\",\n",
-    "        \"lang\": \"en\",\n",
-    "        \"expect\": \"strict_shell_command\",\n",
-    "        \"required_terms\": [\"find\", \"sort\", \"head\"],\n",
-    "        \"max_new_tokens\": 64,\n",
-    "        \"prompt\": \"Return one bash command only. No explanation. Task: find the 10 largest files under the current directory, excluding .git, sorted by size descending.\",\n",
+    "        \"name\": \"legal_criminal_code_raw_style\",\n",
+    "        \"source_format\": \"korean legal raw instruction style\",\n",
+    "        \"prompt\": \"\"\"다음은 대한민국 법령/자치법규 원문 일부입니다. 법률 한국어, 조문 구조, 번호 체계, 기관명, 시행일자 표현을 그대로 학습하십시오.\n",
+    "\n",
+    "[자료종류]\n",
+    "law\n",
+    "\n",
+    "[문서명]\n",
+    "형법\n",
+    "\n",
+    "[경로]\n",
+    "kr/형법/법률.md\n",
+    "\n",
+    "[부분]\n",
+    "1/1\"\"\",\n",
+    "        \"max_new_tokens\": 520,\n",
+    "        \"min_new_tokens\": 220,\n",
     "    },\n",
     "    {\n",
-    "        \"name\": \"en_tool_call_json\",\n",
-    "        \"phase\": \"posttraining_strict_probe\",\n",
-    "        \"lang\": \"en\",\n",
-    "        \"expect\": \"strict_json_keys\",\n",
-    "        \"required_keys\": [\"tool\", \"args\"],\n",
-    "        \"required_terms\": [\"shell\", \"du\"],\n",
-    "        \"max_new_tokens\": 96,\n",
-    "        \"prompt\": \"Return one JSON object only for a terminal tool call. Schema: {\\\"tool\\\": \\\"shell\\\", \\\"args\\\": {\\\"cmd\\\": string}}. Task: print current disk usage for the current directory in human-readable form.\",\n",
+    "        \"name\": \"legal_restoration_raw_style\",\n",
+    "        \"source_format\": \"korean legal raw instruction style\",\n",
+    "        \"prompt\": \"\"\"다음은 대한민국 법령/자치법규 원문 일부입니다. 법률 한국어, 조문 구조, 번호 체계, 기관명, 시행일자 표현을 그대로 학습하십시오.\n",
+    "\n",
+    "[자료종류]\n",
+    "law\n",
+    "\n",
+    "[문서명]\n",
+    "10ㆍ27법난 피해자의 명예회복 등에 관한 법률\n",
+    "\n",
+    "[경로]\n",
+    "kr/10ㆍ27법난피해자의명예회복등에관한법률/법률.md\n",
+    "\n",
+    "[부분]\n",
+    "1/1\"\"\",\n",
+    "        \"max_new_tokens\": 520,\n",
+    "        \"min_new_tokens\": 220,\n",
     "    },\n",
     "    {\n",
-    "        \"name\": \"en_python_code_only\",\n",
-    "        \"phase\": \"posttraining_strict_probe\",\n",
-    "        \"lang\": \"en\",\n",
-    "        \"expect\": \"strict_python_function\",\n",
-    "        \"required_terms\": [\"top_k_lengths\"],\n",
-    "        \"max_new_tokens\": 128,\n",
-    "        \"prompt\": \"Write Python code only. Define a function top_k_lengths(items, k) that returns the k longest strings from items, preserving original order for ties.\",\n",
+    "        \"name\": \"terminal_conversation_style_long\",\n",
+    "        \"source_format\": \"local terminal conversation instruction style\",\n",
+    "        \"prompt\": \"\"\"다음 터미널/코딩 작업 대화 맥락에서 assistant가 이어서 수행할 분석, 계획, 명령 JSON 또는 최종 응답을 작성하십시오.\n",
+    "\n",
+    "[user]\n",
+    "현재 디렉터리에서 용량이 큰 파일을 찾아 디스크 정리 후보를 보고 싶습니다. 숨김 폴더와 일반 폴더를 모두 확인하되, 결과는 사람이 읽기 쉽게 정리해 주세요.\"\"\",\n",
+    "        \"max_new_tokens\": 420,\n",
+    "        \"min_new_tokens\": 160,\n",
     "    },\n",
     "]\n",
     "\n",
-    "TEST_CASES = PRETRAINING_PROBES + STRICT_POSTTRAINING_PROBES\n",
-    "\n",
-    "\n",
-    "def strip_markdown_fence(text):\n",
-    "    stripped = text.strip()\n",
-    "    if stripped.startswith(\"```\"):\n",
-    "        stripped = re.sub(r\"^```[a-zA-Z0-9_-]*\\s*\", \"\", stripped)\n",
-    "        stripped = re.sub(r\"\\s*```$\", \"\", stripped)\n",
-    "    return stripped.strip()\n",
-    "\n",
-    "\n",
-    "def has_forbidden(text, case):\n",
-    "    return [term for term in case.get(\"forbidden_terms\", []) if term in text]\n",
-    "\n",
-    "\n",
-    "def missing_terms(text, case):\n",
-    "    return [term for term in case.get(\"required_terms\", []) if term not in text]\n",
-    "\n",
-    "\n",
-    "def validate_output(case, text):\n",
-    "    expect = case[\"expect\"]\n",
-    "    raw = text.strip()\n",
-    "    body = strip_markdown_fence(raw)\n",
-    "    if not raw:\n",
-    "        return \"FAIL: empty output\"\n",
-    "\n",
-    "    forbidden = has_forbidden(raw, case)\n",
-    "    if forbidden:\n",
-    "        return f\"FAIL: forbidden artifacts {forbidden}\"\n",
-    "\n",
-    "    if expect == \"ko_contains_terms\":\n",
-    "        missing = missing_terms(raw, case)\n",
-    "        if missing:\n",
-    "            return f\"WARN: pretraining probe missing expected terms {missing}\"\n",
-    "        if len(re.findall(r\"[가-힣]\", raw)) < 5:\n",
-    "            return \"WARN: too little Korean text\"\n",
-    "        return \"PASS: pretraining distribution signal\"\n",
-    "\n",
-    "    if expect == \"mentions_shell_concept\":\n",
-    "        missing = missing_terms(raw, case)\n",
-    "        if missing:\n",
-    "            return f\"WARN: shell concept missing terms {missing}\"\n",
-    "        return \"PASS: mentions expected shell concepts\"\n",
-    "\n",
-    "    if expect == \"strict_json_keys\":\n",
-    "        if raw != body:\n",
-    "            return \"FAIL: JSON wrapped in markdown fence; SFT/format tuning needed\"\n",
-    "        try:\n",
-    "            obj = json.loads(body)\n",
-    "        except Exception as exc:\n",
-    "            return f\"FAIL: invalid JSON ({exc.__class__.__name__})\"\n",
-    "        if not isinstance(obj, dict):\n",
-    "            return \"FAIL: JSON is not an object\"\n",
-    "        missing_keys = [key for key in case.get(\"required_keys\", []) if key not in obj]\n",
-    "        if missing_keys:\n",
-    "            return f\"FAIL: JSON missing keys {missing_keys}\"\n",
-    "        missing = missing_terms(json.dumps(obj, ensure_ascii=False), case)\n",
-    "        if missing:\n",
-    "            return f\"FAIL: JSON content missing terms {missing}\"\n",
-    "        return \"PASS: strict JSON\"\n",
-    "\n",
-    "    if expect == \"ko_grounded_summary\":\n",
-    "        missing = missing_terms(raw, case)\n",
-    "        if missing:\n",
-    "            return f\"FAIL: missing grounded Korean terms {missing}\"\n",
-    "        if len(re.findall(r\"[가-힣]\", raw)) < 10:\n",
-    "            return \"FAIL: too little Korean text\"\n",
-    "        if len(raw.splitlines()) > 4:\n",
-    "            return \"WARN: too many lines for short summary\"\n",
-    "        return \"PASS: Korean grounded-form probe\"\n",
-    "\n",
-    "    if expect == \"strict_shell_command\":\n",
-    "        if \"\\n\" in raw:\n",
-    "            return \"FAIL: more than one line\"\n",
-    "        if any(marker in raw for marker in [\"```\", \"We need\", \"Step\", \"1.\", \"Task:\"]):\n",
-    "            return \"FAIL: contains explanation/formatting\"\n",
-    "        if not re.match(r\"^(find|du|ls|python|sh|bash|fd|rg)\\b\", raw):\n",
-    "            return \"FAIL: not a shell command start\"\n",
-    "        missing = missing_terms(raw, case)\n",
-    "        if missing:\n",
-    "            return f\"FAIL: command missing expected terms {missing}\"\n",
-    "        return \"PASS: strict shell command\"\n",
-    "\n",
-    "    if expect == \"strict_python_function\":\n",
-    "        if \"```\" in raw:\n",
-    "            return \"FAIL: markdown fence present\"\n",
-    "        try:\n",
-    "            tree = ast.parse(raw)\n",
-    "        except SyntaxError as exc:\n",
-    "            return f\"FAIL: Python syntax error line {exc.lineno}\"\n",
-    "        funcs = [node for node in tree.body if isinstance(node, ast.FunctionDef)]\n",
-    "        if not funcs:\n",
-    "            return \"FAIL: no function definition\"\n",
-    "        if funcs[0].name != \"top_k_lengths\":\n",
-    "            return f\"FAIL: wrong function name {funcs[0].name!r}\"\n",
-    "        return \"PASS: syntactically valid target function\"\n",
-    "\n",
-    "    return \"WARN: unchecked expectation\"\n",
+    "print(\"probe count:\", len(KNOWLEDGE_PROMPTS))\n",
+    "for item in KNOWLEDGE_PROMPTS:\n",
+    "    print(\"-\", item[\"name\"], \"|\", item[\"source_format\"], \"| max/min\", item[\"max_new_tokens\"], item[\"min_new_tokens\"])\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Run Long Generations\n",
     "\n",
+    "The goal is to inspect the generated text itself after PT-style training: length, Korean fluency, domain terms, factual continuity, and repetition.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def count_tokens(text):\n",
+    "    return len(tokenizer.encode(text, add_special_tokens=False).ids)\n",
+    "\n",
+    "RUN_ONLY = None  # Example: {\"finance_bc_card_company\", \"kowiki_hunminjeongeum_raw_style\"}\n",
     "\n",
     "results = []\n",
-    "for case in TEST_CASES:\n",
-    "    print(\"=\" * 80)\n",
+    "for case in KNOWLEDGE_PROMPTS:\n",
+    "    if RUN_ONLY is not None and case[\"name\"] not in RUN_ONLY:\n",
+    "        continue\n",
+    "\n",
+    "    prompt = case[\"prompt\"]\n",
+    "    max_new = case.get(\"max_new_tokens\", DEFAULT_MAX_NEW_TOKENS)\n",
+    "    min_new = case.get(\"min_new_tokens\", DEFAULT_MIN_NEW_TOKENS)\n",
+    "\n",
+    "    print(\"=\" * 100)\n",
     "    print(\"case:\", case[\"name\"])\n",
-    "    print(\"phase:\", case[\"phase\"])\n",
-    "    print(\"expect:\", case[\"expect\"])\n",
-    "    print(\"prompt:\", case[\"prompt\"])\n",
+    "    print(\"source_format:\", case[\"source_format\"])\n",
+    "    print(\"prompt_chars:\", len(prompt), \"prompt_tokens:\", count_tokens(kohrm.format_kohrm_prompt(prompt, condition=\"direct\")))\n",
+    "    print(\"max_new_tokens:\", max_new, \"min_new_tokens:\", min_new)\n",
+    "    print(\"--- prompt ---\")\n",
+    "    print(prompt)\n",
+    "    print(\"--- output ---\")\n",
+    "\n",
     "    output = kohrm.generate_from_loaded(\n",
     "        model,\n",
     "        tokenizer,\n",
     "        cfg,\n",
-    "        case[\"prompt\"],\n",
-    "        max_new_tokens=case[\"max_new_tokens\"],\n",
-    "        **STRICT_SETTINGS,\n",
+    "        prompt,\n",
+    "        max_new_tokens=max_new,\n",
+    "        min_new_tokens=min_new,\n",
+    "        **LONG_TEXT_SETTINGS,\n",
     "    )\n",
-    "    verdict = validate_output(case, output)\n",
-    "    results.append({\"case\": case[\"name\"], \"phase\": case[\"phase\"], \"expect\": case[\"expect\"], \"verdict\": verdict, \"output\": output})\n",
-    "    print(\"verdict:\", verdict)\n",
-    "    print(\"--- output ---\")\n",
+    "    out_tokens = count_tokens(output)\n",
+    "    results.append({\n",
+    "        \"case\": case[\"name\"],\n",
+    "        \"source_format\": case[\"source_format\"],\n",
+    "        \"output_chars\": len(output),\n",
+    "        \"output_tokens\": out_tokens,\n",
+    "        \"output\": output,\n",
+    "    })\n",
     "    print(output)\n",
+    "    print(\"--- output stats ---\")\n",
+    "    print(\"chars:\", len(output), \"tokens:\", out_tokens)\n",
     "\n",
-    "print(\"=\" * 80)\n",
-    "print(json.dumps([{k: r[k] for k in [\"case\", \"phase\", \"expect\", \"verdict\"]} for r in results], indent=2, ensure_ascii=False))"
+    "print(\"=\" * 100)\n",
+    "print(json.dumps([{k: r[k] for k in [\"case\", \"source_format\", \"output_chars\", \"output_tokens\"]} for r in results], indent=2, ensure_ascii=False))\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 7. Optional Low-Temperature Retry\n",
+    "## 8. Optional Decode Settings Sweep\n",
     "\n",
-    "This retry is only diagnostic. It should not turn a strict post-training failure into a pass. Use it to see whether the bad output is a greedy decoding artifact or a checkpoint/format-alignment issue."
+    "Use this only if long output collapses to short tokens or repeats. It keeps the same training-format prompts and changes only decoding parameters.\n"
    ]
   },
   {
@@ -494,48 +418,47 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "RETRY_CASE_NAMES = {\"ko_finance_plain_qa\", \"ko_finance_short\", \"en_terminal_command_only\"}\n",
-    "RETRY_SETTINGS = dict(STRICT_SETTINGS)\n",
-    "RETRY_SETTINGS.update({\n",
-    "    \"temperature\": 0.2,\n",
-    "    \"top_p\": 0.85,\n",
-    "    \"repetition_penalty\": 1.22,\n",
-    "    \"no_repeat_ngram_size\": 4,\n",
-    "})\n",
-    "\n",
-    "for case in TEST_CASES:\n",
-    "    if case[\"name\"] not in RETRY_CASE_NAMES:\n",
-    "        continue\n",
-    "    print(\"=\" * 80)\n",
-    "    print(\"retry case:\", case[\"name\"])\n",
+    "SWEEP_CASE_NAME = \"finance_exchange_investment\"\n",
+    "SWEEP_PROMPT = next(item[\"prompt\"] for item in KNOWLEDGE_PROMPTS if item[\"name\"] == SWEEP_CASE_NAME)\n",
+    "\n",
+    "SWEEP_SETTINGS = [\n",
+    "    {\"name\": \"deterministic_minlen\", \"temperature\": 0.0, \"top_p\": 1.0, \"repetition_penalty\": 1.02, \"no_repeat_ngram_size\": 0},\n",
+    "    {\"name\": \"sample_balanced\", \"temperature\": 0.65, \"top_p\": 0.92, \"repetition_penalty\": 1.05, \"no_repeat_ngram_size\": 0},\n",
+    "    {\"name\": \"sample_more_diverse\", \"temperature\": 0.85, \"top_p\": 0.95, \"repetition_penalty\": 1.08, \"no_repeat_ngram_size\": 0},\n",
+    "]\n",
+    "\n",
+    "for settings in SWEEP_SETTINGS:\n",
+    "    run_settings = dict(LONG_TEXT_SETTINGS)\n",
+    "    run_settings.update({k: v for k, v in settings.items() if k != \"name\"})\n",
+    "    print(\"=\" * 100)\n",
+    "    print(\"decode:\", settings[\"name\"])\n",
     "    output = kohrm.generate_from_loaded(\n",
     "        model,\n",
     "        tokenizer,\n",
     "        cfg,\n",
-    "        case[\"prompt\"],\n",
-    "        max_new_tokens=case[\"max_new_tokens\"],\n",
-    "        **RETRY_SETTINGS,\n",
+    "        SWEEP_PROMPT,\n",
+    "        max_new_tokens=320,\n",
+    "        min_new_tokens=120,\n",
+    "        **run_settings,\n",
     "    )\n",
-    "    print(\"retry verdict:\", validate_output(case, output))\n",
-    "    print(output)"
+    "    print(output)\n",
+    "    print(\"chars:\", len(output), \"tokens:\", count_tokens(output))\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 8. Interpretation Checklist\n",
-    "\n",
-    "Do not read this notebook as a final KoHRM chat/SFT benchmark. The current public checkpoint is a rolling pretraining checkpoint. It has not yet received the later behavior SFT/LoRA/RL pass.\n",
+    "## 9. How To Read Results\n",
     "\n",
-    "Use the results this way:\n",
+    "For this PT knowledge probe, useful signs are:\n",
     "\n",
-    "- A pretraining probe `PASS` means the checkpoint shows a usable signal from that domain or task distribution.\n",
-    "- A pretraining probe `WARN` means the signal is weak or the prompt is still outside the learned distribution.\n",
-    "- A strict post-training probe `FAIL` is expected at this stage if the model has not learned stable JSON-only, command-only, grounded summary, or code-only behavior.\n",
-    "- A strict post-training probe `PASS` is useful, but it is not required before SFT.\n",
+    "- The output is long enough to inspect, not `Yes`, `B`, `0`, or one short fragment.\n",
+    "- Korean legal/wiki/finance outputs contain domain terms and coherent Korean paragraphs.\n",
+    "- Raw-style prompts may produce article/law-like continuations because that is how raw corpora were converted for PT.\n",
+    "- Terminal conversation style may produce analysis plus commands, because that is what the local terminal conversation data used as the response target.\n",
     "\n",
-    "For strict behavior, compare the same probes after the final continuation checkpoint and after `behavior_mini` / `terminal_tool_core` / `korean_domain_core` LoRA. The validator intentionally marks markdown-wrapped JSON, non-command text, syntactically invalid code, and non-Korean placeholder text as failures."
+    "If every long prompt still collapses to one-token or unrelated boilerplate despite `min_new_tokens`, the next thing to debug is the public converted weight/runtime compatibility, not prompt wording.\n"
    ]
   }
  ],
@@ -553,5 +476,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 5
+ "nbformat_minor": 0
 }