kurogane/HRM-Tiny-JA-500k-2x2-RC1

HRM Tiny JA 500k 2x2 RC1 / 非公式 HRM-Text 小型日本語 RC1

Important / 重要
This is an unofficial experimental model. It is not released, endorsed, or certified by Sapient Intelligence, Hugging Face, or LLM-jp.
本モデルは 非公式の実験モデル です。Sapient Intelligence、Hugging Face、LLM-jp による公式モデル、公式派生、認定モデルではありません。

This repository draft is for the release candidate selected from:

./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2/checkpoint-62500

Suggested public-facing name:

hrm-tiny-ja-500k-2x2-rc1

This checkpoint is intended as a small Japanese instruction / explanation generation experiment based on the HF HRM-Text implementation. It is not a general-purpose chat model and should not be used as a factual authority.

---

1. Model summary / モデル概要

Item	Value
Architecture	HRM-Text style causal LM, PrefixLM prompt masking
Size class	Tiny experimental model, approximately 120M parameter class
Hidden size	512
Vocab size	99,584 model vocab rows
Tokenizer length	99,574 effective tokenizer length
Tokenizer	llm-jp/llm-jp-3-150m-instruct3 tokenizer
Input embeddings	initialized from llm-jp/llm-jp-3-150m-instruct3 input embeddings
LM head	untied, randomly initialized before training
Layers	num_layers_per_stack=2, L stack 2 real layers, H stack 2 real layers
H/L cycles	H_cycles=2, L_cycles=2
PrefixLM	enabled; prompt tokens use token_type_ids=1, response tokens use token_type_ids=0
Final selected checkpoint	checkpoint-62500, corresponding to roughly 1 epoch over 500k examples with batch size 8

日本語説明生成の検証用途を主目的とした小型モデルです。RC1では、2 epoch 学習の最終checkpointではなく、1 epoch相当の checkpoint-62500 を採用しています。1.5 epoch候補はfirst-token指標がやや良い一方で、本文がプロンプトから飛びやすかったため採用していません。

---

2. Prompt format / プロンプト形式

Training and evaluation used the following plain instruction format.

<s>以下は、タスクを説明する指示です。要求を適切に満たす応答を書きなさい。

### 指示:
{instruction}

### 応答:

The model was trained with PrefixLM-style masks. In local inference, prompt tokens should be assigned token_type_ids=1, and generated response tokens should be assigned token_type_ids=0.

---

3. Quick start / 推論サンプル

Install

pip install -U "git+https://github.com/huggingface/transformers.git" torch safetensors

HRM-Text support may require a recent Transformers version. The official HRM-Text documentation shows the general pattern of using AutoTokenizer and AutoModelForCausalLM.generate(...) for HRM-Text style models, while noting that the official HRM-Text-1B base model does not ship a stable chat template. For this RC, use the plain prompt above and the bundled inference script if PrefixLM token_type_ids handling is needed.

Recommended local script

python scripts/inference_hrm_prefixlm.py \
  --model_dir ./release/hrm_tiny_ja_500k_2x2_rc1 \
  --instruction "自然言語処理とは何か" \
  --max_new_tokens 128 \
  --temperature 0.65 \
  --top_p 0.9 \
  --top_k 50 \
  --repetition_penalty 1.2 \
  --no_repeat_ngram_size 4

Python API example

from pathlib import Path
from scripts.inference_hrm_prefixlm import load_model_and_tokenizer, generate_text

model_dir = Path("./release/hrm_tiny_ja_500k_2x2_rc1")
model, tokenizer, device = load_model_and_tokenizer(model_dir)

text = generate_text(
    model=model,
    tokenizer=tokenizer,
    device=device,
    instruction="人工知能と機械学習の違いを説明してください",
    max_new_tokens=128,
    temperature=0.65,
    top_p=0.9,
    top_k=50,
    repetition_penalty=1.2,
    no_repeat_ngram_size=4,
)
print(text)

---

4. Training data / 学習データ

The RC1 dataset is:

./data/hrm_text_llmjp_tok_1024_release_ja_500k_clean

Target composition after filtering and source sampling:

Source	Dataset	Target train examples
magpie	llm-jp/magpie-sft-v1.0	250,000
aya	weblab-GENIAC/aya-ja-evol-instruct-calm3-dpo-masked	105,000
oasst2	llm-jp/oasst2-33k-ja	75,000
oasst1	llm-jp/oasst1-21k-ja	40,000
dolly	llm-jp/databricks-dolly-15k-ja, categories open_qa,general_qa	30,000
Total		500,000

Validation splits:

Split	Size / target	Notes
validation	3,500	source-balanced clean validation
validation_core	500	fixed sanity prompts plus clean validation samples
validation_magpie	700	source-specific validation
validation_aya	700	source-specific validation
validation_oasst2	600	source-specific validation
validation_oasst1	500	source-specific validation
validation_dolly_clean	500	cleaned Dolly validation alias

Key filters:

• Require Japanese text in instruction and response.
• Drop numeric-only, symbol-heavy, refusal/meta, and apology-style responses.
• Drop many list-like or numbered responses.
• Drop repeated phrase / repeated n-gram responses.
• Drop long context extraction tasks and prompt-heavy examples.
• Token constraints: default min_response_tokens=24, max_response_tokens=320, max_prompt_tokens=448, max_total_tokens=896.
• Banned first response tokens include Output, Solution, first, second, はい, いいえ, 以下, list markers, and numeric starts 0 through 20.

See cards/DATASET_CARD_DRAFT.md for a fuller dataset description.

---

5. Training setup / 学習設定

Selected RC1 run:

CUDA_VISIBLE_DEVICES=0 accelerate launch --num_processes 1 train_hrm_tiny_sft.py \
  --dataset_dir ./data/hrm_text_llmjp_tok_1024_release_ja_500k_clean \
  --eval_split_name validation_core \
  --output_dir ./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2 \
  --num_layers_per_stack 2 \
  --max_epochs 2 \
  --per_device_train_batch_size 8 \
  --gradient_accumulation_steps 1 \
  --learning_rate 2e-4 \
  --weight_decay 0.05 \
  --warmup_ratio 0.03 \
  --loss_normalization active_tokens \
  --first_response_token_weight 4 \
  --first_response_token_target first_non_space \
  --embedding_init_mode raw \
  --lm_head_init_mode random \
  --eval_strategy epoch \
  --save_strategy epoch \
  --save_total_limit 4

Training metrics for the full 2-epoch run:

Metric	Value
epoch	2.0
train_loss	1.1787
train_runtime	1:48:06.09
train_samples_per_second	154.176
train_steps_per_second	19.272

Selected checkpoint:

checkpoint-62500

---

6. Evaluation summary / 評価要約

Primary evaluation on release_ja_500k_clean / validation_core, first 256 examples:

Metric	Value
mean loss	3.322312
perplexity	27.724
active label tokens	32,079
active tokens / example	125.309
first-token mean loss	4.430201
first-token perplexity	83.948
first-token top1 / top5 / top20	45.703% / 57.422% / 69.922%

Legacy comparison on old mixfix_20k_ja4x / validation_ja_dolly, first 256 examples:

Metric	Value
mean loss	5.383554
perplexity	217.795
active label tokens	22,463
active tokens / example	87.746

Known behavior:

• The model often selects a relevant first token for definition/explanation prompts.
• It can produce Japanese explanation-like text.
• It is still weak at factual accuracy, multi-sentence coherence, and causal explanations.
• It can hallucinate details, confuse related concepts, or drift into generic explanations.
• It is not suitable for safety-critical or factual-critical use.

See cards/EVAL_SUMMARY.md for selected logs and comparison notes.

To prepate this dataset, you can use prepare_release_ja_dataset_500k.py.

python prepare_release_ja_dataset_500k.py \
  --output_dir ./data/hrm_text_llmjp_tok_1024_release_ja_500k_clean \
  --num_proc 8 \
  --overwrite

より厳しめに長文contextや長い応答を落とす場合はこちらです。

python prepare_release_ja_dataset_500k.py \
  --output_dir ./data/hrm_text_llmjp_tok_1024_release_ja_500k_clean_strict \
  --max_context_chars 700 \
  --max_total_tokens 768 \
  --max_response_tokens 256 \
  --num_proc 8 \
  --overwrite

---

7. Example generations / 生成例

Generation parameters used for sanity checks:

temperature=0.65
top_p=0.9
top_k=50
repetition_penalty=1.15 or 1.2
no_repeat_ngram_size=3 or 4
max_new_tokens=128

Example prompt:

自然言語処理とは何か

Observed RC1-style output excerpt:

自然言語処理（NLP）は、人間が言葉を理解し、生成するための技術で、...

The sample is not guaranteed to be factually accurate. Use the model as an experimental small HRM-Text checkpoint, not as a factual assistant.

---

8. Limitations / 制限事項

• 非公式・研究用の小型モデルです。
• 事実誤認、幻覚、定義の混線、文章の途中崩れがあります。
• 日本語説明生成に寄せていますが、チャットモデルとして安定していません。
• 長文コンテキスト、厳密な数学、コード生成、翻訳、要約、抽出には向いていません。
• 医療・法律・金融・安全に関わる判断には使わないでください。
• Dataset and upstream license compatibility must be checked before public redistribution.

---

9. Release checklist / 公開前チェック

Before public upload:

1. Copy checkpoint-62500 into a clean release directory.
2. Copy tokenizer files from the training output directory.
3. Run scripts/evaluate_rc1.sh and save logs under eval_logs/.
4. Confirm source dataset licenses and redistribution constraints.
5. Update model repo ID, author, date, and license field.
6. Add clear disclaimer: unofficial, experimental, no warranty.

Suggested packaging command:

bash scripts/package_rc1_checkpoint.sh \
  ./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2/checkpoint-62500 \
  ./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2 \
  ./release/hrm_tiny_ja_500k_2x2_rc1