Reza2kn/Motarjem-v0.5-stage1-step24000

Motarjem v0.5 Stage 1 — step 24,000

This is an interim research checkpoint, not a finished Motarjem release. It is a 108M-parameter bidirectional English–Persian translation model continued from tiiuae/Falcon-H1-Tiny-Multilingual-100M-Base.

The checkpoint was saved at step 24,000 of a planned 42,122-step epoch. At this point it had processed 8,784,406,600 sampled tokens globally (56.9% of the Stage-1 token stream).

Important limitation

Although the Falcon-H1 configuration declares max_position_embeddings: 262144, this checkpoint was not trained for 256k-token document translation. The Stage-1 training sequences had:

• median source length: 20 tokens
• 99th-percentile source length: 115 tokens
• maximum source length: 822 tokens
• maximum combined source/target sequence: 1,024 tokens

Long-document translation quality and completeness are unvalidated. Do not describe this checkpoint as a 256k document translator.

Evaluation

Step 24,000 itself was not evaluated. The nearest completed evaluation is step 20,000, using greedy decoding with a 256-token output cap:

Evaluation lane	chrF2
WMT24++ EN→FA	38.58
WMT24++ FA→EN	47.92
Wikipedia holdout EN→FA	42.55
Wikipedia holdout FA→EN	44.15

The four-lane mean was 43.30. The separate QuickMT baseline averaged 50.73 under its own runtime contract, so this checkpoint should not be presented as matching or beating QuickMT.

Prompt format

<|end_of_text|><|translate|><|source-language|>SOURCE<|target-language|>

Use <|en|> and <|fa|> as the language controls.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Reza2kn/Motarjem-v0.5-stage1-step24000"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
).cuda().eval()

source = "Hello world."
prompt_ids = [
    tokenizer.bos_token_id,
    tokenizer.convert_tokens_to_ids("<|translate|>"),
    tokenizer.convert_tokens_to_ids("<|en|>"),
    *tokenizer.encode(source, add_special_tokens=False),
    tokenizer.convert_tokens_to_ids("<|fa|>"),
]
input_ids = torch.tensor([prompt_ids], device="cuda")
with torch.inference_mode():
    output = model.generate(
        input_ids,
        do_sample=False,
        max_new_tokens=256,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(output[0, input_ids.shape[1]:], skip_special_tokens=True))

Training data and provenance

Stage 1 contains 266,050,112 directional training records rendered from the pinned QuickMT union and its two English-source backtranslation datasets. EN→FA records from the QuickMT union were repeated twice; the NewsCrawl and MADLAD backtranslations were admitted only as FA→EN training records.

The QuickMT union does not preserve complete row-level upstream provenance and its dataset card does not specify a dataset-level license. The two backtranslation datasets are circular with the QuickMT teacher lineage. Users must review the original dataset cards and upstream licenses before redistribution or commercial use.

Exact checkpoint hashes and training facts are recorded in training-receipt.json.