|
|
| --- |
| license: mit |
| datasets: |
| - ethanker/nanomind_1m |
| language: |
| - en |
| library_name: transformers |
| tags: |
| - gpt |
| - decoder-only |
| - llama |
| - tiny |
| pipeline_tag: text-generation |
| --- |
| |
| # nanomind-step-002000 (early experiment checkpoint) |
|
|
| This is an early checkpoint (step 2,000) from a small decoder-only GPT-style experiment. It is shared primarily for transparency and to help others reproduce or build upon the setup. This checkpoint is not production-ready. |
|
|
| ## What this is |
| - Model: small LLaMA-style decoder-only (RMSNorm, SwiGLU, RoPE, MQA/GQA-compatible) |
| - Checkpoint: step_002000 from run1 |
| - Data: curated 1M-doc mix (English), hosted at the public dataset repo: [ethanker/nanomind_1m](https://huggingface.co/datasets/ethanker/nanomind_1m) |
| - Intended use: research/experimentation only |
| |
| ## How it was trained (run1) |
| - Script: `train_run1.py` (included here) with the exact launch command in `RUN_COMMAND.txt`. |
| - Key settings used for run1: |
| - seq_len 2048, hidden_size 512, n_layers 16, n_heads 8, n_kv_heads 1 |
| - global_batch_size 64, micro_batch_size 1, AdamW lr 1e-3, warmup 2000 |
| - bf16 autocast, gradient clipping 1.0 |
| |
| ## Quick eval snapshot (for context only) |
| - In-domain ppl (small slice): ~1.06 (expected to be low given early-stage in-domain evaluation) |
| - Generations: fluent but sometimes regurgitative; this is a very early checkpoint |
| |
| ## Optimizations implemented for subsequent runs |
| These were implemented in the training/data pipeline for future iterations (beyond this checkpoint): |
| - Near-duplicate filtering (MinHash+LSH) and stronger boilerplate heuristics |
| - Optional gradient checkpointing and torch.compile for better memory/throughput |
| - Periodic quick perplexity checks on a small token budget |
| |
| References: |
| - Chinchilla compute-optimal scaling: https://arxiv.org/abs/2203.15556 |
| - Deduplication improves LMs: https://arxiv.org/abs/2107.06499 |
| - Dedup mitigates privacy risks: https://arxiv.org/abs/2202.06539 |
| - FlashAttention-3: https://arxiv.org/abs/2407.08608 |
| - YaRN long-context: https://arxiv.org/abs/2309.00071 |
| |
| ## Load and sample |
| ```python |
| from transformers import AutoTokenizer, LlamaForCausalLM |
| import torch |
| m = 'ethanker/nanomind-step-002000' |
| tok = AutoTokenizer.from_pretrained(m, use_fast=True) |
| model = LlamaForCausalLM.from_pretrained(m, torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32) |
| model.eval().to('cuda' if torch.cuda.is_available() else 'cpu') |
| |
| prompt = "Once upon a time," |
| inputs = tok(prompt, return_tensors='pt').to(model.device) |
| out = model.generate(**inputs, do_sample=True, top_p=0.9, temperature=0.8, max_new_tokens=128) |
| print(tok.decode(out[0], skip_special_tokens=True)) |
| ``` |
| |
| ## Files |
| - `model.safetensors`, tokenizer/config files |
| - `train_run1.py` (training code snapshot) |
| - `RUN_COMMAND.txt` (exact command used) |
| |
| ## Notes |
| - Early and exploratory; expect limited generalization and occasional regurgitation. |
| - Please prefer the referenced dataset repo and scripts for reproducibility and your own experiments. |
| |
