Upload folder using huggingface_hub

README.md

@@ -0,0 +1,84 @@
+---
+license: mit
+tags:
+  - medical
+  - gpt
+  - autoresearch
+  - causal-lm
+  - from-scratch
+language:
+  - en
+library_name: pytorch
+pipeline_tag: text-generation
+---
+
+# Medical GPT-50M
+
+A 50M-parameter medical language model trained from scratch on 2.9M medical Q&A examples using the [autoresearch](https://github.com/karpathy/autoresearch) methodology.
+
+## Model Details
+
+| Parameter | Value |
+|-----------|-------|
+| Parameters | 50.3M |
+| Architecture | GPT (RoPE, RMS norm, sliding window, ReluSquared MLP) |
+| Vocabulary | Medical BPE (8192 tokens) |
+| Context length | 2048 tokens |
+| Layers | 8 |
+| Heads | 8 |
+| Head dim | 128 |
+| Window pattern | SSSL |
+| Optimizer | MuonAdamW (Muon for matrices, AdamW for embeddings) |
+| Validation BPB | 1.1217 |
+| Training tokens | 37.0M |
+
+## Training Data
+
+Trained on 3 medical Q&A datasets (2.9M examples, 17.4GB JSONL):
+- **OpenMed/Medical-Reasoning-SFT-Mega** (1.78M rows) — multi-domain medical reasoning with chain-of-thought
+- **lingshu-medical-mllm/ReasonMed** (1.11M rows) — medical reasoning Q&A
+- **FreedomIntelligence/medical-o1-reasoning-SFT** [en] (19.7K rows) — used as validation set
+
+## Experiment Insights
+
+This model was trained using the autoresearch autonomous experimentation loop. Key findings from ~25 experiments:
+
+| Experiment | val_bpb | Insight |
+|-----------|---------|---------|
+| Baseline (batch=128, OOM) | - | L4 has 24GB, not H100's 80GB |
+| batch=32 | 1.252 | First working baseline on L4 |
+| batch=32 + mlr=0.06 + warmdown=0.3 | 1.160 | Higher matrix LR helps medical text |
+| total_batch=2^16, batch=8 | 1.125 | **Key finding**: 4x more optimizer steps >> throughput |
+| + unembedding_lr=0.008 | 1.123 | Small gain from discriminative LRs |
+| + embedding_lr=1.2 | 1.115 | Medical vocabulary needs faster embedding adaptation |
+
+**Key insight**: On time-budgeted training (5 min), smaller total batch size = more optimizer steps = dramatically better val_bpb. This is the single biggest lever.
+
+## How to Use
+
+This is a raw pretrained model with a custom architecture (not HuggingFace Transformers compatible). Load with PyTorch:
+
+```python
+import torch
+import safetensors.torch
+state_dict = safetensors.torch.load_file("model.safetensors")
+# Architecture details in config.json
+```
+
+## Limitations
+
+- **Not instruction-tuned** — this is a base pretrained model
+- **5-minute training budget** — trained for research/exploration, not production
+- **50M parameters** — small model, intended as foundation for embedding/classification experiments
+- **Custom architecture** — not directly compatible with HuggingFace Transformers
+
+## Citation
+
+```
+@misc{medical-gpt-50m,
+  title={Medical GPT-50M: Autonomous Medical LM Research},
+  author={Axone AI},
+  year={2026},
+  url={https://huggingface.co/axonee/medical-gpt-50m}
+}
+```

config.json

@@ -0,0 +1,14 @@
+{
+  "model_type": "gpt-medical",
+  "architecture": "autoresearch-gpt",
+  "vocab_size": 8192,
+  "max_seq_len": 2048,
+  "depth": 8,
+  "num_heads": 8,
+  "head_dim": 128,
+  "aspect_ratio": 64,
+  "window_pattern": "SSSL",
+  "num_params_M": 50.3,
+  "val_bpb": 1.121693739215597,
+  "total_tokens_M": 37.0
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81b341c16bb33e34d7aae9f5cb7ba75f54469974b824517ef41946a1280d071e
+size 159392432

token_bytes.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc27e313a03c6dd716304d3ef9261028799abe9643747da6bff6fd43c45ed6aa
+size 34373

tokenizer.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a6333a6cb981f3f7b61cb266ba8b370bb87c93cf4029ebfdcfb72604bad1d27
+size 94629