---
language:
- en
license: apache-2.0
tags:
- text-generation
- gpt
- from-scratch
- transformer
- causal-lm
- pharmaceutical
- pharma
- medical
- domain-specific
- rope
- rmsnorm
- swiglu
- synthetic-data
pipeline_tag: text-generation
model-index:
- name: PharmaGPT-336M
  results:
  - task:
      type: text-generation
      name: Text Generation
    metrics:
    - name: Validation Loss
      type: loss
      value: 0.3748
    - name: Perplexity
      type: perplexity
      value: 1.45
    - name: Training Loss
      type: loss
      value: 0.3748
---

# PharmaGPT-336M

A **336M parameter GPT** language model trained **entirely from scratch** on 200K synthetic pharmaceutical documents across 6 manufacturing domains.

**No pre-trained weights. No fine-tuning. Every component built from scratch**: custom BPE tokenizer, full transformer architecture (RoPE + RMSNorm + SwiGLU), training loop, and evaluation pipeline.

> **Paper**: [ArXiv preprint (coming soon)]()  
> **Blog**: [Medium article](https://medium.com/@parthchaturvedi_97130/i-built-a-336m-parameter-gpt-from-scratch-for-pharmaceutical-manufacturing-heres-everything-i-69707959aa78)  
> **Code**: Included in this repository

---

## Quick Start

```python
import torch
from tokenizers import Tokenizer

# Download model files from this repo, then:
ckpt = torch.load("best_model.pt", map_location="cpu", weights_only=False)

# Reconstruct model from saved config
from model import GPT  # model.py included in this repo
model = GPT(ckpt["model_config"])
model.load_state_dict(ckpt["model"])
model.eval()

# Load tokenizer
tok = Tokenizer.from_file("tokenizer/tokenizer.json")

# Generate pharmaceutical text
prompt = "<|deviation|>\nDuring manufacturing of Batch B-NDL-2026"
ids = torch.tensor([tok.encode(prompt).ids])
output = model.generate(ids, max_new_tokens=200, temperature=0.8, top_k=50)
print(tok.decode(output[0].tolist()))
```

### Generation with Different Domains

```python
prompts = {
    "deviation": "<|deviation|>\nDuring routine inspection of the tablet coating line",
    "batch_record": "<|batch_record|>\nBATCH PRODUCTION RECORD\nProduct: Metformin HCl 500mg Tablets",
    "sop": "<|sop|>\nSOP-ENV-205 | Environmental Monitoring Program",
    "stability": "<|stability_study|>\nSTABILITY STUDY REPORT\nProduct: Adalimumab 40mg/0.8mL",
    "pharmacovigilance": "<|icsr|>\nA 72-year-old female patient with history of diabetes",
    "scientific": "<|scientific_paper|>\nObjective: To evaluate the impact of granulation",
}

for domain, prompt in prompts.items():
    ids = torch.tensor([tok.encode(prompt).ids])
    out = model.generate(ids, max_new_tokens=150, temperature=0.8, top_k=50)
    print(f"\n{'='*60}\n[{domain.upper()}]\n{'='*60}")
    print(tok.decode(out[0].tolist()))
```

---

## Model Details

| Property | Value |
|---|---|
| **Parameters** | 336,380,928 (336M) |
| **Architecture** | Decoder-only Transformer (GPT) |
| **Embedding Dimension** | 1024 |
| **Attention Heads** | 16 |
| **Layers** | 24 |
| **Context Length** | 512 tokens |
| **Vocabulary** | 32,000 tokens (custom BPE) |
| **Positional Encoding** | Rotary (RoPE), base=10000 |
| **Normalization** | RMSNorm (ε=1e-6) |
| **Activation** | SwiGLU (FFN hidden=2752) |
| **Bias** | None (all linear layers) |
| **Weight Tying** | Embedding ↔ LM Head |
| **Dropout** | 0.1 |

### Architecture Highlights

This model implements the same architectural innovations found in LLaMA/Mistral, all coded from scratch:

- **RoPE** (Rotary Positional Embeddings) — encodes relative position through rotation of Q/K vectors
- **RMSNorm** — faster, simpler alternative to LayerNorm (no mean subtraction)
- **SwiGLU** — gated feed-forward network with Swish activation
- **No bias** in any linear layer — modern simplification
- **Weight tying** — token embedding and output projection share parameters
- **Pre-norm** architecture — normalize before attention/FFN, not after

---

## Training Details

| Parameter | Value |
|---|---|
| **Optimizer** | AdamW (β₁=0.9, β₂=0.95, wd=0.1) |
| **Learning Rate** | 2e-4 (peak), cosine decay |
| **Warmup** | 500 steps (linear) |
| **Batch Size** | 8 micro × 4 grad accum = 32 effective |
| **Iterations** | 15,000 |
| **Precision** | float16 mixed precision |
| **Gradient Clipping** | 1.0 (global norm) |
| **Gradient Checkpointing** | Enabled |
| **Hardware** | NVIDIA T4 (16GB), Kaggle free tier |
| **Training Time** | ~9 hours |
| **Cost** | $0 (free compute) |

### Training Results

| Metric | Value |
|---|---|
| Final Training Loss | 0.3748 |
| Best Validation Loss | 0.3748 |
| Validation Perplexity | 1.45 |
| Tokens Processed | ~245M |

*Note: Low perplexity reflects the structured/templated nature of synthetic training data. Real-world pharmaceutical text would yield higher perplexity.*

---

## Training Data: 6 Pharmaceutical Domains

The model was trained on 200K synthetic documents (~32M tokens) generated across six pharmaceutical manufacturing domains:

### 1. Manufacturing Deviation Reports (~33K samples)
Equipment failures, process excursions, out-of-specification results, root cause analysis (Ishikawa, 5-Why), CAPA documentation following ICH Q10.

### 2. Batch Production Records (~33K samples)
Raw material dispensing, process step documentation, in-process controls, critical process parameters (CPPs), yield calculations, lot disposition decisions.

### 3. Standard Operating Procedures (~33K samples)
Cleaning validation, environmental monitoring, aseptic processing, water system maintenance (WFI, PW), equipment qualification — in Q&A format.

### 4. Stability Studies (~33K samples)
ICH Q1A(R2) study designs, accelerated (40°C/75% RH) and long-term (25°C/60% RH) conditions, assay trending, degradation products, shelf-life determination.

### 5. Pharmacovigilance Case Reports (~33K samples)
Individual Case Safety Reports (ICSRs), adverse event narratives, MedDRA coding, WHO-UMC causality assessment (certain/probable/possible/unlikely).

### 6. Scientific Writing (~33K samples)
Formulation development, Design of Experiments (DoE), analytical method development/validation, dissolution studies, results and discussion sections.

---

## Special Tokens

| Token | Purpose | Example Use |
|---|---|---|
| `<\|deviation\|>` | Start of deviation report | Triggers investigation-style generation |
| `<\|batch_record\|>` | Start of batch record | Triggers manufacturing record format |
| `<\|sop\|>` | Start of SOP document | Triggers procedural/Q&A format |
| `<\|stability_study\|>` | Start of stability study | Triggers ICH-compliant study format |
| `<\|icsr\|>` | Start of pharmacovigilance case | Triggers adverse event narrative |
| `<\|scientific_paper\|>` | Start of scientific writing | Triggers academic/research style |
| `<\|end\|>` | End of document | Marks document boundary |

---

## Repository Contents

```
├── best_model.pt          # Full checkpoint (model weights + config + metadata)
├── config.json            # Architecture specification (JSON)
├── tokenizer/
│   └── tokenizer.json     # Trained BPE tokenizer (32K vocab)
├── model.py               # Complete model source code (GPT + all components)
├── tokenizer.py           # Tokenizer training/loading utilities
└── README.md              # This file
```

### Loading Without `model.py`

If you want to inspect the architecture without running the custom code:

```python
import torch, json

# Load config
with open("config.json") as f:
    config = json.load(f)
print(config)
# {'vocab_size': 32000, 'n_embd': 1024, 'n_head': 16, 'n_layer': 24, ...}

# Load checkpoint metadata
ckpt = torch.load("best_model.pt", map_location="cpu", weights_only=False)
print(f"Keys: {ckpt.keys()}")
print(f"Val loss: {ckpt.get('best_val_loss')}")
print(f"Iteration: {ckpt.get('iter_num')}")
```

---

## Intended Use

### Primary Use Cases
- **Educational**: Understanding how modern GPT architectures work end-to-end
- **Research baseline**: Starting point for pharmaceutical NLP research
- **Template generation**: Generating draft pharmaceutical document structures
- **Domain adaptation**: Fine-tuning on real pharmaceutical data for production use

### Out-of-Scope Uses
- **Clinical decision-making**: This model generates plausible but NOT factually verified content
- **Regulatory submissions**: Generated text requires expert review and verification
- **Production deployment without validation**: The model was trained on synthetic data only
- **General-purpose chat**: This is a domain-specific completion model, not a chatbot

---

## Limitations and Risks

| Limitation | Impact | Mitigation |
|---|---|---|
| Synthetic training data | May generate structurally correct but factually wrong content | Always verify with domain experts |
| 336M parameters | Limited reasoning and knowledge capacity | Use as starting point, not final solution |
| English only | Cannot process multilingual pharmaceutical docs | Extend training data for other languages |
| No instruction tuning | Cannot follow complex instructions | Fine-tune with instruction data |
| Context length (512) | Cannot process long documents in one pass | Chunk documents or extend context |

### Ethical Considerations
- Generated pharmaceutical content should NEVER be used for actual drug manufacturing without expert review
- The model may reproduce biases present in the synthetic data templates
- Not intended as a replacement for qualified pharmaceutical professionals

---

## Citation

If you use PharmaGPT in your research, please cite:

```bibtex
@misc{chaturvedi2026pharmagpt,
  title={PharmaGPT: A Domain-Specific Language Model for Pharmaceutical Manufacturing Intelligence Trained from Scratch on Synthetic Data},
  author={Chaturvedi, Parth},
  year={2026},
  howpublished={\url{https://huggingface.co/ParthChat1802/PharmaGPT-336M}},
}
```

---

## Technical Notes for Reproducibility

### Checkpoint Format

The `best_model.pt` file is a PyTorch checkpoint dictionary containing:

```python
{
    "model": OrderedDict,        # model.state_dict()
    "model_config": GPTConfig,   # dataclass with architecture params
    "config": dict,              # training configuration
    "iter_num": int,             # iteration at save time
    "best_val_loss": float,      # best validation loss achieved
}
```

### System Requirements

- **Inference**: Any machine with 2GB+ RAM and PyTorch installed
- **Training (reproduce)**: NVIDIA GPU with 8GB+ VRAM, or Apple M-series with 16GB+ unified memory
- **Dependencies**: `torch>=2.0`, `tokenizers>=0.13`

### Reproducing Training

```bash
git clone <source-repo>
cd gpt-from-scratch
pip install -r requirements.txt

# Generate data
python -m data.generators.master_generator

# Train tokenizer + model
python -m src.train_pharma
```

Or use the Kaggle notebook for GPU-accelerated training (see repository).

---

## License

**Apache 2.0** — Use freely for any purpose (commercial, research, educational). Attribution appreciated but not legally required beyond the license notice.