---
license: apache-2.0
tags:
- causal-lm
- pretraining
- pytorch
library_name: pytorch
---

# Ember-Ardent-350M

Ember Ardent is a roughly 351M-parameter decoder-only language model checkpoint.

## Important status

This is **not fully pretrained**. It was trained for approximately 3.26B tokens, rather than the planned 10B-token run, and is published as a research checkpoint. It is intentionally left in this state unless someone wants to continue pretraining (CPT) from the included checkpoint.

The checkpoint is not presented as instruction-tuned, chat-tuned, or production-ready. Expect factual errors, repetition, and incomplete capabilities.

## Architecture

- 24 transformer layers
- 1024 hidden size
- 16 attention heads / 4 KV heads (GQA)
- SwiGLU feed-forward layers
- RMSNorm and RoPE
- tied input/output embeddings
- 32,768-token vocabulary
- maximum sequence length 2,048
- DGM learning-rate schedule during pretraining

## Files

- `step-00014000.pt`: native PyTorch checkpoint, including model state and training metadata
- `tokenizer.json`: tokenizer used during training
- `tokenizer_metadata.json`: tokenizer metadata
- `config.json`: architecture and checkpoint metadata

The `.pt` file uses the project's native `ArdentForCausalLM` implementation. It is not directly loadable with `AutoModelForCausalLM` without the Ember Ardent source code.

## Continuing pretraining

Load `step-00014000.pt` with the Ember Ardent training code and resume the optimizer, scheduler, RNG, and token position. Do not treat this checkpoint as a final model.

## Evaluation snapshot

On a short held-out validation sample at publication time:

- loss: 3.51
- perplexity: 33.45

These are development measurements, not standardized benchmark scores.