---
license: apache-2.0
datasets:
- PleIAs/SYNTH
pipeline_tag: text-generation
library_name: transformers
tags:
- conv2d
- conv1d
- hilbert
- chunk-token_memory
- llm
- torch
---

# ConvGPT-v2 B200 Full SYNTH 20h

`mkurman/convgpt-v2-b200-full-synth-20h` is an experimental ConvGPT-v2 language model checkpoint trained on synthetic reasoning/chat data. The latest uploaded checkpoint is `checkpoint-134000`, with intermediate checkpoints available from roughly `checkpoint-47000` through `checkpoint-checkpoint-134000`.

This model is primarily a research artifact for testing convolution-first language modeling at scale. It is **not a standard Transformer**: ConvGPT-v2 replaces dense self-attention blocks with a hybrid causal 1D/2D convolutional architecture plus sparse chunk-token retrieval memory.

## Reproduction - important!

Use the tokenizer from the tokenizer directory, as the one in the checkpoint files is incorrect.

## Model Details

- **Architecture:** ConvGPT-v2 custom causal language model
- **Latest checkpoint:** `checkpoint-134000`
- **Training run:** B200 180GB full-SYNTH 20h run + 72h local (RTX 3090ti, seq 1024, ebs 64, lr 1e-4)
- **Training data:** PleIAs/SYNTH-style synthetic reasoning/chat examples
- **Approx. tokens seen:** ~31.98B tokens
- **Reported CE loss:** ~1.753 nats/token
- **Approx. perplexity:** `exp(1.753) ≈ 5.77`
- **Vocab size:** 32,024
- **Training sequence length:** 512->2048
- **Configured max position capacity:** 65,536 tokens via 256×256 grid
- **Precision:** BF16 training

## Architecture Summary

ConvGPT-v2 is a dense-self-attention-free experimental LM. It uses:

- causal 1D convolution branch
- causal 2D convolution branch over a 256×256 Hilbert-packed token grid
- gated fusion between 1D and 2D branches
- RoPE/no-position hybrid configuration
- sparse `chunk_token_memory` retrieval every 2 layers
- custom Triton kernels for active causal 2D gathering and sparse retrieval paths

Important distinction: the model is **free of standard dense Transformer self-attention**, but it is not strictly “attention-free” because the `chunk_token_memory` router performs sparse attention-like retrieval over selected prior chunks/tokens.

## Repository Contents

This repository includes:

- `checkpoint-61000/` — latest model checkpoint
- earlier checkpoints from approximately `checkpoint-47000` to `checkpoint-61000`
- ConvGPT-v2 source files:
  - `modeling_convgpt_v2.py`
  - `configuration_convgpt_v2.py`
  - `registration.py`
  - `__init__.py`
- training scripts:
  - `train_convgpt_v2_2d_pleias_long.py`
  - `run_convgpt_v2_b200_full_synth_20h.sh`

## Intended Use

This model is intended for:

- research into convolutional / dense-attention-free language model architectures
- experiments with sparse retrieval memory as an alternative to Transformer self-attention
- studying scaling behavior of ConvGPT-v2 on synthetic reasoning/chat data
- checkpoint analysis, BPB/perplexity evaluation, and generation experiments

It is **not** intended for production use, medical advice, legal advice, or safety-critical applications.

## Usage

Because this is a custom architecture, load with trusted remote code:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

repo = "mkurman/convgpt-v2-b200-full-synth-20h"
checkpoint = f"{repo}/checkpoint-134000"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).cuda().eval()

messages = [{"role": "user", "content": "Explain hypertension briefly."}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=False))
```

## Training Notes

The model was trained using a custom ConvGPT-v2 trainer on synthetic reasoning data. The uploaded shell and Python scripts document the exact training setup, including:

- B200-oriented BF16 training
- full PleIAs/SYNTH streaming mode
- causal full-token loss
- checkpoint upload support to Hugging Face Hub

The latest run reached approximately:

```text
checkpoint: checkpoint-61000
cross entropy: ~1.753 nats/token
tokens seen: ~31,981,568,000
per-token perplexity: ~5.77
bits/token: ~2.53
```

## Limitations

This is an experimental research checkpoint. Known limitations:

- generation quality is not yet comparable to mature Transformer LMs
- may hallucinate or produce malformed reasoning
- trained mostly on synthetic data [PleIAs/SYNTH](https://huggingface.co/datasets/PleIAs/SYNTH), so distributional coverage is limited to this dataset
- not instruction-safety tuned
- custom architecture requires `trust_remote_code=True`
- sparse retrieval memory is attention-like, so the model should not be described as strictly attention-free

## Citation / Attribution

If you use this model, please refer to it as:

> ConvGPT-v2 B200 Full SYNTH 20h — an experimental dense-self-attention-free convolutional sparse-retrieval language model by Mariusz Kurman.