---
license: apache-2.0
tags:
  - build-small-hackathon
  - pgsm
  - exactstate-memory
  - non-transformer
  - language-model
  - surprisal
  - fineweb-edu
  - tiny-model
  - tiny-titan
  - well-tuned
datasets:
  - HuggingFaceFW/fineweb-edu
---

# PGSM Text Surprisal Editor Model

This repository contains the trained model weights used by the Hugging Face Space:

https://huggingface.co/spaces/build-small-hackathon/pgsm-text-surprisal-editor

## Model Summary

PGSM Text Surprisal Editor is powered by a compact non-Transformer language model based on a custom ExactState Memory / PGSM architecture.

The model is used to score whole-word surprisal by evaluating how predictable each removed word is from its left and right context.

## Architecture

- Architecture: PGSM / ExactState Memory
- Transformer blocks: 0
- Self-attention layers: 0
- Parameters: approximately 4 million
- Vocabulary: approximately 2k tokens
- Model file: `final_infer.pt`

This model does not use Transformer self-attention. Context is propagated through learned state transitions rather than pairwise attention computations.

## Training

The model was fully trained by the author on approximately 19 billion tokens from FineWeb-Edu.

Training details:

- Training source: FineWeb-Edu
- Training scale: approximately 19B tokens
- Training type: full custom training by the author
- Base architecture: PGSM / ExactState Memory
- Off-the-shelf Transformer checkpoint used: none
- Final inference weights: `final_infer.pt`

## Intended Use

This model is intended for the PGSM Text Surprisal Editor Space, where it powers whole-word surprisal heatmaps for pasted text.

The model is designed for experimentation, visualization, and language-analysis demos rather than production writing assistance or factual generation.

## Limitations

- Very small model size compared with mainstream LLMs
- Compact vocabulary
- Designed for surprisal visualization, not general-purpose chat
- Outputs should be treated as model-analysis signals, not factual judgments
- Training and evaluation details are summarized here for hackathon review

## Hackathon Context

This model supports the Hugging Face Build Small Hackathon submission:

- Track: Thousand Token Wood
- Badges: Tiny Titan, Well-Tuned, Off the Grid, Field Notes

The key goal is to demonstrate a very small, fully trained, non-Transformer language model running locally inside a Hugging Face Space.