captionbert-8192-v2

A 58.3M standalone sentence encoder distilled from the geometric consensus of five BERT-family teachers. No expert models at inference: tokenizer + this model, 768-d L2-normalized output.

12 layers, 512-d, 8 heads, FFN 2048, 8192 position capacity. 0.53x bert-base.

from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2", trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")

emb = model.encode(["a cat on a windowsill", "a feline by the window"])   # (2, 768)
(emb[0] @ emb[1]).item()

Results

Measured in one harness; every model mean-pooled and L2-normalized, no task tuning. erank is the participation ratio of the embedding spectrum -- how many of the 768 directions are actually used.

model	params	STS-B rho	SICK-R rho	self_cos	erank
bert-base	109.5M	.4729	.5865	+.580	32.0
ModernBERT-base	149.0M	.4215	.5479	+.948	--
roberta-base	124.6M	.5436	.6296	+.976	--
albert-base-v2	11.7M	.4784	.5364	+.905	--
distilbert	66.4M	.5717	.6424	+.840	--
captionbert-8192-v2	58.3M	.5747	.6526	+.129	33.4
all-MiniLM-L6-v2 (ref)	22.7M	.8203	.7758	+.023	94.3

It edges every teacher it was distilled from, at 13% of their combined parameters, having never seen a similarity label. It does not reach all-MiniLM-L6-v2, which was contrastively trained on 1B+ curated pairs -- a different comparison class.

Isotropy is the mechanism. Mean-pooled BERT-family embeddings sit in a narrow cone (self_cos .58-.98); this model reads +.129, and cosine discriminates far better in a space that is not collapsed.

With an AMOE anchor (amoe/)

The trunk is frozen; adapters are 1.6M params each. Anchors ship in this repo under amoe/ and toggle bit-exact -- all anchors disabled reproduces the bare trunk exactly, so one artifact serves both.

config	STS-B rho	SICK-R rho
bare trunk	.5747	.6526
+ equiv anchor (all-nli)	.7254	.7550
+ simplify anchor (wiki/altlex/compression)	.7400	.7075
+ 2-anchor dispatch (MOE)	.7524	.7380

The two anchors are complementary along the task axis, and the router separated them: mean |w/z| moved from .310/.380 (blend) to .645/.223 (specialize) over 800 keys-only steps. See amoe-lora.

How it was built

1. Five teachers embedded 33M CC12M llava-next captions (mean-pooled, 768-d).
2. One global whitened Procrustes map per teacher into bert-base's frame, fit on a stratified random sample and reported out-of-sample.
3. Consensus = normalized centroid of the aligned teachers, per chunk.
4. Student trained from scratch: InfoNCE(T=0.07) + per-sample MSE against the consensus. Pure Adam, no weight decay. 26.9M rows, 52,548 steps at batch 2048, ~5.4 h on one RTX 6000 Pro.

Known limits -- read before using

• Consensus rank is ~28.7 of 768. Five BERT-family teachers only agree on about 29 directions. The student uses ~103 in domain but falls back to ~33 on out-of-domain text: the structure it builds on captions does not transfer. This is the model's ceiling and it is a property of the consensus, not the student.
• Alignment quality varies by teacher. Out-of-sample cosine to the bert frame: distil .625, roberta .372, albert .331, modern .327. The ordering tracks architectural distance from bert-base.
• 10 of 66 source chunks lacked ModernBERT, so 54 chunks (~27M rows) were used. No 4-expert fallback: that would change the target definition mid-dataset.
• Trained on image captions. Expect caption-like text to be its strongest domain.
• Single seed. No variance estimate.

Output convention (differs from v1)

field	shape
last_hidden_state	(B, L, 512)	token states
pooler_output	(B, 768)	the embedding, L2-normalized
embedding	(B, 768)	alias

geolip-captionbert-8192 (v1) returned the pooled embedding as last_hidden_state. If porting v1 code, use pooler_output. v1 also shipped an AlignmentBank; v2 does not -- measured on v1, its expert-consistency features varied 0.2% across samples because a rotation round-trip carries no data.

Citation

@misc{abstractphil2026captionbertv2,
  title  = {captionbert-8192-v2: consensus distillation at CC12M scale},
  author = {AbstractPhil},
  year   = {2026},
  url    = {https://huggingface.co/AbstractPhil/captionbert-8192-v2}
}

MIT.