🤏 quecto mode

appvoid

59 5 192

https://ko-fi.com/appvoid

AI & ML interests

singularity through byte-level tokens and small language models - creating applications and ideas out of the void

Recent Activity

liked a model about 20 hours ago

bench-labs/cagliostro-v3

updated a collection 1 day ago

datasets out there

liked a dataset 1 day ago

allenai/dolmino-mix-1124

View all activity

Organizations

Posts 48

Post

103

LLMS for edge devices?
What if we go the reliable route instead of the speed route?
What if we make it run on sbcs with few megabytes available?

That's the idea for the next model.

Keep in tune.

Post

123

We trained a 10.9M byte-level recurrent Transformer on L3 and L6. (Loop 3 and Loop 6)

Yet L4/L5 improved too, L8 held up, and the L3→L6 gain grew during training.

Same weights. More compute. Better predictions.

This is a new architecture for effective compute after several steps beyond original training!

We mixed and matched components like time and mhc into an ouro-like byte-level language model and the result is BET, a byte-level step-elastic transformer that can run computation steps without significant degradation.

One of the coolest parts of this training was discovering how Gradient Descent decided to use the first layer as what we would consider a scratchpad! Totally destroyed for the decoder but somehow makes total sense for the next layer!

I believe looped-transformers are the future of edge computing and this is a first step towards it.

Blogpost: https://medium.com/@appvoidofficial/byte-level-elasticity-182fe2ed1d2f

appvoid/bet-10m