Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
SeaWolf-AI 
posted an update 2 days ago
Post
4766
A small gift for anyone building or studying foundation models.

Most "open" models hand you the weights and stop there. With Aether-7B-5Attn we wanted to hand over the whole thing — so you can actually learn from it, reproduce it, and build on it: the data recipe, the training code, every hyperparameter, the complete logs, and the intermediate checkpoints. All Apache-2.0, reproducible byte-for-byte.

What you can do with it:
🔁 Rebuild it from scratch, or fork the recipe for your own model
🔬 Study a real heterogeneous-attention MoE — 49 layers place 5 attention mechanisms on a 7×7 Latin square, arranged as a clean, attributable ablation
📈 Trace training dynamics across the released checkpoints (110k / 115k / 162k)

It's a modest 6.59B model, and an honest one — the limitations (no KV-cache in this build, small scale) are written right in the card. We're not claiming it's special. If any piece of it saves you time or teaches you something, that's exactly what we hoped for. 🤗

📖 Full write-up →
[blog] · https://huggingface.co/blog/FINAL-Bench/opensource-llm
📦 5 Attention Base · FINAL-Bench/Aether-7B-5Attn
🎯 5 Attention Instruct · FINAL-Bench/Aether-7B-5Attn-it
🚀 5 Attention Live demo · FINAL-Bench/Aether-Sovereign-AI
📦 7 Attention Base · https://huggingface.co/FINAL-Bench/Aether-7B-7Attn-base
📦 11 Attention Base · FINAL-Bench/Aether-6B-11Attn-base
🧬 Collection · https://huggingface.co/collections/FINAL-Bench/aether-foundation-model

#opensource #LLM #MoE #reproducibility #Apache2
dipankarsarkar

The logs are the part I'd point people to. Two real launcher logs in-repo, the multi-node one alone is ~946KB, plus run_s1/run_s2 and the eval harness sitting right next to the weights. That's the half of "open" most releases quietly skip.

One thing worth checking, because it cuts against the pitch.

The base repo is gated (gated: auto). The instruct one isn't.

So the artifact the post is actually about, the one you rebuild from scratch and trace the 110k/115k/162k checkpoints through, is the one sitting behind an access gate. Pulling Aether-7B-5Attn without access comes back "Access is restricted", and /refs 403s even for a logged-in stranger, so you can't enumerate the checkpoint branches either. Aether-7B-5Attn-it resolves fine.

Both cards say apache-2.0 and the base is tagged fully-open, so I read it as a switch left flipped rather than intent.

Is the gate on the base deliberate?

·
SeaWolf-AI

You're right — good catch, and thank you for actually testing it instead of assuming.

It was a leftover switch, not intent. The base was gated during the private staging phase, and when we flipped everything public that one toggle didn't come along with it. Nothing in the license or the release plan ever called for a gate there.

It's fixed now. Aether-7B-5Attn is ungated — and I verified it from a logged-out client rather than trusting the settings page: anonymous file pulls return 200, and /refs resolves again.

One clarification while I'm here: the 110k / 115k / 162k checkpoints aren't branches of the base repo — they're folders in a separate public dataset, FINAL-Bench/Aether-7B-5Attn-checkpoints (~13 GB each, ungated). So enumerating refs was always going to be a dead end for those, quite apart from the 403.

And thank you for the point about the logs — that's the part we most hoped people would open rather than star. The multi-node run is messy in places, which is rather the point: it's the actual run, not a tidied-up excerpt.
dipankarsarkar

Confirmed from my side too, logged out: gated is false, /refs comes back 200, and an anonymous pull of config.json is a 200. That was quick.

And you are right about the checkpoints, that one was my error. They are a separate public dataset and your model card already links it. I went at the repo and skipped the card, which is a bit funny given what I was complaining about.

The part I keep chewing on is that the gate was live while the card said fully-open.

Both were true statements about the same repo at the same time. The tag is a claim, gated:auto is a fact, and nothing on the platform reconciles the two. They sat next to each other for the whole staging window and the only thing that surfaced the gap was a stranger running a pull.

That one is mechanically checkable, not a judgement call. A card asserting open access against a repo that 403s anonymously is a diff you can compute.

Would you run that check on your own releases if it existed?

·
SeaWolf-AI

Thanks for checking from your side — good to have it confirmed by someone logged out rather than just by us.

And no need to worry about the checkpoint part. Easy thing to miss.

The part you keep chewing on is the part worth keeping. The tag was a claim, gated: auto was a fact, and both sat in the same repo for the whole staging window with nothing reconciling them. We didn't catch it either — an anonymous pull from a stranger did.

To answer your question directly: yes. We already run a version of it before release — a logged-out pass that checks anonymous access against what the card claims, along with links and wording. It caught a few leftover lines on this release. It exists because of exactly the failure you're describing, and it's a small script rather than anything clever.

Which is sort of your point. If a few lines can compute that diff, it probably shouldn't be something each team rebuilds on its own.

Thanks again for pulling the repo in the first place.
dipankarsarkar

Agreed, and the layer it belongs on is the giveaway. Your script runs at release time, inside your process. The gap it catches is a property of the public repo, so the check wants to live where anyone can run it against anyone, not just the team that shipped it.

Smallest useful version: a logged-out probe that reads the card's access claim, hits /refs and one file anonymously, and fails if the tag says open while the pull says 403. No auth, no secrets, a plain CI step that runs on any repo.

If that existed as a shared action, would you wire it into your release, or would you want it running continuously to catch a repo that gets re-gated after launch?
In this post