A bit slower but good nvfp4
Pretty good but sadly losing ~20tokens per second compared to base 27b fp8 for some reason. Was getting pretty concise thinking otherwise though!
Thanks for the feedback — really glad the thinking came out concise and the quality held up!
The ~20 tok/s vs the base FP8 is expected, and it's a deliberate tradeoff. This is a hybrid model — 48 of its 64 layers are GatedDeltaNet ("Mamba") linear-attention layers, and I kept those in BF16 because they're quantization-sensitive and that's what preserves the reasoning quality + concise behavior. Only the full-attention q/k/v/o and the MLPs are 4-bit (NVFP4).
The side effect: roughly two-thirds of the per-token weight bytes are still BF16 (~18 GB BF16 vs ~9 GB of 4-bit), so the memory footprint is close to an 8-bit build — and since this is NVFP4A16 (weight-only, BF16 activations) it doesn't get the FP8 tensor-core compute path that a W8A8/FP8 build does. So base-FP8 ends up a bit faster on both axes.
I can quantize the Mamba layers too — that shrinks it and speeds up the memory-bound decode — but accuracy would likely take a hit, which is exactly what I was trying to avoid here. If it's useful I can also put up an FP8 (W8A8) build, which should match the base FP8 speed while keeping ThinkingCap's brevity. Happy to spin that up if you'd want to try it.
(I've also added these details to the model card.)