File size: 473 Bytes
f4c09a8 378ca54 f4c09a8 378ca54 f4c09a8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | Recipe:
```
Main model:
Routed Expert Projections - NVFP4 (calibrated, with input scales for A4)
Attention Projections: BF16
Shared Experts: BF16
MTP:
Routed Expert Projections - BF16
Attention Projections: BF16
Shared Experts: BF16
```
Should fit on 4x RTX 6000. See the slightly smaller -4p67 model for 2x RTX 6000.
KLD: ~0.04
Evals to follow.
Should be used with https://github.com/local-inference-lab/vllm/tree/dev/jovian-judgement |