File size: 473 Bytes
f4c09a8
378ca54
f4c09a8
 
 
 
 
 
 
 
 
 
378ca54
 
 
 
f4c09a8
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Recipe:
```
  Main model:
  
    Routed Expert Projections - NVFP4 (calibrated, with input scales for A4)
    Attention Projections: BF16
    Shared Experts: BF16
  
  MTP:
    Routed Expert Projections - BF16
    Attention Projections: BF16
    Shared Experts: BF16
```

Should fit on 4x RTX 6000. See the slightly smaller -4p67 model for 2x RTX 6000.

KLD: ~0.04

Evals to follow.

Should be used with https://github.com/local-inference-lab/vllm/tree/dev/jovian-judgement