Thanks for this
#5
by jcbtc - opened
Very high quality tune, thanks for this.
I'm going to mix with rocmfpx for AMD machines and see what it can do!
on 5090:
256k context single slot (q8 kv) mtp n3 146.1 tok/s 29.7GB
32k x 8 context 8 slot q8kv mtp off 269.2 tok/s 27.5GB
on 2x 3090 power limit 80%:
256k context (q8 kv) mtp n3 39.7 tok/s 29.64GB
on 2x 3090 power limit 100%:
256k context (q8 kv) mtp n3 44.34 tok/s 29.64GB
artificial-citizen changed discussion status to closed