semancer-27b / app.py

Commit History

torchao int8 weight-only + flash-linear-attention (bnb int8 was ~4 tok/s)
3ebe40d
Running
verified

UnstableLlama commited on

int8 (bitsandbytes) on large slice; scale GPU duration to max_new_tokens
a517d99
verified

UnstableLlama commited on