VRAM - how to run
#6
by SinanAkkoyun - opened
How do you run inference with your API?
May I ask what quantization your demo inference runs at?
This is an unquantized model. You can estimate the total expected VRAM required by adding the size of each checkpoint file.
To run this on a 24GB GPU, you can try this env var or the 4-bit per weight quantized model I linked in the last comment here: https://huggingface.co/allenai/MolmoE-1B-0924/discussions/4