Which litert-lm version are you using in linux ?

by glococo - opened Jun 24

Jun 24

github release 0.13.1 do not support --convert_weights_on_gpu=false
I could only run it:
litert-lm run --max-num-tokens 4096 --backend gpu gemma-4-12B-it
or a little bit more on a 3090.

whhone

LiteRT Community (FKA TFLite) org Jun 25

Hi @glococo , can you clarify more? Why we would like to set convert_weights_on_gpu.

glococo

Jun 25

Hi @whhone , the model card say: "The Linux GPU memory usage number was measured with --disable_cache=true --convert_weights_on_gpu=false. Otherwise it will be about 6GB higher."
But I seems that command option not supported in v0.13.1

whhone

LiteRT Community (FKA TFLite) org Jun 26

I think that benchmark is probably using the https://github.com/google-ai-edge/LiteRT-LM/blob/main/runtime/engine/litert_lm_advanced_main.cc.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment