Question about convert model to ggml

by khanhj - opened Jul 16, 2023

Jul 16, 2023

Hi Symanto,

Thanks for provide this model.
I'm using this method to convert it to ggml: https://github.com/ggerganov/llama.cpp/blob/master/convert.py

This is the error:

params: n_vocab:250002 n_embd:768 n_mult:1536 n_head:12 n_layer:12
Traceback (most recent call last):
  File "/root/github/llama.cpp/convert.py", line 1270, in <module>
    main()
  File "/root/github/llama.cpp/convert.py", line 1262, in main
    output_type = pick_output_type(model, args.outtype)
  File "/root/github/llama.cpp/convert.py", line 1075, in pick_output_type
    wq_type = model["layers.0.attention.wq.weight"].data_type
KeyError: 'layers.0.attention.wq.weight'

May i know what is the weight number of this model?

Thank you in advance.

Lee

abasile

Symanto Research org Aug 1, 2023

Hi @khanhj ,

thanks for the interest in this model. I took a look at the convert.py script that you are using and it seems to work only for specific model. From what I understand, llama.cpp does not currently support sentence transformer models. From a quick search however, I have seen that this library should work in principle with this model. Have you tried it already?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment