benxh

/

Qwen2.5-VL-7B-Instruct-GGUF

Like 7

Any update on this?

FYI, both llama-llava-cli and llama-cli make it produce gibberish:

llama-cli -c 0  --temp 0.2 -m Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf  -p "Provide a full description." 
...
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant


system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 | 

main: interactive mode on.
sampler seed: 1971700995
sampler params: 
    repeat_last_n = 64, repeat_penalty = 1,000, frequency_penalty = 0,000, presence_penalty = 0,000
    dry_multiplier = 0,000, dry_base = 1,750, dry_allowed_length = 2, dry_penalty_last_n = 128000
    top_k = 40, top_p = 0,950, min_p = 0,050, xtc_probability = 0,000, xtc_threshold = 0,100, typical_p = 1,000, temp = 0,200
    mirostat = 0, mirostat_lr = 0,100, mirostat_ent = 5,000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist 
generate: n_ctx = 128000, n_batch = 2048, n_predict = -1, n_keep = 0

== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to the AI.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.

system
Provide a full description.

> Describe yourself and your goals
 1141:14:44:14:114 *1:41:44 *1:4 A4 A4:1:4 A. *4: * A: * A4:4 * * A: * * * * * A:4: * * * * * * * * * * A:4 * A * * 1: * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * A * * * * * A * * * * A * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * A * * * * * * * * * * * A * * * * * * A * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * A * * * * * * * * * * * * * * * * * * A * * * * * * * * * * * * * * * * * * * * * * * * * * * * *^C^C^C^C^C^C

Ref, hand-compiled:

llama-cli --version
version: 4591 (7919256c)
built with Ubuntu clang version 18.1.8 

works as a regular language model but no vision support(LM Studio)

Do we need a mmproj file to enable the vision? Was this split out of the model when converted to gguf?

For me both my bot this quant generate gibberish.
I assume the vision adapter extractor (..._surgery...py) is not implemented (or not correctly) in llama.cpp (not even tried as the language part not working - at least for me -)
so the vision adapters not extracted.

This was a quick and dirty conversion, mmproj is missing, and it only works properly as a LM not a VLM. Await a better version, with official support