Supra-50M-Instruct-GGUF

GGUF quantizations of SupraLabs/Supra-50M-Instruct.

Files

File	Quantization	Size
model.gguf	F16 base GGUF	100M
Q8_0.gguf	Q8_0	54M
Q6_K.gguf	Q6_K	44M
Q4_K_M.gguf	Q4_K_M	36M
Q4_K_S.gguf	Q4_K_S	35M
Q4_0.gguf	Q4_0	33M
Q3_K_S.gguf	Q3_K_S	30M
Q2_K.gguf	Q2_K	28M

Checksums are in SHA256SUMS.

Conversion

model.gguf is the F16 GGUF from the source repository. The quantized files were generated with llama.cpp build b9550 using llama-quantize.

Some K-quant outputs may contain fallback tensor types where tensor dimensions are not divisible by the required K-quant block size. This is normal llama.cpp behavior for this model shape.

Usage

This model was instruction-tuned with the Alpaca prompt format. It is not a ChatML-style multi-turn chat model, so use completion mode and include the prompt template. The GGUF files intentionally do not include tokenizer.chat_template, because llama.cpp chat mode would otherwise try to apply the wrong template.

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
What is the capital of Japan?

### Response:

Example with llama.cpp:

cat > prompt.txt <<'EOF'
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
What is the capital of Japan?

### Response:
EOF

llama-completion \
  -hf di-zhang-fdu/Supra-50M-Instruct-GGUF:Q4_K_M \
  -f prompt.txt \
  -n 128 \
  --temp 0.7 \
  --top-k 50 \
  --top-p 0.9 \
  --repeat-penalty 1.15 \
  -no-cnv

For deterministic checks, use --temp 0 --top-k 1.