Instructions to use legraphista/gemma-2-27b-it-IMat-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use legraphista/gemma-2-27b-it-IMat-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="legraphista/gemma-2-27b-it-IMat-GGUF",
	filename="gemma-2-27b-it.BF16/gemma-2-27b-it.BF16-00001-of-00003.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use legraphista/gemma-2-27b-it-IMat-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S
# Run inference directly in the terminal:
llama cli -hf legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S
# Run inference directly in the terminal:
llama cli -hf legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S
# Run inference directly in the terminal:
./llama-cli -hf legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S
# Run inference directly in the terminal:
./build/bin/llama-cli -hf legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S

Use Docker

docker model run hf.co/legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S

LM Studio
Jan

vLLM

How to use legraphista/gemma-2-27b-it-IMat-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "legraphista/gemma-2-27b-it-IMat-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "legraphista/gemma-2-27b-it-IMat-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S

Ollama
How to use legraphista/gemma-2-27b-it-IMat-GGUF with Ollama:
```
ollama run hf.co/legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S
```

Unsloth Studio

How to use legraphista/gemma-2-27b-it-IMat-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for legraphista/gemma-2-27b-it-IMat-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for legraphista/gemma-2-27b-it-IMat-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for legraphista/gemma-2-27b-it-IMat-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use legraphista/gemma-2-27b-it-IMat-GGUF with Docker Model Runner:
```
docker model run hf.co/legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S
```

Lemonade

How to use legraphista/gemma-2-27b-it-IMat-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull legraphista/gemma-2-27b-it-IMat-GGUF:Q4_K_S

Run and chat with the model

lemonade run user.gemma-2-27b-it-IMat-GGUF-Q4_K_S

List all available models

lemonade list

legraphista commited on Jun 30, 2024

Commit

6799469

verified ·

1 Parent(s): becc67c

Upload imatrix.log with huggingface_hub

Browse files

Files changed (1) hide show

imatrix.log +36 -34

imatrix.log CHANGED Viewed

@@ -1,4 +1,4 @@
-llama_model_loader: loaded meta data with 26 key-value pairs and 508 tensors from gemma-2-27b-it-IMat-GGUF/gemma-2-27b-it.Q8_0.gguf.hardlink.gguf (version GGUF V3 (latest))
 llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
 llama_model_loader: - kv   0:                       general.architecture str              = gemma2
 llama_model_loader: - kv   1:                               general.name str              = gemma-2-27b-it
@@ -12,20 +12,22 @@ llama_model_loader: - kv   8:    gemma2.attention.layer_norm_rms_epsilon f32
 llama_model_loader: - kv   9:                gemma2.attention.key_length u32              = 128
 llama_model_loader: - kv  10:              gemma2.attention.value_length u32              = 128
 llama_model_loader: - kv  11:                          general.file_type u32              = 7
-llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
-llama_model_loader: - kv  13:                         tokenizer.ggml.pre str              = default
-llama_model_loader: - kv  14:                      tokenizer.ggml.tokens arr[str,256000]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
-llama_model_loader: - kv  15:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...
-llama_model_loader: - kv  16:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, ...
-llama_model_loader: - kv  17:                tokenizer.ggml.bos_token_id u32              = 2
-llama_model_loader: - kv  18:                tokenizer.ggml.eos_token_id u32              = 1
-llama_model_loader: - kv  19:            tokenizer.ggml.unknown_token_id u32              = 3
-llama_model_loader: - kv  20:            tokenizer.ggml.padding_token_id u32              = 0
-llama_model_loader: - kv  21:               tokenizer.ggml.add_bos_token bool             = true
-llama_model_loader: - kv  22:               tokenizer.ggml.add_eos_token bool             = false
-llama_model_loader: - kv  23:                    tokenizer.chat_template str              = {{ bos_token }}{% if messages[0]['rol...
-llama_model_loader: - kv  24:            tokenizer.ggml.add_space_prefix bool             = false
-llama_model_loader: - kv  25:               general.quantization_version u32              = 2
 llama_model_loader: - type  f32:  185 tensors
 llama_model_loader: - type q8_0:  323 tensors
 llm_load_vocab: special tokens cache size = 261
@@ -100,45 +102,45 @@ llama_new_context_with_model: KV self size  =  184.00 MiB, K (f16):   92.00 MiB,
 llama_new_context_with_model:  CUDA_Host  output buffer size =     0.98 MiB
 llama_new_context_with_model:      CUDA0 compute buffer size =  1704.31 MiB
 llama_new_context_with_model:  CUDA_Host compute buffer size =    10.01 MiB
-llama_new_context_with_model: graph nodes  = 1709
 llama_new_context_with_model: graph splits = 121
 system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
 compute_imatrix: tokenizing the input ..
-compute_imatrix: tokenization took 102.378 ms
 compute_imatrix: computing over 128 chunks with batch_size 512
 compute_imatrix: 1.92 seconds per pass - ETA 4.08 minutes
-[1]39.6511,[2]16.5757,[3]13.6939,[4]15.6944,[5]16.8349,[6]17.6097,[7]19.6449,[8]21.2138,[9]22.7663,
 save_imatrix: stored collected data after 10 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[10]19.5954,[11]18.7601,[12]20.1620,[13]21.1839,[14]21.3214,[15]22.4495,[16]22.5613,[17]22.7834,[18]23.9931,[19]23.7468,
 save_imatrix: stored collected data after 20 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[20]23.8365,[21]26.0599,[22]25.6103,[23]25.1318,[24]25.6891,[25]25.7517,[26]25.1686,[27]25.8776,[28]26.5172,[29]26.5127,
 save_imatrix: stored collected data after 30 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[30]27.2144,[31]24.9851,[32]23.7414,[33]22.8572,[34]22.1536,[35]21.5105,[36]22.1975,[37]23.0630,[38]23.1560,[39]23.3739,
 save_imatrix: stored collected data after 40 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[40]23.4627,[41]23.6640,[42]25.0051,[43]25.9131,[44]26.9078,[45]27.6585,[46]27.0582,[47]26.4096,[48]26.8484,[49]27.2839,
 save_imatrix: stored collected data after 50 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[50]26.9025,[51]26.6051,[52]26.6338,[53]27.0460,[54]27.5828,[55]28.1862,[56]28.5826,[57]28.5708,[58]28.5400,[59]28.0093,
 save_imatrix: stored collected data after 60 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[60]27.6445,[61]27.2285,[62]27.0626,[63]27.1479,[64]27.4686,[65]27.1084,[66]27.0599,[67]27.0251,[68]26.9609,[69]26.8687,
 save_imatrix: stored collected data after 70 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[70]26.7705,[71]26.7926,[72]26.7099,[73]26.7881,[74]26.7220,[75]26.5072,[76]26.5858,[77]26.7008,[78]26.6471,[79]26.4994,
 save_imatrix: stored collected data after 80 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[80]26.6450,[81]26.7715,[82]26.8064,[83]27.0250,[84]27.0770,[85]26.6203,[86]26.4729,[87]26.2858,[88]26.3698,[89]26.3433,
 save_imatrix: stored collected data after 90 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[90]26.5397,[91]26.5333,[92]26.4208,[93]26.3047,[94]26.1210,[95]26.0503,[96]25.8447,[97]25.8144,[98]25.7297,[99]25.7554,
 save_imatrix: stored collected data after 100 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[100]25.6738,[101]25.8975,[102]26.0385,[103]26.1709,[104]26.5591,[105]26.8560,[106]26.8766,[107]26.8606,[108]26.7007,[109]26.6811,
 save_imatrix: stored collected data after 110 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[110]26.4028,[111]26.0091,[112]25.5502,[113]25.6821,[114]25.7196,[115]25.6433,[116]25.5595,[117]25.5877,[118]25.6440,[119]25.6503,
 save_imatrix: stored collected data after 120 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-[120]25.6006,[121]25.5261,[122]25.4242,[123]25.4475,[124]25.6331,[125]25.8680,[126]26.1323,[127]26.2467,[128]26.3842,
 save_imatrix: stored collected data after 128 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
-llama_print_timings:        load time =    4083.77 ms
 llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
-llama_print_timings: prompt eval time =  232410.77 ms / 65536 tokens (    3.55 ms per token,   281.98 tokens per second)
 llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
-llama_print_timings:       total time =  236107.85 ms / 65537 tokens
-Final estimate: PPL = 26.3842 +/- 0.60805

+llama_model_loader: loaded meta data with 28 key-value pairs and 508 tensors from gemma-2-27b-it-IMat-GGUF/gemma-2-27b-it.Q8_0.gguf.hardlink.gguf (version GGUF V3 (latest))
 llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
 llama_model_loader: - kv   0:                       general.architecture str              = gemma2
 llama_model_loader: - kv   1:                               general.name str              = gemma-2-27b-it
 llama_model_loader: - kv   9:                gemma2.attention.key_length u32              = 128
 llama_model_loader: - kv  10:              gemma2.attention.value_length u32              = 128
 llama_model_loader: - kv  11:                          general.file_type u32              = 7
+llama_model_loader: - kv  12:              gemma2.attn_logit_softcapping f32              = 50.000000
+llama_model_loader: - kv  13:             gemma2.final_logit_softcapping f32              = 30.000000
+llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = llama
+llama_model_loader: - kv  15:                         tokenizer.ggml.pre str              = default
+llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,256000]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
+llama_model_loader: - kv  17:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...
+llama_model_loader: - kv  18:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, ...
+llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 2
+llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 1
+llama_model_loader: - kv  21:            tokenizer.ggml.unknown_token_id u32              = 3
+llama_model_loader: - kv  22:            tokenizer.ggml.padding_token_id u32              = 0
+llama_model_loader: - kv  23:               tokenizer.ggml.add_bos_token bool             = true
+llama_model_loader: - kv  24:               tokenizer.ggml.add_eos_token bool             = false
+llama_model_loader: - kv  25:                    tokenizer.chat_template str              = {{ bos_token }}{% if messages[0]['rol...
+llama_model_loader: - kv  26:            tokenizer.ggml.add_space_prefix bool             = false
+llama_model_loader: - kv  27:               general.quantization_version u32              = 2
 llama_model_loader: - type  f32:  185 tensors
 llama_model_loader: - type q8_0:  323 tensors
 llm_load_vocab: special tokens cache size = 261
 llama_new_context_with_model:  CUDA_Host  output buffer size =     0.98 MiB
 llama_new_context_with_model:      CUDA0 compute buffer size =  1704.31 MiB
 llama_new_context_with_model:  CUDA_Host compute buffer size =    10.01 MiB
+llama_new_context_with_model: graph nodes  = 1850
 llama_new_context_with_model: graph splits = 121
 system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
 compute_imatrix: tokenizing the input ..
+compute_imatrix: tokenization took 94.256 ms
 compute_imatrix: computing over 128 chunks with batch_size 512
 compute_imatrix: 1.92 seconds per pass - ETA 4.08 minutes
+[1]12.2429,[2]6.2081,[3]5.2588,[4]6.2085,[5]6.7166,[6]7.2390,[7]7.6746,[8]8.1610,[9]8.5380,
 save_imatrix: stored collected data after 10 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[10]7.7409,[11]7.6231,[12]8.3069,[13]8.8175,[14]8.9820,[15]9.5963,[16]9.7448,[17]9.8649,[18]10.2368,[19]10.1102,
 save_imatrix: stored collected data after 20 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[20]10.2363,[21]10.9791,[22]10.9424,[23]10.8230,[24]11.0184,[25]10.9711,[26]10.8238,[27]11.0156,[28]11.1952,[29]11.2073,
 save_imatrix: stored collected data after 30 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[30]11.4849,[31]10.7086,[32]10.2419,[33]9.8871,[34]9.6029,[35]9.3962,[36]9.5205,[37]9.7559,[38]9.8816,[39]10.0292,
 save_imatrix: stored collected data after 40 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[40]10.1462,[41]10.2011,[42]10.6384,[43]10.9116,[44]11.2397,[45]11.4389,[46]11.2550,[47]11.0805,[48]11.2637,[49]11.4301,
 save_imatrix: stored collected data after 50 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[50]11.2782,[51]11.1894,[52]11.2327,[53]11.4036,[54]11.5967,[55]11.8117,[56]11.9157,[57]11.9185,[58]11.9380,[59]11.7810,
 save_imatrix: stored collected data after 60 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[60]11.6650,[61]11.5287,[62]11.4071,[63]11.4760,[64]11.5875,[65]11.4612,[66]11.4694,[67]11.4358,[68]11.4182,[69]11.3768,
 save_imatrix: stored collected data after 70 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[70]11.3167,[71]11.3025,[72]11.2862,[73]11.3380,[74]11.2888,[75]11.1897,[76]11.1629,[77]11.1599,[78]11.1381,[79]11.0667,
 save_imatrix: stored collected data after 80 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[80]11.1144,[81]11.1690,[82]11.1841,[83]11.2762,[84]11.2937,[85]11.1199,[86]11.0622,[87]10.9503,[88]10.9787,[89]10.9774,
 save_imatrix: stored collected data after 90 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[90]11.0439,[91]11.0167,[92]10.9757,[93]10.9302,[94]10.8609,[95]10.8306,[96]10.7705,[97]10.7326,[98]10.6794,[99]10.7099,
 save_imatrix: stored collected data after 100 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[100]10.7050,[101]10.8097,[102]10.8830,[103]10.9458,[104]11.0763,[105]11.1774,[106]11.1839,[107]11.1889,[108]11.1470,[109]11.1662,
 save_imatrix: stored collected data after 110 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[110]11.0690,[111]10.9087,[112]10.7355,[113]10.7965,[114]10.8346,[115]10.8238,[116]10.8008,[117]10.8403,[118]10.8670,[119]10.8845,
 save_imatrix: stored collected data after 120 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+[120]10.8804,[121]10.8749,[122]10.8379,[123]10.8537,[124]10.9275,[125]11.0106,[126]11.0989,[127]11.1369,[128]11.1788,
 save_imatrix: stored collected data after 128 chunks in gemma-2-27b-it-IMat-GGUF/imatrix.dat
+llama_print_timings:        load time =    4035.26 ms
 llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_print_timings: prompt eval time =  226486.47 ms / 65536 tokens (    3.46 ms per token,   289.36 tokens per second)
 llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
+llama_print_timings:       total time =  230157.77 ms / 65537 tokens
+Final estimate: PPL = 11.1788 +/- 0.20186