Instructions to use qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M

Use Docker

docker model run hf.co/qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M

Ollama
How to use qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF with Ollama:
```
ollama run hf.co/qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF with Docker Model Runner:
```
docker model run hf.co/qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M
```

Lemonade

How to use qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull qwp4w3hyb/Yi-1.5-34B-Chat-16K-iMat-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Yi-1.5-34B-Chat-16K-iMat-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

qwp4w3hyb commited on May 23, 2024

Commit

f46713d

verified ·

1 Parent(s): 0be659e

Upload folder using huggingface_hub

Browse files

Files changed (22) hide show

.gitattributes +21 -0
imat-bf16-gmerged.dat +3 -0
yi-1.5-34b-chat-16k-bf16.split-00001-of-00002.gguf +3 -0
yi-1.5-34b-chat-16k-bf16.split-00002-of-00002.gguf +3 -0
yi-1.5-34b-chat-16k-imat-IQ1_S.gguf +3 -0
yi-1.5-34b-chat-16k-imat-IQ2_M.gguf +3 -0
yi-1.5-34b-chat-16k-imat-IQ2_S.gguf +3 -0
yi-1.5-34b-chat-16k-imat-IQ2_XS.gguf +3 -0
yi-1.5-34b-chat-16k-imat-IQ2_XXS.gguf +3 -0
yi-1.5-34b-chat-16k-imat-IQ3_M.gguf +3 -0
yi-1.5-34b-chat-16k-imat-IQ3_S.gguf +3 -0
yi-1.5-34b-chat-16k-imat-IQ3_XS.gguf +3 -0
yi-1.5-34b-chat-16k-imat-IQ3_XXS.gguf +3 -0
yi-1.5-34b-chat-16k-imat-IQ4_NL.gguf +3 -0
yi-1.5-34b-chat-16k-imat-IQ4_XS.gguf +3 -0
yi-1.5-34b-chat-16k-imat-Q4_0.gguf +3 -0
yi-1.5-34b-chat-16k-imat-Q4_K_M.gguf +3 -0
yi-1.5-34b-chat-16k-imat-Q4_K_S.gguf +3 -0
yi-1.5-34b-chat-16k-imat-Q5_K_M.gguf +3 -0
yi-1.5-34b-chat-16k-imat-Q5_K_S.gguf +3 -0
yi-1.5-34b-chat-16k-imat-Q6_K.gguf +3 -0
yi-1.5-34b-chat-16k-imat-Q8_0.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,24 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+imat-bf16-gmerged.dat filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-bf16.split-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-bf16.split-00002-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+yi-1.5-34b-chat-16k-imat-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

imat-bf16-gmerged.dat ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:22ca36c936869306c78615e0ed921f017b45bdcc710d98acb22f0e92825e8a88
+size 15251093

yi-1.5-34b-chat-16k-bf16.split-00001-of-00002.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d68ed6d731baa2dd765c5626aa081fd0490138b2dd73c1fc5a7b6f389308afed
+size 35389566208

yi-1.5-34b-chat-16k-bf16.split-00002-of-00002.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:90c301c30c90974b27529bc78a3e6a87b533849135aacf8dd8d2cced889c2571
+size 33391513152

yi-1.5-34b-chat-16k-imat-IQ1_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5df614df7d2f265b430da4ede08b3bdbba37c0436917ccf23dcfe1e4eeb89e87
+size 7498980256

yi-1.5-34b-chat-16k-imat-IQ2_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b9ed266797d2b11e24f8d141944794826701d951da7baf94c97f4783d59cbede
+size 11794762656

yi-1.5-34b-chat-16k-imat-IQ2_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b5c281eb50367e336b1bd26f7b5d4039efc03425f59e47566584f4c82ec0c6d2
+size 10891021216

yi-1.5-34b-chat-16k-imat-IQ2_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e0c99efde496e5b1c0b4ab9135b10329d6ca7ef0c280d22fcd985b3a90cf1ce2
+size 10306542496

yi-1.5-34b-chat-16k-imat-IQ2_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fd64aac265e13916cad5093cd188e99f80e2355eec12dc533f0ecb3be1847e7f
+size 9306463136

yi-1.5-34b-chat-16k-imat-IQ3_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:32ce719a7bef19137dcedf365a64f77466248c5a193298ad28a3420612625e37
+size 15564700576

yi-1.5-34b-chat-16k-imat-IQ3_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:06bf2b4f4edce3b431770ae0f86fdf50afc4c995fdf1923d3cd33493cee515bc
+size 15018785696

yi-1.5-34b-chat-16k-imat-IQ3_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9ed180c95f2182b78b8824cfded1ed461873346882ceaf787f8fa26c6f88134e
+size 14234319776

yi-1.5-34b-chat-16k-imat-IQ3_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a6ff6b20b5f1e2b6e0fa8aca7baf0879fd32ab8a83a4dcae1d89ba47fb7956bd
+size 13333875616

yi-1.5-34b-chat-16k-imat-IQ4_NL.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9d4792d7c75ab51aa7d528b646d98c07af273d67a2488fcbccbb3470d38d1271
+size 19521579936

yi-1.5-34b-chat-16k-imat-IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:05f505d292dcc939fa544011e3dd69f3d75754df9469f45a5cfef3cbca1d0d79
+size 18475051936

yi-1.5-34b-chat-16k-imat-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d2a15f84ebdc1ad7b8a056b9e632785f678c6cea7c361f3051749e35627ed5cf
+size 19530754976

yi-1.5-34b-chat-16k-imat-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:45e20ddc1c44e2d698599b9d9c6ac7b383a59544ef6c593692965918abb55543
+size 20658711456

yi-1.5-34b-chat-16k-imat-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7bb5e5ed2c3b9893141ebe990e7f5842038bff35d438ad7706d7a06d9dc39fcd
+size 19598650272

yi-1.5-34b-chat-16k-imat-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:53a4e5b264a52384b580a8027a0390e1b088832777893c21a9be7279a1d0dcfe
+size 24321846176

yi-1.5-34b-chat-16k-imat-Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:46dfc46fbd82099f378b7a45e4dabbf2347ca230e2281b8adcce2cf5471fe0d0
+size 23707691936

yi-1.5-34b-chat-16k-imat-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a8fa5fca8f3a6735b92a55bfb69114bca12c9f86417bcee815007230e8bbb1c6
+size 28213926816

yi-1.5-34b-chat-16k-imat-Q8_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:69ccbd2ef1259016c05206a12529becb2422341abdc8022f6118972d839f4adf
+size 36542282656