Instructions to use TheBloke/dolphin-2_2-yi-34b-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/dolphin-2_2-yi-34b-GGUF with Transformers:

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("TheBloke/dolphin-2_2-yi-34b-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use TheBloke/dolphin-2_2-yi-34b-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M

Use Docker

docker model run hf.co/TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use TheBloke/dolphin-2_2-yi-34b-GGUF with Ollama:
```
ollama run hf.co/TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use TheBloke/dolphin-2_2-yi-34b-GGUF with Docker Model Runner:
```
docker model run hf.co/TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M
```

Lemonade

How to use TheBloke/dolphin-2_2-yi-34b-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TheBloke/dolphin-2_2-yi-34b-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.dolphin-2_2-yi-34b-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

TheBloke commited on Nov 14, 2023

Commit

e2b81cf

1 Parent(s): 20692fc

Update BOS to newline to improve generation (GGUF made with llama.cpp commit bd90eca)

Browse files

As per Kerfuffle's recommendation in https://huggingface.co/TheBloke/Nous-Capybara-34B-GGUF/discussions/1

Files changed (12) hide show

dolphin-2_2-yi-34b.Q2_K.gguf +1 -1
dolphin-2_2-yi-34b.Q3_K_L.gguf +1 -1
dolphin-2_2-yi-34b.Q3_K_M.gguf +1 -1
dolphin-2_2-yi-34b.Q3_K_S.gguf +1 -1
dolphin-2_2-yi-34b.Q4_0.gguf +1 -1
dolphin-2_2-yi-34b.Q4_K_M.gguf +1 -1
dolphin-2_2-yi-34b.Q4_K_S.gguf +1 -1
dolphin-2_2-yi-34b.Q5_0.gguf +1 -1
dolphin-2_2-yi-34b.Q5_K_M.gguf +1 -1
dolphin-2_2-yi-34b.Q5_K_S.gguf +1 -1
dolphin-2_2-yi-34b.Q6_K.gguf +1 -1
dolphin-2_2-yi-34b.Q8_0.gguf +1 -1

dolphin-2_2-yi-34b.Q2_K.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3035021ddfb63e45d152b9bded7450a0f16681d8b1ff45d3ea17bbc1f9e5bda6
 size 14555875264

 version https://git-lfs.github.com/spec/v1
+oid sha256:ae509b6384c83eb55caffd9b533d4e467dfabef84ba602d1b81282141fba92ca
 size 14555875264

dolphin-2_2-yi-34b.Q3_K_L.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:630d01eeea904b8ac8d3c3cb7a1219f7512be06acebfb46a28683290bb65de97
 size 18139445184

 version https://git-lfs.github.com/spec/v1
+oid sha256:d15f01e119506c05b477e7ca90834e5a8c9602b97731531c1332df4b5c100e02
 size 18139445184

dolphin-2_2-yi-34b.Q3_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:247e1dbf921ead33041cf44f5c9eed519e1705aa6853fe80a70f71d1e5be26e4
 size 16636573632

 version https://git-lfs.github.com/spec/v1
+oid sha256:704bfb6b38b203d093e722a8075c32512661d0163e5ca8d96518560504e460ee
 size 16636573632

dolphin-2_2-yi-34b.Q3_K_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:11e43b7b3a7237c878d20df19f5fdc2fa68833913883757ea5efea88ee07984b
 size 14960293824

 version https://git-lfs.github.com/spec/v1
+oid sha256:dd5f93fdcd7acea1b29aa846eed3515c468771aeea4649dcf1b088589266e125
 size 14960293824

dolphin-2_2-yi-34b.Q4_0.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1430c37869acdc21117fe6c7e7aef209d446fccd8eb381e06aa4452c9fad889e
 size 19466528704

 version https://git-lfs.github.com/spec/v1
+oid sha256:eed22a4e87efd8a048bca99405fd6ccc04ec7c55099f09384ec16bfecfa4b013
 size 19466528704

dolphin-2_2-yi-34b.Q4_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bdffc6a6be260e500f5b318b27036c7379c2870ed925a844b22bdaccf150d792
 size 20658710464

 version https://git-lfs.github.com/spec/v1
+oid sha256:54b31b580ce0ff3813db91490897360727d524529875c1e1a01f687f91bf3678
 size 20658710464

dolphin-2_2-yi-34b.Q4_K_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95e56c0860bf47be7051189139cf30147a9380b1aab03cfa65968f84aa359d43
 size 19543599040

 version https://git-lfs.github.com/spec/v1
+oid sha256:2f15d38ba4336789f8a4dbdbc0c8ac4eb90e63324b183bb4d3bafbc551305584
 size 19543599040

dolphin-2_2-yi-34b.Q5_0.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:750c32b68df428f14e257f51ebe116062324f92b4da5128fe4f4597e71dbd9bb
 size 23707690944

 version https://git-lfs.github.com/spec/v1
+oid sha256:f6f42e731fbd2f7f4319cab8e036d8336dd41bc8bfd6787b31f74050536a2c5c
 size 23707690944

dolphin-2_2-yi-34b.Q5_K_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1a2efce79789e1af69a298282df189459099ba2666e961c1b70144cfc55c0aa0
 size 24321845184

 version https://git-lfs.github.com/spec/v1
+oid sha256:43575ab5b287f3fd4dd7cfd52563924f5fcbe773078814d026f05fe4b2e6ae3d
 size 24321845184

dolphin-2_2-yi-34b.Q5_K_S.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:227e5951202bff11f4d210eef2afb6b690f373fb4b1ff8ce94b83e2209d37820
 size 23707690944

 version https://git-lfs.github.com/spec/v1
+oid sha256:7766f11bdd72cec9be7d5605b32cb6ac7231607991975ea5389998c8975ceb88
 size 23707690944

dolphin-2_2-yi-34b.Q6_K.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7a8abc87602ab7baeec809f4bf77544a549a2eecc8577b8243d967ca3308c32b
 size 28213925824

 version https://git-lfs.github.com/spec/v1
+oid sha256:3c6da91edd44aa48609859fc5250813ef1d4faa40ea1fb18d4f95f9e04700840
 size 28213925824

dolphin-2_2-yi-34b.Q8_0.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2f2d00d3cfa7f0a2ebc1d164c127dbbd48e24323b935cefeb934bcc75489eba9
 size 36542281664

 version https://git-lfs.github.com/spec/v1
+oid sha256:b32e812e4e74ed0b0529c9aa845a1bcbc460b1c333e6d3b28ab362ac8e98bc15
 size 36542281664