Instructions to use tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K

Use Docker

docker model run hf.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K

LM Studio
Jan
Ollama
How to use tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF with Ollama:
```
ollama run hf.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K
```
Unsloth Desktop
Docker Model Runner
How to use tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF with Docker Model Runner:
```
docker model run hf.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K
```

Lemonade

How to use tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF:Q2_K

Run and chat with the model

lemonade run user.CUBOX-SOLAR-DPO-v0.3-GGUF-Q2_K

List all available models

lemonade list

Atomic Chat

morriszms commited on Dec 14, 2024

Commit

87aa663

verified ·

1 Parent(s): 4b39297

Upload folder using huggingface_hub

Browse files

Files changed (14) hide show

.gitattributes +12 -0
CUBOX-SOLAR-DPO-v0.3-Q2_K.gguf +3 -0
CUBOX-SOLAR-DPO-v0.3-Q3_K_L.gguf +3 -0
CUBOX-SOLAR-DPO-v0.3-Q3_K_M.gguf +3 -0
CUBOX-SOLAR-DPO-v0.3-Q3_K_S.gguf +3 -0
CUBOX-SOLAR-DPO-v0.3-Q4_0.gguf +3 -0
CUBOX-SOLAR-DPO-v0.3-Q4_K_M.gguf +3 -0
CUBOX-SOLAR-DPO-v0.3-Q4_K_S.gguf +3 -0
CUBOX-SOLAR-DPO-v0.3-Q5_0.gguf +3 -0
CUBOX-SOLAR-DPO-v0.3-Q5_K_M.gguf +3 -0
CUBOX-SOLAR-DPO-v0.3-Q5_K_S.gguf +3 -0
CUBOX-SOLAR-DPO-v0.3-Q6_K.gguf +3 -0
CUBOX-SOLAR-DPO-v0.3-Q8_0.gguf +3 -0
README.md +79 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+CUBOX-SOLAR-DPO-v0.3-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

CUBOX-SOLAR-DPO-v0.3-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:eb27e6c6033990639c14042ec6a46b8bfcb3860c50d5be4f27f538fcf623381a
+size 4003232288

CUBOX-SOLAR-DPO-v0.3-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b48dd705e2b8b6c2f7a86a5e7a5916fc4613c04f37dde1faed40a48bd8f6ee95
+size 5650749984

CUBOX-SOLAR-DPO-v0.3-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e26fb32c996e1160de094b11947dd573a35bfb718a78ca5222a11dead422eefb
+size 5195668000

CUBOX-SOLAR-DPO-v0.3-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7accfbac3470eca8ac51d4db9600ee0ecd98a05412cb89bb9ba6097421dab01c
+size 4664564256

CUBOX-SOLAR-DPO-v0.3-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5e63a0b285dde0469fb869910ccad5fc6f1d90582f79b7ecb9c27221833db468
+size 6072384032

CUBOX-SOLAR-DPO-v0.3-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d2a9d46cc0044c4be7c594b18bf4c92e84d7dfcf8d122f9e5761adbef0705e9f
+size 6461667872

CUBOX-SOLAR-DPO-v0.3-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7de83660bb16911f31e0bbe1a9da742d1c82d826dd95ee745e4fb3221e9130a4
+size 6118521376

CUBOX-SOLAR-DPO-v0.3-Q5_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1463972bb169e180c0ab1d4bda7f89562326a3c99a7e8a110ea91c7629281bf0
+size 7397390880

CUBOX-SOLAR-DPO-v0.3-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1e57cf802e9f3ca91f1cc239b84c728554fdea3b775722406dc8acbec6dc6973
+size 7597931040

CUBOX-SOLAR-DPO-v0.3-Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a330d3b85fb81c5837e76f48e73cb1481698b7e05d401b0d51c72b626666bb13
+size 7397390880

CUBOX-SOLAR-DPO-v0.3-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6abe309c64691bffc8112df771a0f364b8abe57af52d6647848ea3064db27a40
+size 8805210656

CUBOX-SOLAR-DPO-v0.3-Q8_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:49bfffee8069e89814bb369da8a1fccb40f8d0650cfa81cb2625d5be19db5158
+size 11404155424

README.md ADDED Viewed

	@@ -0,0 +1,79 @@

+---
+library_name: transformers
+license: apache-2.0
+language:
+- ko
+base_model: jwkweon/CUBOX-SOLAR-DPO-v0.3
+tags:
+- TensorBlock
+- GGUF
+---
+<div style="width: auto; margin-left: auto; margin-right: auto">
+<img src="https://i.imgur.com/jC7kdl8.jpeg" alt="TensorBlock" style="width: 100%; min-width: 400px; display: block; margin: auto;">
+</div>
+<div style="display: flex; justify-content: space-between; width: 100%;">
+    <div style="display: flex; flex-direction: column; align-items: flex-start;">
+        <p style="margin-top: 0.5em; margin-bottom: 0em;">
+            Feedback and support: TensorBlock's  <a href="https://x.com/tensorblock_aoi">Twitter/X</a>, <a href="https://t.me/TensorBlock">Telegram Group</a> and <a href="https://x.com/tensorblock_aoi">Discord server</a>
+        </p>
+    </div>
+</div>
+## jwkweon/CUBOX-SOLAR-DPO-v0.3 - GGUF
+This repo contains GGUF format model files for [jwkweon/CUBOX-SOLAR-DPO-v0.3](https://huggingface.co/jwkweon/CUBOX-SOLAR-DPO-v0.3).
+The files were quantized using machines provided by [TensorBlock](https://tensorblock.co/), and they are compatible with llama.cpp as of [commit b4242](https://github.com/ggerganov/llama.cpp/commit/a6744e43e80f4be6398fc7733a01642c846dce1d).
+<div style="text-align: left; margin: 20px 0;">
+    <a href="https://tensorblock.co/waitlist/client" style="display: inline-block; padding: 10px 20px; background-color: #007bff; color: white; text-decoration: none; border-radius: 5px; font-weight: bold;">
+        Run them on the TensorBlock client using your local machine ↗
+    </a>
+</div>
+## Prompt template
+```
+```
+## Model file specification
+| Filename | Quant type | File Size | Description |
+| -------- | ---------- | --------- | ----------- |
+| [CUBOX-SOLAR-DPO-v0.3-Q2_K.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q2_K.gguf) | Q2_K | 4.003 GB | smallest, significant quality loss - not recommended for most purposes |
+| [CUBOX-SOLAR-DPO-v0.3-Q3_K_S.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q3_K_S.gguf) | Q3_K_S | 4.665 GB | very small, high quality loss |
+| [CUBOX-SOLAR-DPO-v0.3-Q3_K_M.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q3_K_M.gguf) | Q3_K_M | 5.196 GB | very small, high quality loss |
+| [CUBOX-SOLAR-DPO-v0.3-Q3_K_L.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q3_K_L.gguf) | Q3_K_L | 5.651 GB | small, substantial quality loss |
+| [CUBOX-SOLAR-DPO-v0.3-Q4_0.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q4_0.gguf) | Q4_0 | 6.072 GB | legacy; small, very high quality loss - prefer using Q3_K_M |
+| [CUBOX-SOLAR-DPO-v0.3-Q4_K_S.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q4_K_S.gguf) | Q4_K_S | 6.119 GB | small, greater quality loss |
+| [CUBOX-SOLAR-DPO-v0.3-Q4_K_M.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q4_K_M.gguf) | Q4_K_M | 6.462 GB | medium, balanced quality - recommended |
+| [CUBOX-SOLAR-DPO-v0.3-Q5_0.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q5_0.gguf) | Q5_0 | 7.397 GB | legacy; medium, balanced quality - prefer using Q4_K_M |
+| [CUBOX-SOLAR-DPO-v0.3-Q5_K_S.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q5_K_S.gguf) | Q5_K_S | 7.397 GB | large, low quality loss - recommended |
+| [CUBOX-SOLAR-DPO-v0.3-Q5_K_M.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q5_K_M.gguf) | Q5_K_M | 7.598 GB | large, very low quality loss - recommended |
+| [CUBOX-SOLAR-DPO-v0.3-Q6_K.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q6_K.gguf) | Q6_K | 8.805 GB | very large, extremely low quality loss |
+| [CUBOX-SOLAR-DPO-v0.3-Q8_0.gguf](https://huggingface.co/tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF/blob/main/CUBOX-SOLAR-DPO-v0.3-Q8_0.gguf) | Q8_0 | 11.404 GB | very large, extremely low quality loss - not recommended |
+## Downloading instruction
+### Command line
+Firstly, install Huggingface Client
+```shell
+pip install -U "huggingface_hub[cli]"
+```
+Then, downoad the individual model file the a local directory
+```shell
+huggingface-cli download tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF --include "CUBOX-SOLAR-DPO-v0.3-Q2_K.gguf" --local-dir MY_LOCAL_DIR
+```
+If you wanna download multiple model files with a pattern (e.g., `*Q4_K*gguf`), you can try:
+```shell
+huggingface-cli download tensorblock/CUBOX-SOLAR-DPO-v0.3-GGUF --local-dir MY_LOCAL_DIR --local-dir-use-symlinks False --include='*Q4_K*gguf'
+```