How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf DiYaZeN/aya-sl-biz-8b:Q4_K_M# Run inference directly in the terminal:
llama cli -hf DiYaZeN/aya-sl-biz-8b:Q4_K_MUse pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf DiYaZeN/aya-sl-biz-8b:Q4_K_M# Run inference directly in the terminal:
./llama-cli -hf DiYaZeN/aya-sl-biz-8b:Q4_K_MBuild from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf DiYaZeN/aya-sl-biz-8b:Q4_K_M# Run inference directly in the terminal:
./build/bin/llama-cli -hf DiYaZeN/aya-sl-biz-8b:Q4_K_MUse Docker
docker model run hf.co/DiYaZeN/aya-sl-biz-8b:Q4_K_MQuick Links
Aya Sl Biz 8B
This is a GGUF format quantized version of a fine-tuned CohereForAI/aya-23-8B model.
Model Details
- Original Model: CohereForAI/aya-23-8B
- Quantization Type: Q4_K_M
- Format: GGUF
- Conversion Date: 2024-10-31
- Framework: llama.cpp
Usage
This model can be used with llama.cpp. Here's how to use it:
# Basic usage
./llama-cli -m path_to_model.gguf -n 512 --prompt "Your prompt here"
# Chat format
./llama-cli -m path_to_model.gguf --temp 0.7 --repeat-penalty 1.2 -n 512 --prompt "<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>You are Command-R, a helpful AI assistant.<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Your prompt here<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>"
Quantization Details
This model was quantized using the Q4_K_M format, which offers a good balance between model size and performance. The quantization was performed using llama.cpp's quantization tools.
Original model size: ~16GB Quantized model size: ~4.7GB
License
This model is released under the Apache 2.0 license.
- Downloads last month
- 5
Hardware compatibility
Log In to add your hardware
4-bit
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf DiYaZeN/aya-sl-biz-8b:Q4_K_M# Run inference directly in the terminal: llama cli -hf DiYaZeN/aya-sl-biz-8b:Q4_K_M