Hugging Face's logo

Lewdiculous
/
LemonadeRP-4.5.3-GGUF-IQ-Imatrix

Transformers
GGUF
English
quantized
roleplay
imatrix
mistral
Merge
Model card Files
xet
 Community
1
How to use from
llama.cpp
Install (macOS, Linux) 
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Lewdiculous/LemonadeRP-4.5.3-GGUF-IQ-Imatrix:
# Run inference directly in the terminal:
llama cli -hf Lewdiculous/LemonadeRP-4.5.3-GGUF-IQ-Imatrix:
Install from WinGet (Windows) 
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Lewdiculous/LemonadeRP-4.5.3-GGUF-IQ-Imatrix:
# Run inference directly in the terminal:
llama cli -hf Lewdiculous/LemonadeRP-4.5.3-GGUF-IQ-Imatrix:
Use pre-built binary 
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Lewdiculous/LemonadeRP-4.5.3-GGUF-IQ-Imatrix:
# Run inference directly in the terminal:
./llama-cli -hf Lewdiculous/LemonadeRP-4.5.3-GGUF-IQ-Imatrix:
Build from source code 
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Lewdiculous/LemonadeRP-4.5.3-GGUF-IQ-Imatrix:
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Lewdiculous/LemonadeRP-4.5.3-GGUF-IQ-Imatrix:
Use Docker 
docker model run hf.co/Lewdiculous/LemonadeRP-4.5.3-GGUF-IQ-Imatrix:
Quick Links

This repository hosts GGUF-IQ-Imatrix quantizations for KatyTheCutie/LemonadeRP-4.5.3.

--contextsize 8192

What does "Imatrix" mean?

It stands for Importance Matrix, a technique used to improve the quality of quantized models. The Imatrix is calculated based on calibration data, and it helps determine the importance of different model activations during the quantization process. The idea is to preserve the most important information during quantization, which can help reduce the loss of model performance, especially when the calibration data is diverse. [1] [2]

For imatrix data generation, kalomaze's groups_merged.txt with added roleplay chats was used, you can find it here.

Steps:

Base⇢ GGUF(F16)⇢ Imatrix-Data(F16)⇢ GGUF(Imatrix-Quants)

Quants:

    quantization_options = [
        "Q4_K_M", "IQ4_XS", "Q5_K_M", "Q5_K_S", "Q6_K",
        "Q8_0", "IQ3_M", "IQ3_S", "IQ3_XXS"
    ]

If you want anything that's not here or another model, feel free to request.

Original model information:

image/png Lemonade RP 4.5.3

8192 context length.

7B roleplay focused model, creativity and less cliché is the focus of this merge. SillyTavern settings: image/png

image/png Models used in merge:

  • NeverSleep/Noromaid-7B-0.4-DPO
  • cgato/Thespis-7b-v0.5-SFTTest-2Epoch
  • NurtureAI/neural-chat-7b-v3-1-16k
  • cgato/Thespis-CurtainCall-7b-v0.2.2
  • tavtav/eros-7b-test

Feedback is always greatly appreciated! <3

Downloads last month
412
GGUF
Model size
7B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including Lewdiculous/LemonadeRP-4.5.3-GGUF-IQ-Imatrix