Instructions to use gghfez/Writer-Large-2411-v2.1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use gghfez/Writer-Large-2411-v2.1-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="gghfez/Writer-Large-2411-v2.1-GGUF",
	filename="BF16/Writer-Large-2411-v2.1.bf16-00001-of-00005.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use gghfez/Writer-Large-2411-v2.1-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use gghfez/Writer-Large-2411-v2.1-GGUF with Ollama:
```
ollama run hf.co/gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M
```

Unsloth Studio

How to use gghfez/Writer-Large-2411-v2.1-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for gghfez/Writer-Large-2411-v2.1-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for gghfez/Writer-Large-2411-v2.1-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for gghfez/Writer-Large-2411-v2.1-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use gghfez/Writer-Large-2411-v2.1-GGUF with Docker Model Runner:
```
docker model run hf.co/gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M
```

Lemonade

How to use gghfez/Writer-Large-2411-v2.1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull gghfez/Writer-Large-2411-v2.1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Writer-Large-2411-v2.1-GGUF-Q4_K_M

List all available models

lemonade list

Writer-Large-2411-v2.1

GGUF Quants of gghfez/Writer-Large-2411-v2.1

Creative-Writing Control-Vectors available here: gghfez/Writer-Large-2411-v2.1-control-vectors

Overview

This model is built on Mistral-Large-Instruct-2411 and optimized for creative writing purposes. The base model excels at following instructions and handling details in long context when using the new prompt template.

Key Improvements

Reduced positivity bias
Reduced AI tropes and repetitive language patterns in story generation
Enhanced performance with longer context stories (multiple chapters) and roleplay sessions
Improved steering capabilities for roleplay via [OOC] instructions
Better handling of "group chat" scenarios

Usage

Prompt Template

The model requires a system prompt in the Mistral-V7 format. If you omit [SYSTEM_PROMPT] [/SYSTEM_PROMPT], the model:

May not follow instructions properly at short contexts
Can become repetitive at longer contexts

Example:

[SYSTEM_PROMPT]You are an award winning writer. Assist the user.[/SYSTEM_PROMPT][INST] Write the opening chapter of ... [/INST]

SillyTavern Integration

Story String:

[SYSTEM_PROMPT] {{#if system}}{{system}}[/SYSTEM_PROMPT] [INST]
{{/if}}{{#if wiBefore}}{{wiBefore}}
{{/if}}{{#if description}}{{description}}
{{/if}}{{#if personality}}{{personality}}
{{/if}}{{#if scenario}}{{scenario}}
{{/if}}{{#if wiAfter}}{{wiAfter}}
{{/if}}{{#if persona}}{{persona}}
{{/if}}{{trim}}[/INST] Understood.</s>

For response steering, use [OOC] commands, e.g.:

[OOC] Have them interrupted by a loud explosion in a nearby factory
[OOC] Have her refuse to sell it and suggest another merchant instead

Technical Details

Training

QLoRA training at 32768 context
Merged with gghfez/Mistral-Large-Instruct-2411 at bf16
jukofyork/Creative writing control vectors were applied during synthetic dataset generation
Includes standard assistant instruct data for long-context stability
Note: Performance on code tasks may be reduced compared to base model
Note: No attempt was made to remove 'Name-Slop', so you'll still encounter Lily and Elara if you don't specify character names

Context Length

Base model: 131,072 tokens
Training range: 1024-32728 tokens
Training context window: 32768 tokens

Testing Environments

Tested with exllamav2 4.5bpw on:

Downloads last month: 15

GGUF

Model size

123B params

Architecture

llama

Hardware compatibility

2-bit

3-bit

4-bit

16-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for gghfez/Writer-Large-2411-v2.1-GGUF

Base model

mistralai/Mistral-Large-Instruct-2411

Finetuned

gghfez/Writer-Large-2411-v2.1

Quantized

(9)

this model