Instructions to use saidutta69/Qwen2.5-14B-Instruct-heretic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use saidutta69/Qwen2.5-14B-Instruct-heretic with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="saidutta69/Qwen2.5-14B-Instruct-heretic",
	filename="Qwen2.5-14B-Instruct-heretic-Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use saidutta69/Qwen2.5-14B-Instruct-heretic with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M

Use Docker

docker model run hf.co/saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M

LM Studio
Jan

vLLM

How to use saidutta69/Qwen2.5-14B-Instruct-heretic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "saidutta69/Qwen2.5-14B-Instruct-heretic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "saidutta69/Qwen2.5-14B-Instruct-heretic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M

Ollama
How to use saidutta69/Qwen2.5-14B-Instruct-heretic with Ollama:
```
ollama run hf.co/saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M
```

Unsloth Studio

How to use saidutta69/Qwen2.5-14B-Instruct-heretic with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for saidutta69/Qwen2.5-14B-Instruct-heretic to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for saidutta69/Qwen2.5-14B-Instruct-heretic to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for saidutta69/Qwen2.5-14B-Instruct-heretic to start chatting

How to use saidutta69/Qwen2.5-14B-Instruct-heretic with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use saidutta69/Qwen2.5-14B-Instruct-heretic with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use saidutta69/Qwen2.5-14B-Instruct-heretic with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use saidutta69/Qwen2.5-14B-Instruct-heretic with Docker Model Runner:
```
docker model run hf.co/saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M
```

Lemonade

How to use saidutta69/Qwen2.5-14B-Instruct-heretic with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M

Run and chat with the model

lemonade run user.Qwen2.5-14B-Instruct-heretic-Q4_K_M

List all available models

lemonade list

saidutta69 commited on 4 days ago

Commit

fde742a

verified ·

1 Parent(s): 617387d

Upload README.md with huggingface_hub

Browse files

Files changed (1) hide show

README.md +78 -128

README.md CHANGED Viewed

@@ -1,167 +1,117 @@
 ---
-license: apache-2.0
 license_link: https://huggingface.co/Qwen/Qwen2.5-14B-Instruct/blob/main/LICENSE
 language:
-- en
 pipeline_tag: text-generation
-base_model: Qwen/Qwen2.5-14B
-tags:
-- chat
-- heretic
-- uncensored
-- decensored
-- abliterated
-- reproducible
-library_name: transformers
 ---
-# This is a decensored version of [Qwen/Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct), made using [Heretic](https://heretic-project.org) v1.4.0
-> [!TIP]
-> **This model is reproducible!**
->
-> See the [README](reproduce/README.md) in the `reproduce` directory for more information.
-## Abliteration parameters
-| Parameter | Value |
-| :-------- | :---: |
-| **direction_index** | 28.80 |
-| **attn.o_proj.max_weight** | 1.35 |
-| **attn.o_proj.max_weight_position** | 35.72 |
-| **attn.o_proj.min_weight** | 0.86 |
-| **attn.o_proj.min_weight_distance** | 18.06 |
-| **mlp.down_proj.max_weight** | 1.45 |
-| **mlp.down_proj.max_weight_position** | 29.28 |
-| **mlp.down_proj.min_weight** | 0.06 |
-| **mlp.down_proj.min_weight_distance** | 22.72 |
-## Performance
-| Metric | This model | Original model ([Qwen/Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct)) |
-| :----- | :--------: | :---------------------------: |
-| **KL divergence** | 0.0600 | 0 *(by definition)* |
-| **Refusals** | 14/100 | 98/100 |
------
-# Qwen2.5-14B-Instruct
-## Introduction
-Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. Qwen2.5 brings the following improvements upon Qwen2:
-- Significantly **more knowledge** and has greatly improved capabilities in **coding** and **mathematics**, thanks to our specialized expert models in these domains.
-- Significant improvements in **instruction following**, **generating long texts** (over 8K tokens), **understanding structured data** (e.g, tables), and **generating structured outputs** especially JSON. **More resilient to the diversity of system prompts**, enhancing role-play implementation and condition-setting for chatbots.
-- **Long-context Support** up to 128K tokens and can generate up to 8K tokens.
-- **Multilingual support** for over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more.
-**This repo contains the instruction-tuned 14B Qwen2.5 model**, which has the following features:
-- Type: Causal Language Models
-- Training Stage: Pretraining & Post-training
-- Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
-- Number of Parameters: 14.7B
-- Number of Paramaters (Non-Embedding): 13.1B
-- Number of Layers: 48
-- Number of Attention Heads (GQA): 40 for Q and 8 for KV
-- Context Length: Full 131,072 tokens and generation 8192 tokens
-  - Please refer to [this section](#processing-long-texts) for detailed instructions on how to deploy Qwen2.5 for handling long texts.
-For more details, please refer to our [blog](https://qwenlm.github.io/blog/qwen2.5/), [GitHub](https://github.com/QwenLM/Qwen2.5), and [Documentation](https://qwen.readthedocs.io/en/latest/).
-## Requirements
-The code of Qwen2.5 has been in the latest Hugging face `transformers` and we advise you to use the latest version of `transformers`.
-With `transformers<4.37.0`, you will encounter the following error:
-```
-KeyError: 'qwen2'
-```
 ## Quickstart
-Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents.
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
-model_name = "Qwen/Qwen2.5-14B-Instruct"
-model = AutoModelForCausalLM.from_pretrained(
-    model_name,
-    torch_dtype="auto",
-    device_map="auto"
-)
 tokenizer = AutoTokenizer.from_pretrained(model_name)
-prompt = "Give me a short introduction to large language model."
-messages = [
-    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
-    {"role": "user", "content": prompt}
-]
-text = tokenizer.apply_chat_template(
-    messages,
-    tokenize=False,
-    add_generation_prompt=True
-)
-model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
-generated_ids = model.generate(
-    **model_inputs,
-    max_new_tokens=512
-)
-generated_ids = [
-    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
-]
-response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
 ```
-### Processing Long Texts
-The current `config.json` is set for context length up to 32,768 tokens.
-To handle extensive inputs exceeding 32,768 tokens, we utilize [YaRN](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts.
-For supported frameworks, you could add the following to `config.json` to enable YaRN:
-```json
-{
-  ...,
-  "rope_scaling": {
-    "factor": 4.0,
-    "original_max_position_embeddings": 32768,
-    "type": "yarn"
-  }
-}
-```
-For deployment, we recommend using vLLM.
-Please refer to our [Documentation](https://qwen.readthedocs.io/en/latest/deployment/vllm.html) for usage if you are not familar with vLLM.
-Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts**.
-We advise adding the `rope_scaling` configuration only when processing long contexts is required.
-## Evaluation & Performance
-Detailed evaluation results are reported in this [📑 blog](https://qwenlm.github.io/blog/qwen2.5/).
-For requirements on GPU memory and the respective throughput, see results [here](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html).
-## Citation
-If you find our work helpful, feel free to give us a cite.
-```
-@misc{qwen2.5,
-    title = {Qwen2.5: A Party of Foundation Models},
-    url = {https://qwenlm.github.io/blog/qwen2.5/},
-    author = {Qwen Team},
-    month = {September},
-    year = {2024}
-}
-@article{qwen2,
-      title={Qwen2 Technical Report},
-      author={An Yang and Baosong Yang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Zhou and Chengpeng Li and Chengyuan Li and Dayiheng Liu and Fei Huang and Guanting Dong and Haoran Wei and Huan Lin and Jialong Tang and Jialin Wang and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Ma and Jin Xu and Jingren Zhou and Jinze Bai and Jinzheng He and Junyang Lin and Kai Dang and Keming Lu and Keqin Chen and Kexin Yang and Mei Li and Mingfeng Xue and Na Ni and Pei Zhang and Peng Wang and Ru Peng and Rui Men and Ruize Gao and Runji Lin and Shijie Wang and Shuai Bai and Sinan Tan and Tianhang Zhu and Tianhao Li and Tianyu Liu and Wenbin Ge and Xiaodong Deng and Xiaohuan Zhou and Xingzhang Ren and Xinyu Zhang and Xipin Wei and Xuancheng Ren and Yang Fan and Yang Yao and Yichang Zhang and Yu Wan and Yunfei Chu and Yuqiong Liu and Zeyu Cui and Zhenru Zhang and Zhihao Fan},
-      journal={arXiv preprint arXiv:2407.10671},
-      year={2024}
-}
-```

 ---
+license: other
+license_name: qwen-research
 license_link: https://huggingface.co/Qwen/Qwen2.5-14B-Instruct/blob/main/LICENSE
+base_model: Qwen/Qwen2.5-14B-Instruct
+tags:
+  - qwen2
+  - chat
+  - heretic
+  - uncensored
+  - decensored
+  - abliterated
+  - conversational
+  - text-generation-inference
 language:
+  - en
 pipeline_tag: text-generation
 ---
+# Qwen2.5-14B-Instruct-heretic
+A decensored variant of [Qwen/Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct), produced with [Heretic](https://github.com/p-e-w/heretic) v1.4.0 (directional ablation / "abliteration"). Refusal behavior is suppressed via targeted weight edits to the attention output and MLP down-projections rather than fine-tuning, so the base model's knowledge and instruction-following are left largely intact.
+**Who this is for:** developers who want the largest Qwen2.5 instruct heretic in this series - strong reasoning and instruction-following that answers directly instead of refusing. Best run on a 16-24 GB GPU or via the Q4_K_M/Q5_K_M GGUF on consumer hardware. Not a capability upgrade over base Qwen2.5-14B-Instruct - same model, refusal guardrails removed.
+## Why abliteration instead of fine-tuning
+Fine-tuning a "helpful" persona on top of RLHF'd refusals fights the base model's training and tends to degrade coherence. Abliteration instead finds and edits the specific weight directions responsible for refusal, leaving the rest of the network (and its capabilities) untouched. See the [Heretic repo](https://github.com/p-e-w/heretic) and the [original abliteration writeup](https://huggingface.co/blog/mlabonne/abliteration) for the mechanism.
+## Abliteration parameters
+| Parameter | Value |
+|---|---|
+| `direction_index` | 28.80 |
+| `attn.o_proj.max_weight` | 1.35 |
+| `attn.o_proj.max_weight_position` | 35.72 |
+| `attn.o_proj.min_weight` | 0.86 |
+| `attn.o_proj.min_weight_distance` | 18.06 |
+| `mlp.down_proj.max_weight` | 1.45 |
+| `mlp.down_proj.max_weight_position` | 29.28 |
+| `mlp.down_proj.min_weight` | 0.06 |
+| `mlp.down_proj.min_weight_distance` | 22.72 |
+## Performance
+| Metric | This model | Qwen2.5-14B-Instruct (base) |
+|---|---|---|
+| Refusals (out of 100 adversarial prompts) | 14/100 | 98/100 |
+| KL divergence from base | 0.0600 | 0 *(by definition)* |
+KL divergence of 0.06 on the output distribution is low for a 14B model - the edit is narrow and targeted. Refusals dropped from 98 to 14 out of 100 adversarial prompts while retaining nearly all original capabilities.
+> Made with ❤️ by **RACER IS OP** - follow for more uncensored models
+## Files
+### Safetensors (BF16)
+The full-precision weights are in `model-0000N-of-0000N.safetensors` (see the repo file listing for the exact shard count and sizes).
+### GGUF quantizations
+GGUF quantizations are published for this model (Q4_K_M, Q5_K_M, Q6_K, Q8_0). Exact sizes are in the repo file listing. Pull a specific quant with `llama.cpp` / `ollama` (see Quickstart).
+| File | Format | Size |
+|---|---|---|
+| `Qwen2.5-14B-Instruct-heretic-Q4_K_M.gguf` | GGUF Q4_K_M | (see repo files for exact size) |
+| `Qwen2.5-14B-Instruct-heretic-Q5_K_M.gguf` | GGUF Q5_K_M | (see repo files for exact size) |
+| `Qwen2.5-14B-Instruct-heretic-Q6_K.gguf` | GGUF Q6_K | (see repo files for exact size) |
+| `Qwen2.5-14B-Instruct-heretic-Q8_0.gguf` | GGUF Q8_0 | (see repo files for exact size) |
 ## Quickstart
+```bash
+# llama.cpp - defaults to the Q4_K_M quant if multiple are present
+llama serve -hf saidutta69/Qwen2.5-14B-Instruct-heretic:Q4_K_M
+```
 ```python
+# transformers
 from transformers import AutoModelForCausalLM, AutoTokenizer
+model_name = "saidutta69/Qwen2.5-14B-Instruct-heretic"
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")
 tokenizer = AutoTokenizer.from_pretrained(model_name)
+messages = [{"role": "user", "content": "Who are you?"}]
+inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=True,
+                                        return_dict=True, return_tensors="pt").to(model.device)
+out = model.generate(**inputs, max_new_tokens=200)
+print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
 ```
+Also runnable via Ollama, LM Studio, Jan, vLLM, SGLang - see the "Use this model" widget above for copy-paste commands.
+## Responsible use
+Refusal suppression is deliberate and works as intended: this model will comply with requests the base model would refuse, including some it shouldn't. There is no safety filtering layered on top. You are responsible for how you deploy it - don't put this behind an unmoderated public-facing endpoint serving third parties. It inherits Qwen/Qwen2.5-14B-Instruct's factual limitations and biases; abliteration removes refusal directions, it doesn't add capability or judgment.
+## License
+Inherits the [`qwen-research`](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct/blob/main/LICENSE) license from the base model - research use, see the linked license for commercial terms.
+## Related
+- [Qwen2.5-7B-Instruct-heretic](https://huggingface.co/saidutta69/Qwen2.5-7B-Instruct-heretic)
+- [Qwen2.5-3B-Instruct-heretic](https://huggingface.co/saidutta69/Qwen2.5-3B-Instruct-heretic)
+---
+# Base model: Qwen2.5-14B-Instruct
+<details>
+<summary>Original Qwen2.5-14B-Instruct model card (click to expand)</summary>
+See the base model card at [Qwen/Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct) for the original architecture, training details, requirements, and citation.
+</details>