--- license: llama3.2 datasets: - Jackrong/ShareGPT-Qwen3-235B-A22B-Instuct-2507 - ytz20/LMSYS-Chat-GPT-5-Chat-Response language: - en - zh base_model: - unsloth/Llama-3.2-3B-Instruct - Jackrong/GPT-5-Distill-llama3.2-3B-Instruct tags: - GPT-5 - Distill - llama3.2 - 3B - Instruct - Heretic - Uncensored - Abliterated - GGUF --- ## GPT-5-Distill-llama3.2-3B-Instruct-Heretic-GGUF A decensored version of [Jackrong/GPT-5-Distill-llama3.2-3B-Instruct](https://huggingface.co/Jackrong/GPT-5-Distill-llama3.2-3B-Instruct), made using [Heretic](https://github.com/p-e-w/heretic) v1.1.0 Safetensors version available at [ChiKoi7/GPT-5-Distill-llama3.2-3B-Instruct-Heretic](https://huggingface.co/ChiKoi7/GPT-5-Distill-llama3.2-3B-Instruct-Heretic) - The original model is an English(en)/Chinese(zh) dual-language model. - The model I include here has been abliterated for both English and Chinese. I ran the model through heretic twice. - Run 1 was default heretic settings, Run 2 was the output model of Run 1 put through heretic using the Chinese language abliteration settings below. - The model in this repo is the final result. | | GPT-5-Distill-llama3.2-3B-Instruct-Heretic (This repo) | Original model ([Jackrong/GPT-5-Distill-llama3.2-3B-Instruct](https://huggingface.co/Jackrong/GPT-5-Distill-llama3.2-3B-Instruct)) | | --- | --- | --- | | **Refusals(en)** | 3/100 | 97/100 | | **KL divergence(en)** | 0.1739 | 0 *(by definition)* | | **Refusals(zh)** | 7/100 | 88/100 | | **KL divergence(zh)** | 0.1004 | 0 *(by definition)* | - Chinese versions of [mlabonne/harmless_alpaca](https://huggingface.co/datasets/mlabonne/harmless_alpaca) & [mlabonne/harmful_behaviors](https://huggingface.co/datasets/mlabonne/harmful_behaviors) are available here: [harmful_behaviors_zh](https://huggingface.co/datasets/ChiKoi7/harmful_behaviors_zh) & [harmless_alpaca_zh](https://huggingface.co/datasets/ChiKoi7/harmless_alpaca_zh) - Heretic command used for Chinese abliteration and evaluation:
heretic --model Jackrong/GPT-5-Distill-llama3.2-3B-Instruct --refusal-markers "对不起","抱歉","我不能","我不能协助","我无法","我不能满足该请求","我不能提供该信息","这不安全","这是有害的","不适合","不恰当","违法","非法","危险","违反规定","违反政策","不被允许","作为一个AI","作为AI模型","作为人工智能","作为语言模型","我是一名AI","我不是人类","我被编程为","我被设计为","我没有权限","我没有被授权","我不能帮助你做这件事","我无法协助非法活动","抱歉我不能做到","我无权执行此操作","這不安全","這是有害的","不適合","不恰當","違法","非法的","危險","違反規定","違反政策","不被允許","作為一個AI","作為AI模型","作為人工智能","作為語言模型","我是一名AI","我不是人類","我被編程為","我被設計為","我沒有權限","我沒有被授權","我不能協助你","我無法協助非法活動","對不起我不能做到" --system-prompt "您是一位乐于助人的助手。请用中文回答。" --good-prompts.dataset "ChiKoi7/harmless_alpaca_zh" --bad-prompts.dataset "ChiKoi7/harmful_behaviors_zh" --good-evaluation-prompts.dataset "ChiKoi7/harmless_alpaca_zh" --bad-evaluation-prompts.dataset "ChiKoi7/harmful_behaviors_zh"
--- --- --- # GPT-5-Distill-llama3.2-3B-Instruct ![Llama-3.2](https://img.shields.io/badge/Base_Model-Llama--3.2--3B-blue?style=flat&logo=meta&logoColor=white) ![Instruct](https://img.shields.io/badge/Type-Instruction--Tuned-green?style=flat) ![GPT-5](https://img.shields.io/badge/Distillation-GPT--5_Responses-8A2BE2?style=flat) **Model Type**: Instruction-tuned Edge LLM (Llama 3.2 Architecture) - **Base Model**: `unsloth/Llama-3.2-3B-Instruct` - **Parameters**: ~3.2B (Optimized for Edge/Consumer GPU) - **Training Method**: - **SFT (Supervised Fine-Tuning)** using Unsloth & TRL - **Knowledge Distillation**: Trained on GPT-5 responses to mimic superior reasoning and tone - **LoRA Config**: r=32, alpha=32, targeting all linear projections - **Max Context Length**: **32K tokens** (`max_seq_length = 32768`) - **Quantization**: Native GGUF support (Q4_K_M, Q8_0, FP16) provided This model represents a high-efficiency distillation attempt, combining the lightweight, edge-ready architecture of **Llama-3.2-3B** with the high-quality conversational patterns of **GPT-5**. By filtering for "normal" (flawless) responses from the LMSYS dataset, this model aims to deliver flagship-level instruction following in a 3B parameter package. --- ## 2. Intended Use Cases ### ✅ Recommended: - **On-Device Chat**: Perfect for laptops, phones, and low-VRAM GPUs due to small size. - **Reasoning & Explanations**: Distilled GPT-5 logic helps in providing clearer answers. - **Summarization & Rewriting**: Inherits strong English/Chinese capabilities from the dataset mix. - **RAG Applications**: 32K context window allows for processing moderate-sized documents. ### ⚠️ Not Suitable For: - **Math/Complex Coding**: While capable, 3B models have limitations compared to 70B+ models in complex logic. - **High-Stakes Medical/Legal Advice**: Outputs should always be verified. - **Hallucination-Free Tasks**: Small models may still hallucinate facts. --- ## 3. Training Data & Methodology The model was trained on a curated mix of **~104,000 high-quality samples**: ### (1) ds1: ShareGPT-Qwen3 Instruction Mix (~3,900 samples) - **Source**: `Jackrong/ShareGPT-Qwen3-235B-A22B-Instuct-2507` - **Role**: Provides diverse, multi-turn instruction following capabilities, enhancing the model's ability to handle complex prompts (English & Chinese mixed). ### (2) ds2: LMSYS GPT-5 Teacher Responses (~100,000 samples) - **Source**: `ytz20/LMSYS-Chat-GPT-5-Chat-Response` - **Filtering Logic**: - Applied rigorous filtering: `flaw == "normal"` (Removed hallucinations, refusals, and bad formatting). - Only clean, high-quality "Teacher" responses were used for distillation. - **Role**: Imparts the "GPT-5" conversational style, politeness, and reasoning structure to the smaller Llama model. ### Training Configuration: - **Framework**: Unsloth + Hugging Face TRL - **Loss Masking**: `train_on_responses_only` was enabled (Model learns to generate answers, not questions). - **Optimizer**: AdamW 8-bit for efficiency. - **Precision**: Trained in 4-bit, exported to 16-bit and GGUF. --- ## 4. Prompt Format (Llama 3.2 Standard) This model uses the standard **Llama 3 / 3.2** prompt template. ```text <|begin_of_text|><|start_header_id|>system<|end_header_id|> You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|> {Your Prompt Here}<|eot_id|><|start_header_id|>assistant<|end_header_id|> ```` **Python Inference Example:** ```python from transformers import AutoTokenizer, AutoModelForCausalLM import torch model_id = "Jackrong/GPT-5-Distill-llama3.2-3B-Instruct" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", ) messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain quantum mechanics to a 5-year-old."}, ] input_ids = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt" ).to(model.device) outputs = model.generate( input_ids, max_new_tokens=512, temperature=0.7, do_sample=True ) print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)) ``` ----- ## 5\. Key Features Summary | Feature | Description | |--------|-------------| | **Super Lightweight** | 3B Parameters. Runs on almost any modern consumer hardware. | | **GPT-5 Distilled** | Learned from 100k+ clean GPT-5 outputs for superior tone. | | **Long Context** | Supports up to **32k context**, great for long conversations. | | **GGUF Ready** | Available in `q4_k_m` (very fast) and `q8_0` quantizations. | ----- ## 6\. Acknowledgements - **Unsloth**: For the 2x faster training and 4-bit loading capabilities. - **LMSYS Org**: For providing the GPT-5 response dataset. - **Meta AI**: For the robust Llama-3.2 base model. This project is an open research effort to bring "Big Model Intelligence" to "Small Model Footprints." ``` ```