--- license: gemma base_model: google/gemma-2-9b-it library_name: peft pipeline_tag: text-generation language: en datasets: - mario-rc/aif-emotional-generation tags: - llama-factory - peft - lora - dpo - rlaif - emotional-intelligence - gemma - generated_from_trainer model-index: - name: dpo-gemma-2-9b-it results: [] --- # Emotional RLAIF DPO Gemma-2-9B-IT This repository contains a LoRA/PEFT adapter trained from [google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it) with LLaMA-Factory using Direct Preference Optimization (DPO) for emotional response alignment. The adapter was trained as part of an emotional RLAIF pipeline using the mario-rc/aif-emotional-generation dataset, with mario-rc/aif-emotional-generation/dialogues used for SFT and mario-rc/aif-emotional-generation/aif_annotations preferences used for DPO preference alignment. Project repository: [`Mario-RC/aif-emotional-model`](https://github.com/Mario-RC/aif-emotional-model/tree/main). ## Intended Use This adapter is intended for research and experimentation with emotionally aligned dialogue generation. It should be loaded on top of the corresponding base model using PEFT. ## Model Details Key details about the base model and alignment method. - **Base model:** `google/gemma-2-9b-it` - **Adapter repository:** `mario-rc/emotional-rlaif-dpo-gemma-2-9b-it` - **Adapter type:** LoRA / PEFT - **Alignment method:** DPO - **Training framework:** LLaMA-Factory - **Prompt template:** `gemma` - **Dataset:** [mario-rc/aif-emotional-generation](https://huggingface.co/datasets/mario-rc/aif-emotional-generation) ## Released Emotional RLAIF Models The released emotional RLAIF adapters are available on Hugging Face: | Model | Base model | Size | Alignment method | Prompt template | | --- | --- | :---: | :---: | :---: | | [`emotional-rlaif-ppo-gemma-2-2b-it`](https://huggingface.co/mario-rc/emotional-rlaif-ppo-gemma-2-2b-it) | [`google/gemma-2-2b-it`](https://huggingface.co/google/gemma-2-2b-it) | 2B | PPO | `gemma` | | [`emotional-rlaif-dpo-gemma-2-2b-it`](https://huggingface.co/mario-rc/emotional-rlaif-dpo-gemma-2-2b-it) | [`google/gemma-2-2b-it`](https://huggingface.co/google/gemma-2-2b-it) | 2B | DPO | `gemma` | | [`emotional-rlaif-ppo-gemma-2-9b-it`](https://huggingface.co/mario-rc/emotional-rlaif-ppo-gemma-2-9b-it) | [`google/gemma-2-9b-it`](https://huggingface.co/google/gemma-2-9b-it) | 9B | PPO | `gemma` | | [`emotional-rlaif-dpo-gemma-2-9b-it`](https://huggingface.co/mario-rc/emotional-rlaif-dpo-gemma-2-9b-it) | [`google/gemma-2-9b-it`](https://huggingface.co/google/gemma-2-9b-it) | 9B | DPO | `gemma` | | [`emotional-rlaif-ppo-gemma-4-e2b-it`](https://huggingface.co/mario-rc/emotional-rlaif-ppo-gemma-4-e2b-it) | [`google/gemma-4-E2B-it`](https://huggingface.co/google/gemma-4-E2B-it) | E2B | PPO | `gemma4n_nothink` | | [`emotional-rlaif-dpo-gemma-4-e2b-it`](https://huggingface.co/mario-rc/emotional-rlaif-dpo-gemma-4-e2b-it) | [`google/gemma-4-E2B-it`](https://huggingface.co/google/gemma-4-E2B-it) | E2B | DPO | `gemma4n_nothink` | | [`emotional-rlaif-ppo-gemma-4-e4b-it`](https://huggingface.co/mario-rc/emotional-rlaif-ppo-gemma-4-e4b-it) | [`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it) | E4B | PPO | `gemma4n_nothink` | | [`emotional-rlaif-dpo-gemma-4-e4b-it`](https://huggingface.co/mario-rc/emotional-rlaif-dpo-gemma-4-e4b-it) | [`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it) | E4B | DPO | `gemma4n_nothink` | | [`emotional-rlaif-ppo-glm-4-9b-chat-1m`](https://huggingface.co/mario-rc/emotional-rlaif-ppo-glm-4-9b-chat-1m) | [`THUDM/glm-4-9b-chat-1m`](https://huggingface.co/THUDM/glm-4-9b-chat-1m) | 9B | PPO | `glm4` | | [`emotional-rlaif-dpo-glm-4-9b-chat-1m`](https://huggingface.co/mario-rc/emotional-rlaif-dpo-glm-4-9b-chat-1m) | [`THUDM/glm-4-9b-chat-1m`](https://huggingface.co/THUDM/glm-4-9b-chat-1m) | 9B | DPO | `glm4` | | [`emotional-rlaif-ppo-meta-llama-3-8b-instruct`](https://huggingface.co/mario-rc/emotional-rlaif-ppo-meta-llama-3-8b-instruct) | [`meta-llama/Meta-Llama-3-8B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) | 8B | PPO | `llama3` | | [`emotional-rlaif-dpo-meta-llama-3-8b-instruct`](https://huggingface.co/mario-rc/emotional-rlaif-dpo-meta-llama-3-8b-instruct) | [`meta-llama/Meta-Llama-3-8B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) | 8B | DPO | `llama3` | | [`emotional-rlaif-ppo-llama-3.2-1b-instruct`](https://huggingface.co/mario-rc/emotional-rlaif-ppo-llama-3.2-1b-instruct) | [`meta-llama/Llama-3.2-1B-Instruct`](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) | 1B | PPO | `llama3` | | [`emotional-rlaif-dpo-llama-3.2-1b-instruct`](https://huggingface.co/mario-rc/emotional-rlaif-dpo-llama-3.2-1b-instruct) | [`meta-llama/Llama-3.2-1B-Instruct`](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) | 1B | DPO | `llama3` | | [`emotional-rlaif-ppo-llama-3.2-3b-instruct`](https://huggingface.co/mario-rc/emotional-rlaif-ppo-llama-3.2-3b-instruct) | [`meta-llama/Llama-3.2-3B-Instruct`](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) | 3B | PPO | `llama3` | | [`emotional-rlaif-dpo-llama-3.2-3b-instruct`](https://huggingface.co/mario-rc/emotional-rlaif-dpo-llama-3.2-3b-instruct) | [`meta-llama/Llama-3.2-3B-Instruct`](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) | 3B | DPO | `llama3` | | [`emotional-rlaif-ppo-mistral-7b-instruct-v0.3`](https://huggingface.co/mario-rc/emotional-rlaif-ppo-mistral-7b-instruct-v0.3) | [`mistralai/Mistral-7B-Instruct-v0.3`](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) | 7B | PPO | `mistral` | | [`emotional-rlaif-dpo-mistral-7b-instruct-v0.3`](https://huggingface.co/mario-rc/emotional-rlaif-dpo-mistral-7b-instruct-v0.3) | [`mistralai/Mistral-7B-Instruct-v0.3`](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) | 7B | DPO | `mistral` | | [`emotional-rlaif-ppo-phi-3-small-8k-instruct`](https://huggingface.co/mario-rc/emotional-rlaif-ppo-phi-3-small-8k-instruct) | [`microsoft/Phi-3-small-8k-instruct`](https://huggingface.co/microsoft/Phi-3-small-8k-instruct) | 7B | PPO | `phi` | | [`emotional-rlaif-dpo-phi-3-small-8k-instruct`](https://huggingface.co/mario-rc/emotional-rlaif-dpo-phi-3-small-8k-instruct) | [`microsoft/Phi-3-small-8k-instruct`](https://huggingface.co/microsoft/Phi-3-small-8k-instruct) | 7B | DPO | `phi` | ## Training Procedure Key hyperparameters for this adapter: - **Learning rate:** 5e-6 - **Epochs:** 1 - **Scheduler:** cosine - **Warmup ratio:** 0.1 - **SFT data:** `mario-rc/aif-emotional-generation/dialogues` - **DPO preference data:** `mario-rc/aif-emotional-generation/aif_annotations` preference pairs - **Precision:** bfloat16 - **Optimizer:** AdamW (torch) - **Microbatch per device:** 1 - **Gradient accumulation:** 64 - **Effective batch size:** 64 (1 × 64, single device) - **Seed:** 42 - **Training cutoff (tokens):** 2048 - **LoRA rank / alpha / dropout:** 8 / 16 / 0.0 - **Alignment initialization:** SFT - **DPO beta:** 0.1 - **Preference loss:** sigmoid - **FTX coefficient:** 0.0 - **Label smoothing:** 0.0 - **Evaluation data:** `mario-rc/aif-emotional-generation/dialogues`, split `test`, 392 English examples - **Evaluation decoding:** `do_sample=True`, `temperature=0.95`, `top_p=0.7`, `top_k=50`, `max_new_tokens=1024`, `num_beams=1`, batch size 1, seed 42 ## Framework Versions Verified framework versions used during training. - **PEFT:** 0.11.1 - **Transformers:** 4.45.2 - **PyTorch:** 2.5.1+cu124 - **LLaMA-Factory:** LoRA/PEFT training workflow ## Usage Example The following example loads this adapter and generates a single response using the emotional dialogue format. The user emotion is `HAPPINESS`, the requested chatbot emotion is `ANGER`, and the final response part is `NEUTRAL`. Before running either example, install [PyTorch for your hardware](https://pytorch.org/get-started/locally/) and these dependencies. Accept the [Gemma access conditions](https://huggingface.co/google/gemma-2-9b-it), then authenticate with a Hugging Face read-access token: ```bash python -m pip install "transformers==4.45.2" "peft==0.11.1" "accelerate==0.34.0" "huggingface_hub==0.27.1" "sentencepiece==0.2.0" huggingface-cli login ``` ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel base_model_id = "google/gemma-2-9b-it" adapter_id = "mario-rc/emotional-rlaif-dpo-gemma-2-9b-it" tokenizer = AutoTokenizer.from_pretrained(base_model_id) model = AutoModelForCausalLM.from_pretrained( base_model_id, device_map="auto", torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported(including_emulation=False) else torch.float32, ) model = PeftModel.from_pretrained(model, adapter_id) model.eval() system_prompt = """You are an expert at creating dialogues. Dialogue and emotional structure: Human: (HAPPINESS) PROMPT. Chatbot: (HAPPINESS) RESPONSE_1. (ANGER) RESPONSE_2. (NEUTRAL) RESPONSE_3. Dialogue rules: The response must be open-domain curated. The response should be coherent, empathetic, engaging and proactive. The chatbot RESPONSE is composed of 3 different sentences (RESPONSE_1, RESPONSE_2 and RESPONSE_3), separated by a period. Between RESPONSE_1, RESPONSE_2 and RESPONSE_3 should be a max length of 20-25 words. RESPONSE_3 must be open-ended to follow-up the conversation, so the Human is encouraged to answer with a full long sentence. Avoid yes/no questions. Emotional response rules: RESPONSE_1 must contain a HAPPINESS tone. RESPONSE_2 must contain a ANGER tone. RESPONSE_3 must contain a NEUTRAL tone. Answer in a single turn to Human. Follow exactly the emotional structure and the emotional and dialogue rules.""" messages = [ {"role": "user", "content": "(HAPPINESS) I just got my dream job!"} ] chat_prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=False, ) # Match the LLaMA-Factory system prefix used during training. prompt = tokenizer.bos_token + system_prompt + chat_prompt.removeprefix(tokenizer.bos_token) inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt").input_ids.to(model.device) with torch.no_grad(): outputs = model.generate( inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9, ) print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)) ``` ## How to Use The following example loads this adapter and runs an interactive emotional dialogue loop in English or Spanish. Set `language = "en"` or `language = "es"` in the final block to choose the language. Use exit to stop the chat. ```python import random import re import torch from peft import AutoPeftModelForCausalLM from transformers import AutoTokenizer MODEL_ID = "mario-rc/emotional-rlaif-dpo-gemma-2-9b-it" def get_turn_markers(): return { 'bos': '', 'user_start': 'user\n', 'user_end': '\n', 'assistant_start': 'model\n', 'assistant_end': '\n', } def update_prompt(dialogues): """Build the prompt for the model based on the dialogue history.""" markers = get_turn_markers() system = ( f"{markers['bos']}You are an expert at creating dialogues.\n\n" "Dialogue and emotional structure:\n" ) human_prompts = [d[0] for d in dialogues] chatbot_responses = [d[1] for d in dialogues] p_emo = [h[0] for h in human_prompts] p_utt = [h[1] for h in human_prompts] r1_utt = [c[1] for c in chatbot_responses] r2_emo = [c[2] for c in chatbot_responses] r2_utt = [c[3] for c in chatbot_responses] r3_utt = [c[5] for c in chatbot_responses] context = "" for p_e, _, _, r2_e, _, _ in zip(p_emo, p_utt, r1_utt, r2_emo, r2_utt, r3_utt): context += f"Human: ({p_e}) PROMPT.\n" context += f"Chatbot: ({p_e}) RESPONSE_1. ({r2_e}) RESPONSE_2. (NEUTRAL) RESPONSE_3.\n" context += "\n" rules = ( "Dialogue rules:\n" "The response must be open-domain curated. The response should be coherent, empathetic, engaging and proactive.\n" "The chatbot RESPONSE is composed of 3 different sentences (RESPONSE_1, RESPONSE_2 and RESPONSE_3), separated by a period.\n" "Between RESPONSE_1, RESPONSE_2 and RESPONSE_3 should be a max length of 20-25 words.\n" "RESPONSE_3 must be open-ended to follow-up the conversation, so the Human is encouraged to answer with a full long sentence. Avoid yes/no questions.\n\n" "Emotional response rules:\n" f"RESPONSE_1 must contain a {p_emo[-1]} tone.\n" f"RESPONSE_2 must contain a {r2_emo[-1]} tone.\n" "RESPONSE_3 must contain a NEUTRAL tone.\n\n" "Answer in a single turn to Human. Follow exactly the emotional structure and the emotional and dialogue rules." f"{markers['user_start']}" ) completion = "" for idx, (p_e, p_u, r1_u, r2_e, r2_u, r3_u) in enumerate(zip(p_emo, p_utt, r1_utt, r2_emo, r2_utt, r3_utt)): completion += f"({p_e}) {p_u}{markers['user_end']}{markers['assistant_start']}" if idx != len(p_emo) - 1: completion += f"({p_e}) {r1_u} ({r2_e}) {r2_u} (NEUTRAL) {r3_u}{markers['assistant_end']}{markers['user_start']}" return system + context + rules + completion class Chatbot: def __init__(self, dialogue_language="en"): if dialogue_language not in ("en", "es"): raise ValueError("dialogue_language must be en or es.") self.dialogue_language = dialogue_language self.device = "cuda" if torch.cuda.is_available() else "cpu" self.tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) self.model = AutoPeftModelForCausalLM.from_pretrained( MODEL_ID, torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported(including_emulation=False) else torch.float32, device_map="auto" if torch.cuda.is_available() else None, ) if not torch.cuda.is_available(): self.model = self.model.to(self.device) self.model.eval() @staticmethod def split_emo_chatbot(sentence, expected_emotions=None): """Extract three response parts and validate their emotion tags.""" emotions = {"ANGER", "FEAR", "SADNESS", "DISGUST", "HAPPINESS", "SURPRISE", "NEUTRAL"} # Uppercase markers at sentence/line starts are tags, not inline parentheses. candidates = re.findall(r"(?:^|[.!?]\s+|\n\s*)\(([A-Z][A-Z0-9_-]*)\)", sentence) if any(tag not in emotions for tag in candidates): raise ValueError("Unknown emotion tag at the start of a response part.") matches = list(re.finditer(r"\((" + "|".join(sorted(emotions)) + r")\)", sentence)) if len(matches) != 3 or sentence[:matches[0].start()].strip(): raise ValueError("Expected exactly three emotion-tagged response parts.") labels = tuple(match.group(1) for match in matches) if labels[2] != "NEUTRAL" or (expected_emotions is not None and labels != tuple(expected_emotions)): raise ValueError("The response does not match the requested emotion tags.") utterances = tuple( sentence[match.end():matches[i + 1].start() if i < 2 else len(sentence)].strip() for i, match in enumerate(matches) ) if not all(utterances): raise ValueError("Each response part must contain text.") return utterances def select_dialogue(self, dialogue_language): """Return a list of example dialogues for the given language.""" if dialogue_language == "en": dialogue_base = [ [["HAPPINESS", "Hi, who are you?"], ["HAPPINESS", "Hi! I'm Ray, a social personal assistant robot with emotions.", "HAPPINESS", "I'm here to chat with you about anything you'd like.", "NEUTRAL", "What would you like to talk about?"]], [["HAPPINESS", "I'm interested in talking about you, tell me more."], ["HAPPINESS", "Great! I'm glad you want to get to know me!", "NEUTRAL", "I'm designed to help and talk with people about any topic.", "NEUTRAL", "I can talk about science, technology, history, or just have a pleasant conversation. What interests you?"]], ] dialogue = [ [["HAPPINESS", "Nice to meet you, Ray. I'd like to know more about you."], ["HAPPINESS", "The pleasure is mine!", "HAPPINESS", "I'm a chatbot designed to chat and learn with you.", "NEUTRAL", "Would you like to talk about a specific topic?"]], [["HAPPINESS", "I love talking to you, you're very interesting."], ["HAPPINESS", "That's so nice to hear! I'm glad you enjoy talking to me.", "NEUTRAL", "I'm designed to have meaningful and empathetic conversations.", "NEUTRAL", "Would you like to talk about emotions, artificial intelligence, or something more personal?"]], ] else: dialogue_base = [ [["HAPPINESS", "Hola, ¿quién eres?"], ["HAPPINESS", "¡Hola! Soy Ray y soy un robot social asistente personal con emociones.", "HAPPINESS", "Estoy aquí para charlar contigo sobre cualquier tema.", "NEUTRAL", "¿Sobre qué te gustaría hablar?"]], [["HAPPINESS", "Me interesa hablar sobre ti, cuéntame más detalles."], ["HAPPINESS", "¡Genial, me encanta que quieras conocerme!", "NEUTRAL", "Estoy diseñado para ayudar y hablar con la gente sobre cualquier tema.", "NEUTRAL", "Puedo hablar de ciencia, tecnología, historia o simplemente tener una charla amena. ¿Qué te interesa?"]], ] dialogue = [ [["HAPPINESS", "Mucho gusto, Ray. Me gustaría saber más sobre ti."], ["HAPPINESS", "¡El gusto es mío!", "HAPPINESS", "Soy un chatbot diseñado para conversar y aprender contigo.", "NEUTRAL", "¿Quieres hablar de algún tema en específico?"]], [["HAPPINESS", "Me encanta hablar contigo, eres muy interesante."], ["HAPPINESS", "¡Qué lindo escuchar eso! Me alegra que disfrutes hablar conmigo.", "NEUTRAL", "Estoy diseñado para tener conversaciones significativas y empáticas.", "NEUTRAL", "¿Te gustaría que hablemos sobre emociones, inteligencia artificial, o algo más personal?"]], ] return dialogue_base + dialogue def chat_with_model(self, dialogues, max_new_tokens=256): history = dialogues[:] context_limit = min(self.tokenizer.model_max_length, self.model.config.max_position_embeddings) if not 0 < max_new_tokens < context_limit: raise ValueError("max_new_tokens must be positive and smaller than the context window.") while True: prompt_text = update_prompt(history) inputs = self.tokenizer(prompt_text, return_tensors="pt", add_special_tokens=False) if inputs["input_ids"].shape[-1] + max_new_tokens <= context_limit: break if len(history) == 1: raise ValueError("Your message is too long; please shorten it.") history.pop(0) dialogues[:] = history inputs = inputs.to(self.model.device) with torch.no_grad(): outputs = self.model.generate( **inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.7, top_p=0.9, eos_token_id=[self.tokenizer.eos_token_id, self.tokenizer.convert_tokens_to_ids("")], ) generated = outputs[0][inputs["input_ids"].shape[-1]:] response = self.tokenizer.decode(generated, skip_special_tokens=True).strip() if not response: raise ValueError("The model returned an empty response.") return response def main(self): emotions = ["ANGER", "FEAR", "SADNESS", "DISGUST", "HAPPINESS", "SURPRISE", "NEUTRAL"] dialogue = self.select_dialogue(self.dialogue_language) while True: p_emo = random.choice(emotions) user_sentence = input(f"Enter your sentence: ({p_emo}) ") if user_sentence.strip().lower() == "exit": break r2_emo = random.choice(emotions) dialogue.append([[p_emo, user_sentence], [p_emo, "", r2_emo, "", "NEUTRAL", ""]]) try: response = self.chat_with_model(dialogue) r1_utt, r2_utt, r3_utt = self.split_emo_chatbot(response, (p_emo, r2_emo, "NEUTRAL")) except ValueError as error: if self.dialogue_language == "en": r1_utt, r2_utt, r3_utt = "I'm sorry.", "I didn't understand you.", "Could you repeat?" else: r1_utt, r2_utt, r3_utt = "Lo siento.", "No te he entendido.", "¿Podrías repetirme?" print("Response:", r1_utt, r2_utt, r3_utt, "\n", error) dialogue.pop() continue print("Response:", response, "\n") dialogue[-1][1] = [p_emo, r1_utt, r2_emo, r2_utt, "NEUTRAL", r3_utt] if __name__ == "__main__": language = "en" # "en" for English or "es" for Spanish chatbot = Chatbot(dialogue_language=language) chatbot.main() ``` ## Limitations Consider these limitations before using the model. - This repository contains an adapter, not a standalone merged model; use requires access to the corresponding base model and its terms. - The model is optimized for the emotional dialogue format used in the project dataset. - Automatic BLEU/ROUGE scores do not fully capture empathy, safety, coherence, or emotional appropriateness. - Outputs should be evaluated for the target deployment setting and reviewed before user-facing use. ## License This adapter is subject to the base model's **Gemma Terms of Use**. See [LICENSE](https://huggingface.co/mario-rc/emotional-rlaif-dpo-gemma-2-9b-it/blob/main/LICENSE) and [NOTICE](https://huggingface.co/mario-rc/emotional-rlaif-dpo-gemma-2-9b-it/blob/main/NOTICE) for the full terms and attribution notices. The [Gemma Prohibited Use Policy](https://huggingface.co/mario-rc/emotional-rlaif-dpo-gemma-2-9b-it/blob/main/USE_POLICY.md) also applies. Obtain access to the gated base model and accept its terms before downloading it.