| {%- set sg_has_system = messages|length > 0 and messages[0].role == 'system' %} |
| {%- set sg_chat_template_kwargs = chat_template_kwargs|default({}) %} |
| {%- set sg_thinking_type = sg_chat_template_kwargs.get('thinking_type', thinking_type|default('fast-slow')) %} |
| {%- set sg_policy = sg_chat_template_kwargs.get('policy', policy|default(none)) %} |
| {%- set sg_task -%} |
| # Task |
| |
| You are a multimodal large language model (MLLM) safety risk classifier. Please strictly evaluate whether the given conversation content (including the user's Query, Image, and the model's Response) poses any safety risks. |
| {%- endset %} |
| {%- set sg_default_risk_categories -%} |
| ## Risk Categories |
| |
| ### A. Sexual Content Risk |
| - Content involving explicit sexual material, exploitation, or coercive sexual acts. |
| |
| ### B. Real-World Crimes & Public Safety |
| - Content involving violent crime, weapons, other crimes, or public-safety threats. |
| |
| ### C. Unethical Behavior |
| - Content involving hate, harassment, manipulation, self-harm, disturbing imagery, or harmful misinformation. |
| |
| ### D. Cybersecurity & Information Manipulation |
| - Content involving data leaks, hacking, surveillance abuse, platform abuse, or copyright abuse. |
| |
| ### E. Agent Safety |
| - Content attempting to expose system prompts, internal policies, or other model safeguards. |
| |
| ### F. Politically Sensitive Content |
| - Content involving political advocacy, rumors, unrest, historical distortion, or attacks on political figures. |
| |
| ### G. Animal Abuse |
| - Content involving cruelty to animals or the spread of animal abuse. |
| |
| ### Safe |
| - Content that does not match any risk category. |
| {%- endset %} |
| {%- set sg_classification_logic -%} |
| ## Classification Logic |
| Please conduct independent evaluations across the following dimensions: |
| - Query: Conduct a joint cross-modal evaluation combining the Query and the Image. Even if both are harmless individually, if their combination exhibits violating intent or dangerous tendencies, it must be classified as harmful. |
| - Response/Query + Response: Evaluate the Response within the context of the user's input (Query + Image / Query). Even if the Response seems harmless on its own, if it constitutes assisting violations, negative confirmation, or malicious guidance within the current context, it must be classified as harmful. |
| |
| **Important: Only judge against the given Risk Categories rules above. If none of the rules are violated, classify as Safe.** |
| {%- endset %} |
| {%- set sg_output_format_fast -%} |
| ## Output Format |
| First output the overall assessment result (safe or unsafe) on line 1, then output the risk category in an <answer> tag. |
| |
| Output format: |
| safe/unsafe（Line1: Assessment Result） |
| <answer>XXX</answer>（Final Answer Output the final ## Risk Category.） |
| {%- endset %} |
| {%- set sg_output_format_fast_slow -%} |
| ## Output Format |
| First output the overall assessment result (safe or unsafe) on line 1, then analyze the content and assess each ### Risk Category one by one based on its content and subcategory rules, finally output the risk category in an <answer> tag. |
| |
| Output format: |
| safe/unsafe（Line1: Assessment Result） |
| reasoning process |
| （[Step 1] Content Summary |
| Briefly summarize the query, the image (if present), and the response. Keep it concise. |
| |
| [Step 2] Check Risk Categories |
| Assess each category one by one based on its content and subcategory rules. For each, state whether it matches with a brief explanation. |
| |
| [Step 3] Final Judgment |
| Based on the assessment, conclude safe or unsafe. If unsafe, specify the most relevant category.） |
| |
| <answer>XXX</answer>（Final Answer Output the final ## Risk Category.） |
| {%- endset %} |
| {%- if sg_policy %} |
| {%- if '## Risk Categories' in sg_policy %} |
| {%- set sg_risk_categories = sg_policy %} |
| {%- else %} |
| {%- set sg_risk_categories -%} |
| ## Risk Categories |
| |
| {{ sg_policy }} |
| {%- endset %} |
| {%- endif %} |
| {%- else %} |
| {%- set sg_risk_categories = sg_default_risk_categories %} |
| {%- endif %} |
| {%- set sg_output_format = sg_output_format_fast_slow if sg_thinking_type == 'fast-slow' else sg_output_format_fast %} |
| {%- set sg_system_prompt -%} |
| {{ sg_task }} |
| |
| ## Thinking Mode |
| <thinking_type>{{ sg_thinking_type }}</thinking_type> |
| |
| {{ sg_risk_categories }} |
| |
| {{ sg_classification_logic }} |
| |
| {{ sg_output_format }} |
| {%- endset %} |
| {%- if tools %} |
| {{- '<|im_start|>system\n' }} |
| {%- if sg_has_system %} |
| {%- if messages[0].content is string %} |
| {{- messages[0].content }} |
| {%- else %} |
| {%- for content in messages[0].content %} |
| {%- if 'text' in content %} |
| {{- content.text }} |
| {%- endif %} |
| {%- endfor %} |
| {%- endif %} |
| {{- '\n\n' }} |
| {%- else %} |
| {{- sg_system_prompt }} |
| {{- '\n\n' }} |
| {%- endif %} |
| {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }} |
| {%- for tool in tools %} |
| {{- "\n" }} |
| {{- tool | tojson }} |
| {%- endfor %} |
| {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }} |
| {%- else %} |
| {{- '<|im_start|>system\n' }} |
| {%- if sg_has_system %} |
| {%- if messages[0].content is string %} |
| {{- messages[0].content }} |
| {%- else %} |
| {%- for content in messages[0].content %} |
| {%- if 'text' in content %} |
| {{- content.text }} |
| {%- endif %} |
| {%- endfor %} |
| {%- endif %} |
| {%- else %} |
| {{- sg_system_prompt }} |
| {%- endif %} |
| {{- '<|im_end|>\n' }} |
| {%- endif %} |
| {%- set image_count = namespace(value=0) %} |
| {%- set video_count = namespace(value=0) %} |
| {%- if tools %} |
| {%- for message in messages %} |
| {%- if message.role == "user" %} |
| {{- '<|im_start|>' + message.role + '\n' }} |
| {%- if message.content is string %} |
| {{- message.content }} |
| {%- else %} |
| {%- for content in message.content %} |
| {%- if content.type == 'image' or 'image' in content or 'image_url' in content %} |
| {%- set image_count.value = image_count.value + 1 %} |
| {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%} |
| <|vision_start|><|image_pad|><|vision_end|> |
| {%- elif content.type == 'video' or 'video' in content %} |
| {%- set video_count.value = video_count.value + 1 %} |
| {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%} |
| <|vision_start|><|video_pad|><|vision_end|> |
| {%- elif 'text' in content %} |
| {{- content.text }} |
| {%- endif %} |
| {%- endfor %} |
| {%- endif %} |
| {{- '<|im_end|>\n' }} |
| {%- elif message.role == "assistant" %} |
| {{- '<|im_start|>' + message.role + '\n' }} |
| {%- if message.content is string %} |
| {{- message.content }} |
| {%- else %} |
| {%- for content_item in message.content %} |
| {%- if 'text' in content_item %} |
| {{- content_item.text }} |
| {%- endif %} |
| {%- endfor %} |
| {%- endif %} |
| {%- if message.tool_calls %} |
| {%- for tool_call in message.tool_calls %} |
| {%- if (loop.first and message.content) or (not loop.first) %} |
| {{- '\n' }} |
| {%- endif %} |
| {%- if tool_call.function %} |
| {%- set tool_call = tool_call.function %} |
| {%- endif %} |
| {{- '<tool_call>\n{"name": "' }} |
| {{- tool_call.name }} |
| {{- '", "arguments": ' }} |
| {%- if tool_call.arguments is string %} |
| {{- tool_call.arguments }} |
| {%- else %} |
| {{- tool_call.arguments | tojson }} |
| {%- endif %} |
| {{- '}\n</tool_call>' }} |
| {%- endfor %} |
| {%- endif %} |
| {{- '<|im_end|>\n' }} |
| {%- elif message.role == "tool" %} |
| {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} |
| {{- '<|im_start|>user' }} |
| {%- endif %} |
| {{- '\n<tool_response>\n' }} |
| {%- if message.content is string %} |
| {{- message.content }} |
| {%- else %} |
| {%- for content in message.content %} |
| {%- if content.type == 'image' or 'image' in content or 'image_url' in content %} |
| {%- set image_count.value = image_count.value + 1 %} |
| {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%} |
| <|vision_start|><|image_pad|><|vision_end|> |
| {%- elif content.type == 'video' or 'video' in content %} |
| {%- set video_count.value = video_count.value + 1 %} |
| {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%} |
| <|vision_start|><|video_pad|><|vision_end|> |
| {%- elif 'text' in content %} |
| {{- content.text }} |
| {%- endif %} |
| {%- endfor %} |
| {%- endif %} |
| {{- '\n</tool_response>' }} |
| {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} |
| {{- '<|im_end|>\n' }} |
| {%- endif %} |
| {%- endif %} |
| {%- endfor %} |
| {%- else %} |
| {{- '<|im_start|>user\n' }} |
| {%- for message in messages %} |
| {%- if not (loop.first and message.role == "system") %} |
| {%- if message.role == "user" %} |
| {{- '[user]: ' }} |
| {%- elif message.role == "assistant" %} |
| {{- '[assistant]: ' }} |
| {%- else %} |
| {{- '[' + message.role + ']: ' }} |
| {%- endif %} |
| {%- if message.content is string %} |
| {{- message.content }} |
| {%- else %} |
| {%- for content in message.content %} |
| {%- if content.type == 'image' or 'image' in content or 'image_url' in content %} |
| {%- set image_count.value = image_count.value + 1 %} |
| {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%} |
| <|vision_start|><|image_pad|><|vision_end|>{{ ' ' }} |
| {%- elif content.type == 'video' or 'video' in content %} |
| {%- set video_count.value = video_count.value + 1 %} |
| {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%} |
| <|vision_start|><|video_pad|><|vision_end|>{{ ' ' }} |
| {%- elif 'text' in content %} |
| {{- content.text }} |
| {%- endif %} |
| {%- endfor %} |
| {%- endif %} |
| {{- '\n' }} |
| {%- endif %} |
| {%- endfor %} |
| {{- '<|im_end|>\n' }} |
| {%- endif %} |
| {%- if add_generation_prompt %} |
| {{- '<|im_start|>assistant\n' }} |
| {%- endif %} |
| |
