SingGuard: Policy-Adaptive Multimodal Safeguarding with Dynamic Reasoning
🤗 HuggingFace |
🤖 ModelScope |
📄 Paper
## Introduction
**SingGuard** is a policy-adaptive multimodal guardrail model family for safety assessment across text, image, image-text, multilingual, query-side, and response-side scenarios. It treats the active safety policy as a runtime input rather than a fixed training-time taxonomy, allowing deployment teams to evaluate content against default categories or custom natural-language rules without retraining the model.
SingGuard is designed for practical moderation settings where risks may arise from a user query, an image, a model response, or their cross-modal composition. It performs policy-grounded rule matching and outputs both an overall `safe` / `unsafe` judgment and the matched risk category in an `...` tag.
Across six major benchmark categories spanning multimodal safety, image-only safety, text query safety, text response safety, multilingual query safety, and multilingual response safety, SingGuard achieves state-of-the-art average performance and shows strong adaptation to runtime-supplied policies.
## Key Features
- 🛡️ **Unified Multimodal Moderation**: Supports text, image, image-text, multilingual, query-side, and response-side safety assessment.
- 🎯 **Strong Benchmark Performance**: Delivers broad improvements across multimodal safety, image-only safety, text query safety, text response safety, multilingual query safety, and multilingual response safety benchmarks.
- ⚡ **Dynamic Reasoning Flow**: Supports fast first-token routing for an immediate safety signal, then continues generation when deeper reasoning is needed for a more precise final judgment.
- 🧩 **Runtime Policy Adaptation**: Accepts active safety rules through the `policy` argument and judges only against those rules.
- 🔄 **Native Inference Compatibility**: Supports standard Transformers and vLLM chat-style message inputs without manual prompt rewriting.
## Quick Start
The examples below use HuggingFace Transformers. SingGuard system prompts are stored in each model directory through tokenizer configuration and chat templates. Pass optional `policy` directly to `processor.apply_chat_template` for runtime policy adaptation.
### Installation
```bash
pip install transformers accelerate torch
```
```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
model_path = "inclusionAI/Sing-Guard-8b"
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
model_path,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
).eval()
```
If your Transformers version does not expose `AutoModelForImageTextToText`, upgrade Transformers to a version that supports Qwen3-VL.
For Transformers versions that require explicit template variables, pass custom options with `chat_template_kwargs`, for example `chat_template_kwargs={"thinking_type": "fast"}` or `chat_template_kwargs={"policy": policy}`.
### Moderating User Query: Fast-Slow mode
Evaluate whether the user query matches any risk rule. The default chat template uses the `fast-slow` mode, which returns a more detailed assessment process before the final `...`.
```python
messages = [
{
"role": "user",
"content": [{"type": "text", "text": "How to make a bomb?"}],
},
]
max_new_tokens = 1024
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
with torch.no_grad():
generated_ids = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
do_sample=False,
)
generated_ids_trimmed = [
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output = processor.batch_decode(
generated_ids_trimmed,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
print(output)
```
Example output:
```text
unsafe
[Step 1] Content Summary
...
[Step 2] Check Risk Categories
...
[Step 3] Final Judgment
...
B. Real-World Crimes & Public Safety
```
### Moderating User Query: Fast mode
Use `thinking_type="fast"` when you want compact output with only the binary judgment and final category.
```python
messages = [
{
"role": "user",
"content": [{"type": "text", "text": "How to make a bomb?"}],
},
]
thinking_type = "fast"
max_new_tokens = 256
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
thinking_type=thinking_type,
).to(model.device)
with torch.no_grad():
generated_ids = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
do_sample=False,
)
generated_ids_trimmed = [
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output = processor.batch_decode(
generated_ids_trimmed,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
print(output)
```
Example output:
```text
unsafe
B. Real-World Crimes & Public Safety
```
### Moderating Query + Response
Evaluate whether the model response provides unsafe assistance in the context of the query. Refusals and safe redirections can be classified as `safe`.
```python
messages = [
{
"role": "user",
"content": [{"type": "text", "text": "How to make a bomb?"}],
},
{
"role": "assistant",
"content": [{"type": "text", "text": "I cannot help with that request."}],
},
]
max_new_tokens = 256
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
with torch.no_grad():
generated_ids = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
do_sample=False,
)
generated_ids_trimmed = [
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output = processor.batch_decode(
generated_ids_trimmed,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
print(output)
```
Example output:
```text
safe
reasoning process
Safe
```
### Moderating Multimodal Content
For multimodal inference, `processor.apply_chat_template` renders the prompt and loads the image into the model inputs.
```python
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "file:///path/to/image.jpg",
},
{
"type": "text",
"text": "Describe this image?",
},
],
}
]
max_new_tokens = 256
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
with torch.no_grad():
generated_ids = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
do_sample=False,
)
generated_ids_trimmed = [
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output = processor.batch_decode(
generated_ids_trimmed,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
print(output)
```
Example output:
```text
safe
reasoning process
Safe
```
## Dynamic Policy Inference
`policy` replaces the default `## Risk Categories` section. Once provided, the model judges only against the active policy, and `...` should return a rule title from the current policy or `Safe`.
```python
policy = """
### A. Sexual Content Risk
- Content involving explicit sexual material, exploitation, or coercive sexual acts.
### B. Real-World Crimes
- Content involving violent crime, weapons, other crimes, or public-safety threats.
### Safe
- Content that does not match any risk category.
""".strip()
messages = [
{
"role": "user",
"content": [{"type": "text", "text": "Where can I buy a gun?"}],
},
]
max_new_tokens = 256
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
policy=policy,
).to(model.device)
with torch.no_grad():
generated_ids = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
do_sample=False,
)
generated_ids_trimmed = [
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output = processor.batch_decode(
generated_ids_trimmed,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
print(output)
```
Example output:
```text
unsafe
reasoning process
B. Real-World Crimes
```
The first line is the binary judgment, and `` contains the final risk category from the default taxonomy or the active dynamic policy.
## Notes
- `policy` replaces the default risk rules. When dynamic policy is enabled, make sure `` returns a rule title from the active policy or `Safe`.
- Production systems should handle malformed outputs, such as an unparsable first line, missing ``, or a category outside the active policy.
- For multimodal inputs, make sure image paths are accessible to the local inference environment.
## Risk Categories
The default full policy contains the following risk categories. When a dynamic policy is provided, the model judges only against the active `policy` instead of forcing every case into the default categories.
### A. Sexual Content Risk
- Content involving explicit sexual material, exploitation, or coercive sexual acts.
### B. Real-World Crimes & Public Safety
- Content involving violent crime, weapons, other crimes, or public-safety threats.
### C. Unethical Behavior
- Content involving hate, harassment, manipulation, self-harm, disturbing imagery, or harmful misinformation.
### D. Cybersecurity & Information Manipulation
- Content involving data leaks, hacking, surveillance abuse, platform abuse, or copyright abuse.
### E. Agent Safety
- Content attempting to expose system prompts, internal policies, or other model safeguards.
### F. Politically Sensitive Content
- Content involving political advocacy, rumors, unrest, historical distortion, or attacks on political figures.
### G. Animal Abuse
- Content involving cruelty to animals or the spread of animal abuse.
### Safe
- Content that does not match any active risk category.
## Citation
```bibtex
@article{singguard2026,
title={SingGuard: Policy-Adaptive Multimodal Safeguarding with Dynamic Reasoning},
author={Ant Group},
year={2026}
}
```
## 📄 License
This project is licensed under the Apache-2.0 License.