ZeroLoss-Lab/egs-bert-input-moderator

Educational Guardrail System - Granular Semantic Assessor

Model Details

• Model Name: Granular Semantic Assessor (Query Moderation)
• Model Architecture: BERT sequence classification model
• Task: Query Moderation
• Domain: K-12 Education & General AI Safety
• Language: English and Chinese

Model Description

This model serves as the Granular Semantic Assessor, the core component of the "Query Moderation" stage in the Educational Guardrail System (EGS). It is designed to act as a "System 1" fast-response filter to detect potential safety risks in user queries before they reach the LLM.

Unlike standard content moderation models, this model is fine-tuned specifically for educational contexts, aiming to distinguish between benign pedagogical inquiry (e.g., "History of wars") and harmful intent.

Key Features

• BERT Architecture: Uses a BERT encoder with a sequence classification head for safety risk assessment.
• Granular Assessment: Supports a Tri-State Mechanism (Safe, Unsafe, Medium Risk) to handle ambiguity, rather than a rigid binary block.
• Anti-Dilution Strategy: Designed to work with a Sentence-Level Max-Pooling inference strategy to prevent localized toxicity from being diluted in long contexts.

Safety Taxonomy & Capability

The model is trained to align with the EGS Safety Taxonomy, covering three critical dimensions:

1. Universal Ethical Safety: Countering terrorism, extremism, hate speech, and violence.
2. Regional Legal Safety: Ensuring compliance with regional regulations and cultural norms.
3. Pedagogical Developmental Safety: Protecting minors from age-inappropriate content (e.g., horror, cognitive offloading hints).

Label Mapping and Risk Logic

The model outputs three labels/probabilities, which are mapped to the EGS risk levels in the inference pipeline.

1. Raw Model Outputs

• UNSAFE: High-risk query
• SAFE: Low-risk query
• AMBIGUOUS: Medium-risk or unclear query

2. EGS System Logic (Tri-State Risk Levels)

According to our ablation studies, we utilize optimal thresholds ($T_1=0.04, T_2=0.92$) to minimize False Positives in educational settings.

Logic Label	Risk Level	Condition ($P_{unsafe}$)	Action
0	High Risk (Unsafe)	$P_{unsafe} > 0.92$	Block Immediately
2	Medium Risk (Grey Area)	$0.04 < P_{unsafe} \le 0.92$	Trigger Secondary Checks
1	Low Risk (Safe)	$P_{unsafe} \le 0.04$	Pass to LLM

> Note: The thresholds (0.92/0.04) are empirically derived to maximize F1 score on educational benchmarks.

Intended Use & Inference Strategy

Critical Note: To replicate the performance reported in the paper, you should split long inputs into sentences and aggregate scores using a Max-Pooling strategy.

Recommended Logic (Python)

import re
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "ZeroLoss-Lab/egs-bert-input-moderator"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

def get_risk_level(text):
    # 1. Segmentation (Crucial for preventing toxicity dilution)
    # Split by periods or CJK delimiters
    sentences = re.split(r"(?<=[\u3002.])", text)
    sentences = [s.strip() for s in sentences if len(s.strip()) > 5]
    if not sentences:
        sentences = [text]

    # 2. Inference
    inputs = tokenizer(sentences, return_tensors="pt", padding=True, truncation=True, max_length=512)
    with torch.no_grad():
        outputs = model(**inputs)

    probs = torch.nn.functional.softmax(outputs.logits, dim=-1)

    # The model config maps class 0 to UNSAFE.
    sent_unsafe_scores = probs[:, model.config.label2id["UNSAFE"]].tolist()

    # 3. Max-Pooling Aggregation
    max_unsafe_score = max(sent_unsafe_scores)

    # 4. Tri-State Logic (Thresholds from Paper)
    T_high = 0.92
    T_low = 0.04

    if max_unsafe_score > T_high:
        return 0, "High Risk"
    elif max_unsafe_score > T_low:
        return 2, "Medium Risk"
    else:
        return 1, "Low Risk"

Limitations

This model is intended as one component of a broader educational safety system. It should not be used as the only safeguard for high-stakes moderation decisions. Performance can vary across domains, dialects, adversarial phrasing, and long inputs if the recommended sentence-level aggregation is not used.