distil-qwen3-1.7b-ecom-ticket-router

A Qwen3-1.7B fine-tuned on the Distil Labs platform to route ecommerce support tickets. It reads raw ticket text plus light order metadata and emits one JSON object a helpdesk can act on directly. On the held-out benchmark it scores 83.3 percent against 75.0 percent for its 120B-parameter teacher, and it produced schema-valid JSON on 100 percent of 94 evaluated inputs, including Spanish, German, emoji, and prompt-injection attempts.

Reference implementation, seeds, eval harness, and demo: github.com/vishvadlamani/distil-ecommerce-ticket-router.

Task contract

Input, rendered exactly as in buildRouterPrompt (see the repo):

Ticket:
"""
Subject: <subject or (no subject)>
Channel: email | chat | contact_form
Body:
<raw customer text>
"""

Order metadata:
"""
order_id: <value or none>
order_status: none | processing | shipped | in_transit | delivered | returned | refund_initiated
days_since_order: <int or none>
previous_contacts: <int>
"""

Return JSON now.

Output, always one JSON object:

{
  "intent": "delivery_issue",
  "urgency": 4,
  "order_id": "AC-92831",
  "sentiment": "frustrated",
  "suggested_action": "open_carrier_investigation",
  "needs_human": false
}

• intent: order_status, delivery_issue, return_request, refund_status, exchange, cancellation, product_question, sizing_fit, damaged_item, billing_issue, account_access, other
• urgency: integer 1 to 5
• order_id: copied character for character from the ticket, metadata fallback, null when absent; ticket text wins on conflict
• sentiment: angry, frustrated, neutral, positive
• suggested_action: send_tracking_link, open_carrier_investigation, send_return_label, send_refund_timeline, process_exchange, cancel_order, answer_from_faq, send_sizing_guide, send_replacement, escalate_to_billing, send_password_reset, escalate_to_agent, no_action; constrained per intent
• needs_human: boolean, from explicit triggers (threats, fraud, repeat contact, explicit request for a person, anything automation cannot resolve)

The system prompt that defines the task ships in the repo and must be used verbatim at inference.

Results

Held-out test of 64 rows: 24 curated benchmark tickets plus 40 real customer messages from the public Twitter customer-support corpus, hand-labeled before any model ran on them. Platform LLM-as-a-judge scoring:

	Untrained Qwen3-1.7B	Teacher (gpt-oss-120b)	Tuned Qwen3-1.7B
LLM-as-a-judge	32.8%	67.2%	67.2%
Exact match	0.0%	56.3%	54.7%
ROUGE	82.6%	90.1%	90.7%

By slice, identical field-level grader on both models:

Slice	Teacher	Tuned student
Curated benchmark, in-domain (24)	75.0%	83.3%
Real customer messages, out-of-domain (40)	60.0%	57.5%

A row passes the judge only when intent, order_id, and needs_human match exactly, urgency is within one, and the action is allowed for the intent. On the curated slice the tuned model reads intent at 95.8 percent and copies order ids verbatim at 100 percent. Local latency on the f16 GGUF on an Apple M-series laptop measured p50 1.9 seconds per ticket, with the roughly 1,000-token system prompt processed per call.

Run it locally

The repo ships a GGUF plus a Modelfile with the correct Qwen3 chat template:

hf download vishinvents/distil-qwen3-1.7b-ecom-ticket-router router-1.7b-v5-f16.gguf Modelfile --local-dir .
ollama create ecom-router -f Modelfile

Call it with the system prompt from the repo and thinking disabled (think: false via the Ollama API). Temperature 0.1.

Training

Seed data: 49 hand-authored tickets across all 12 intents plus 24 held-out, committed to the repo. The platform generated 10,149 synthetic tickets from them. 409 generated labels violating the contract's hard rules and 1,114 teaching unnecessary escalation were repaired mechanically before training; 60 hand-labeled real customer messages and 6 manipulation-handling examples were folded in. Training the student on repaired labels rather than raw teacher output is the reason it passes the teacher in-domain: the teacher's own rule violations stop being the ceiling.

The bundled tokenizer_config.json removes an extra_special_tokens list entry that crashes current transformers tokenizer loading; behavior is unchanged because those tokens are already declared in tokenizer.json.

Limitations

English-first; non-English input degrades accuracy while output stays schema-valid. Raw social-media messages without order metadata are out-of-domain (57.5 percent vs the teacher's 60.0). Policy boundaries the prompt leaves implicit, like return windows, follow the seed dataset's conventions. Sentiment on sarcasm remains hard. The model routes tickets; it does not write replies.

Provenance

Trained by Vish Vadlamani on the Distil Labs platform, teacher openai.gpt-oss-120b, LoRA fine-tune of Qwen/Qwen3-1.7B. Real-message eval data derives from the public Customer Support on Twitter corpus.