sevra-qwen3-1.7b-gate

This repository contains the SEVRA recoverability gate based on Qwen/Qwen3-1.7B. It is a 4-bit QLoRA-trained sequence-classification adapter that predicts whether candidate-specific active verification is likely to repair a completed base attempt. It does not solve the original task itself.

SEVRA is introduced in Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning. Source code and serving abstractions are available at Sajib-006/SEVRA.

Intended use

Use the gate after a base solver has produced an attempt. The gate receives the problem, attempted solution, and serving-visible features; a calibrated threshold decides whether to accept the base answer or invoke active verification. Gold answers are unavailable to the gate at inference time.

Development metrics

The adapter was trained on MATH recovery outcomes with an 80/20 example-level split.

Metric	Value
Development accuracy	0.9000
Development AUROC	0.9570
Development AUPRC	0.7534
Frozen operating threshold	0.022995
Development policy accuracy	0.9275
Development intervention rate	0.2700
Development harmful-flip rate	0.0075

These are gate-development results, not standalone solver accuracy. See the paper for MATH500, GSM8K, CommonsenseQA, budget-matched, and harmful-flip evaluations.

Loading

import torch
from peft import PeftConfig, PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo_id = "Sajib-006/sevra-qwen3-1.7b-gate"
adapter = PeftConfig.from_pretrained(repo_id)
tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
base = AutoModelForSequenceClassification.from_pretrained(
    adapter.base_model_name_or_path,
    num_labels=1,
    problem_type="multi_label_classification",
    torch_dtype="auto",
    device_map="auto",
)
base.config.pad_token_id = tokenizer.pad_token_id
model = PeftModel.from_pretrained(base, repo_id).eval()

Use sevra.format_gate_input from the GitHub package to construct inputs. Apply sigmoid to the single output logit and use the frozen threshold reported above unless recalibrating on a separate development workload.

Training configuration

• Objective: binary helpful-fix prediction
• Base model: Qwen/Qwen3-1.7B
• Adapter: LoRA rank 16, alpha 32, dropout 0.05
• Target modules: attention and MLP projection layers
• Epochs: 3
• Maximum sequence length: 1,536
• Class weighting: inverse helpful-fix frequency
• Checkpoint selection: development AUPRC
• Threshold selection: development policy accuracy, then lower action-token cost

Limitations

The gate was trained on English mathematical reasoning produced by a frozen Qwen3-4B solver. Transfer to other solvers, languages, prompting formats, and workloads should be recalibrated and audited for harmful flips. Gate scores are recoverability estimates, not correctness probabilities.

Citation

@article{dip2026think,
  title={Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning},
  author={Dip, Sajib Acharjee and Zhou, Dawei and Zhang, Liqing},
  journal={arXiv preprint arXiv:2606.19808},
  year={2026}
}