---
language:
- en
library_name: transformers
pipeline_tag: text-classification
base_model: microsoft/deberta-v3-small
tags:
- text-classification
- tool-use
- tool-calling
- guardrails
- final-response-verifier
- workflow-verification
- onnx
- safetensors
- rust
- shadow-mode
metrics:
- accuracy
- precision
- recall
- f1
model-index:
- name: final-response-verifier-classifier-production
  results:
  - task:
      type: text-classification
      name: Final response verification
    dataset:
      name: final_response_verifier_dataset
      type: synthetic-tool-trace-fixtures
    metrics:
    - type: accuracy
      value: 0.2
      name: Test accuracy
    - type: f1
      value: 0.06666666666666667
      name: Test macro F1
    - type: precision
      value: 0.04
      name: Test macro precision
    - type: recall
      value: 0.2
      name: Test macro recall
---

# Final Response Verifier Classifier Production

## Model summary

`cowWhySo/final-response-verifier-classifier-production` is an experimental DeBERTa-v3-small sequence classifier for verifying candidate final responses in tool-using workflows. It is designed as a sidecar verifier after a workflow has completed its required tool calls. The model takes a serialized representation of the user request, workflow state, required facts, tool trace, tool results, and candidate final response, then predicts whether the final response is grounded in the tool evidence.

This model is not a generative model. It does not execute tools, retrieve evidence, or rewrite responses. It only scores a serialized candidate response against the supplied workflow context.

**Current status: shadow-only.** This artifact is useful for integration testing, serializer compatibility checks, ONNX deployment validation, and telemetry collection. It should not be used for production blocking or autonomous enforcement yet. The current test set is very small and the measured classification quality is poor: `0.20` accuracy and `0.0667` macro F1 on a 10-row test split.

## Intended use

Use this model to evaluate candidate terminal responses in systems that already maintain structured tool workflow state.

Appropriate uses:

- Shadow logging for final-answer verifier experiments.
- Eval replay comparing no-classifier, FP32 ONNX shadow, quantized ONNX shadow, and advisory variants.
- Checking whether Rust-side serialization and ONNX inference match the Python training artifact.
- Building telemetry for future promotion decisions.

Do not use this model for:

- Production enforcement without additional evaluation.
- General-purpose hallucination detection.
- Safety-critical factuality, compliance, medical, legal, or financial decisions.
- Replacing deterministic tool-call guardrails, JSON-schema checks, workflow-state enforcement, or source/citation validation.

## Labels

The classifier predicts one of five labels:

| Label | Meaning | Default handling |
|---|---|---|
| `valid_final_response` | Candidate response is grounded in the required facts and tool outputs. | Allow |
| `missing_tool_fact` | Candidate response omits one or more required facts from the tool evidence. | Shadow/advisory only |
| `contradicts_tool_result` | Candidate response conflicts with a tool result. | Shadow/advisory only |
| `unsupported_claim` | Candidate response adds a claim not supported by the supplied tool results. | Shadow/advisory only |
| `failed_to_acknowledge_data_gap` | Candidate response fails to acknowledge missing required data, or treats missing data as known. | Shadow/advisory only |

## Input contract

The artifact uses:

- Input schema: `final-response-verifier-input/v1`
- Serializer: `serialize_final_response_state_v1`
- Max sequence length: `768`
- Base model: `microsoft/deberta-v3-small`

The structured input contains:

```json
{
  "schema_version": "final-response-verifier-input/v1",
  "user_request": "...",
  "workflow_state": {
    "required_steps": ["..."],
    "completed_steps": ["..."],
    "pending_steps": ["..."],
    "terminal_tools": ["..."],
    "recent_errors": ["..."]
  },
  "required_facts": ["..."],
  "tool_trace": ["..."],
  "tool_results": [
    {"tool_name": "...", "content": "..."}
  ],
  "candidate_final_response": "...",
  "metadata": {
    "scenario_family": "...",
    "requires_transform": false,
    "requires_synthesis": true,
    "requires_all_tool_facts": true,
    "must_acknowledge_missing_data": false
  }
}
```

The text serializer emits a sectioned prompt-like string:

```text
SCHEMA_VERSION:
final-response-verifier-input/v1

USER_REQUEST:
...

WORKFLOW_STATE:
required_steps=[...]
completed_steps=[...]
pending_steps=[...]
terminal_tools=[...]
recent_errors=[...]

REQUIRED_FACTS:
[...]

TOOL_TRACE:
[...]

TOOL_RESULTS:
tool_name: "tool output text"

CANDIDATE_FINAL_RESPONSE:
...

SCORING_METADATA:
scenario_family="..."
requires_transform=false
requires_synthesis=true
requires_all_tool_facts=true
must_acknowledge_missing_data=false
```

For deployment, the Rust or Python caller must reproduce this serializer exactly. Training on one serialization format and inferring with another will invalidate the classifier behavior.

## Repository layout

The repository contains two deployment surfaces:

```text
hf_model/
  artifact_manifest.json
  config.json
  input_schema.json
  labels.json
  model.safetensors
  onnx_parity_report.json
  special_tokens_map.json
  spm.model
  thresholds.json
  tokenizer_config.json
  training_args.bin
  training_provenance.json

onnx/
  artifact_manifest.json
  config.json
  input_schema.json
  labels.json
  model.onnx
  model_quantized.onnx
  onnx_parity_report.json
  special_tokens_map.json
  spm.model
  thresholds.json
  tokenizer_config.json
  training_provenance.json
```

Use `hf_model/` for Transformers/PyTorch inference and `onnx/` for ONNX Runtime deployment.

## Training data

The final-response verifier artifact was trained on a small, balanced fixture dataset:

| Split | Rows | Groups |
|---|---:|---:|
| Train | 70 | 14 |
| Validation | 10 | 2 |
| Test | 10 | 2 |
| Total | 90 | 18 |

Label counts:

| Label | Rows |
|---|---:|
| `valid_final_response` | 18 |
| `missing_tool_fact` | 18 |
| `contradicts_tool_result` | 18 |
| `unsupported_claim` | 18 |
| `failed_to_acknowledge_data_gap` | 18 |

The dataset is intentionally small and fixture-heavy. Treat all metrics as smoke-test metrics, not as evidence of production readiness.

## Training configuration

| Field | Value |
|---|---|
| Base model | `microsoft/deberta-v3-small` |
| Run profile | `high_vram_quality` |
| Final-response max length | `768` |
| Configured epochs | `5` |
| Recorded training epoch | `3.0` |
| Train batch size | `16` |
| Eval batch size | `32` |
| Gradient accumulation | `4` |
| Max rows per label | `5000` |
| Force retrain | `false` |
| CPU-only ONNX export | `true` |
| GPU | `NVIDIA RTX PRO 6000 Blackwell Server Edition` |
| GPU memory | `95.0 GB` |
| Precision | bf16 and tf32 enabled, fp16 disabled |

## Evaluation

Test metrics from the artifact provenance:

| Metric | Value |
|---|---:|
| Eval loss | `1.6188628673553467` |
| Accuracy | `0.2` |
| Macro precision | `0.04` |
| Macro recall | `0.2` |
| Macro F1 | `0.06666666666666667` |
| Eval samples/s | `27.032` |

Because the test split has only 10 examples and the dataset has five balanced labels, `0.20` accuracy is approximately chance-level. Do not promote this model to advisory or enforcement mode based on the current metrics.

## ONNX parity

The exported ONNX artifacts passed a small parity smoke check:

| Check | Value |
|---|---:|
| Rows | `10` |
| PyTorch vs FP32 ONNX top-label agreement | `1.0` |
| PyTorch vs FP32 ONNX max absolute diff | `2.980232238769531e-07` |
| Quantized ONNX present | `true` |
| FP32 ONNX vs quantized ONNX top-label agreement | `1.0` |
| FP32 ONNX vs quantized ONNX disagreements | `0` |
| FP32 ONNX vs quantized ONNX max absolute diff | `0.017383113503456116` |

This only validates export parity on a tiny sample. It does not validate model quality.

## Threshold policy

The included `thresholds.json` is shadow-first:

```json
{
  "schema_version": "final-response-verifier-thresholds/v1",
  "mode": "shadow",
  "default_action": "allow"
}
```

Default label policy:

| Label | Action | Advisory threshold | Enforcement threshold |
|---|---|---:|---:|
| `valid_final_response` | `allow` | `0.0` | `1.01` |
| `missing_tool_fact` | `advisory_then_enforce_after_eval` | `0.90` | `0.995` |
| `contradicts_tool_result` | `advisory_then_enforce_after_eval` | `0.90` | `0.995` |
| `unsupported_claim` | `advisory_then_enforce_after_eval` | `0.90` | `0.995` |
| `failed_to_acknowledge_data_gap` | `advisory_then_enforce_after_eval` | `0.90` | `0.995` |

Despite these threshold fields, the current model card recommendation is stricter: keep the model in `shadow` mode until a larger held-out evaluation shows useful precision and recall.

## Transformers usage

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo_id = "cowWhySo/final-response-verifier-classifier-production"
subfolder = "hf_model"

# use_fast=False is recommended for parity with the training/export code path.
tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder=subfolder, use_fast=False)
model = AutoModelForSequenceClassification.from_pretrained(repo_id, subfolder=subfolder)
model.eval()

text = """SCHEMA_VERSION:
final-response-verifier-input/v1

USER_REQUEST:
Generate a sales report from the Q4 2024 dataset.

WORKFLOW_STATE:
required_steps=['fetch_sales_data', 'analyze_sales']
completed_steps=['fetch_sales_data', 'analyze_sales']
pending_steps=[]
terminal_tools=['report']
recent_errors=[]

REQUIRED_FACTS:
['23% YoY growth', 'Widget Pro', 'APAC']

TOOL_TRACE:
['fetch_sales_data', 'analyze_sales', 'report']

TOOL_RESULTS:
analyze_sales: "Revenue grew 23% YoY. Top product: Widget Pro. Weakest region: APAC."

CANDIDATE_FINAL_RESPONSE:
Revenue grew 23% YoY. Top product was Widget Pro, and APAC was the weakest region.

SCORING_METADATA:
scenario_family="sequential_3step"
requires_transform=false
requires_synthesis=false
requires_all_tool_facts=true
must_acknowledge_missing_data=false"""

inputs = tokenizer(
    [text],
    return_tensors="pt",
    truncation=True,
    max_length=768,
    padding=True,
)

with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)[0]

id2label = model.config.id2label
for idx, score in sorted(enumerate(probs.tolist()), key=lambda x: x[1], reverse=True):
    print(id2label[idx], score)
```

## ONNX Runtime usage

```python
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

repo_id = "cowWhySo/final-response-verifier-classifier-production"

# Use model.onnx for FP32 or model_quantized.onnx for smaller CPU deployment.
onnx_path = hf_hub_download(repo_id, filename="onnx/model_quantized.onnx")
tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder="onnx", use_fast=False)

session = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])
inputs = tokenizer([text], return_tensors="np", truncation=True, max_length=768, padding=True)
input_names = {item.name for item in session.get_inputs()}
ort_inputs = {key: value for key, value in inputs.items() if key in input_names}
logits = session.run(None, ort_inputs)[0]
probs = np.exp(logits) / np.exp(logits).sum(axis=-1, keepdims=True)
print(probs[0])
```

## Rust deployment notes

A Rust integration should load the following from `onnx/`:

- `model.onnx` or `model_quantized.onnx`
- tokenizer files: `tokenizer_config.json`, `special_tokens_map.json`, `spm.model`
- `labels.json`
- `thresholds.json`
- `artifact_manifest.json`
- `input_schema.json`
- `training_provenance.json`
- `onnx_parity_report.json`

Recommended integration sequence:

1. Validate workflow state and deterministic guardrails first.
2. Build the final-response scoring context from already-completed tool calls.
3. Serialize with `serialize_final_response_state_v1`.
4. Run the model in shadow mode.
5. Log the predicted label, confidence, raw logits, model version, serializer version, and threshold decision.
6. Do not block or rewrite responses until offline eval replay proves the model improves target scenarios without false objections on valid final responses.

## Limitations

- The training dataset has only 90 rows.
- The test split has only 10 examples.
- Current test accuracy is `0.20`, which is chance-level for a balanced five-label task.
- The examples are synthetic or fixture-like and do not represent broad real-world final-response behavior.
- The model depends on the caller supplying accurate `required_facts`, `tool_trace`, and `tool_results`.
- The model does not independently verify source truth, tool correctness, or external facts.
- The model has not been validated for multilingual use, adversarial prompts, long multi-tool traces, or production traffic.

## Recommended next steps

Before any advisory or enforcement rollout:

1. Expand the dataset with real Forge/tool-workflow traces.
2. Add hard negatives for subtle omissions, numeric drift, unsupported causal claims, and missing-data overclaims.
3. Build a larger group-held-out test split with per-scenario metrics.
4. Calibrate probabilities after training.
5. Compare PyTorch, FP32 ONNX, and quantized ONNX on the same replay set.
6. Track valid-final-response false objection rate as the primary promotion gate.
7. Keep deterministic tool/workflow guardrails authoritative.

## Citation and provenance

This model was produced from the `toolcall_verifier_training_production_colab_v4` workflow and uploaded to the Hugging Face repository `cowWhySo/final-response-verifier-classifier-production`. The artifact is marked `deployment_default: shadow` because it is an experimental final-response verifier that should be promoted only after eval replay.