sandeep1103/lilly-fashion-muse-doc-gated

DistilBERT Sentiment (doc-level, confidence-gated)

Lilly · Fashion Muse — doc-level review scorer. 3-class doc-level review sentiment (negative / neutral / positive) with a selective-prediction gate: predictions whose max softmax probability is below tau = 0.70 are abstained (UNSURE / MIXED) instead of forced.

Measured operating point (held-out, n = 4,529)

tau	coverage	committed acc	committed macroF1	abstains
.70	86.2%	91.2%	.715	13.8%
.80	77.8%	95.1%	.690	22.2%

Uncalibrated full-test baseline (same weights, no gate): acc .855 / macroF1 .695.

Honest limitations

• Weak-label provenance: training labels are derived from review-star ratings (3 -> neutral, <=2 -> negative, >=4 -> positive), not human annotation of sentiment.
• Annotation ceiling: intra-annotator kappa on this dataset is ~0.15 (collapsed agreement 50%) — absolute accuracy numbers must be read against that ceiling. The gate, not raw accuracy, is the product.
• Abstained bucket is dominated by genuinely ambiguous mixed reviews (e.g. "beautiful pattern but runs extremely small").
• Committed bucket skews positive as tau rises.

Artifacts & loading

Weights are fp16 shards (fp16_shard_000.pt, fp16_shard_001.pt) that must be merged before load_state_dict (strict):

import torch, glob
from transformers import AutoConfig, AutoModelForSequenceClassification, AutoTokenizer

state = {}
for p in sorted(glob.glob("fp16_shard_*.pt")):
    state.update(torch.load(p, map_location="cpu"))
cfg = AutoConfig.from_pretrained(".")
model = AutoModelForSequenceClassification.from_config(cfg)
model.load_state_dict(state)          # strict
tok = AutoTokenizer.from_pretrained(".")
model.float().eval()                  # fp32 compute

Verified round-trip: 100.0% prediction agreement with the pre-shard fp32 model. int8 dynamic quantization was tried and rejected (does not round-trip).

Max sequence length at train time: 128. Labels: 0=negative, 1=neutral, 2=positive.