🦈 sharky-0.5B

sharky is a self-hosted, open Jev/Kev-style System One decision model for network-capture vulnerability triage. Given a Wireshark capture flow rendered as hex bytes (state), it answers a fixed battery of typed questions with calibrated probabilities — in one prefill pass, no text generation. The routing decision (auto-close / review / alert) is computed in application code from those probabilities, never in a prompt.

Architecture

Component	Details
Backbone	openbmb/MiniCPM4-0.5B (433.9M params, Apache-2.0), frozen
Adaptation	kev-style LoRA r=16, α=32, dropout 0.05 on all attention + MLP projections (q/k/v/o/gate/up/down_proj)
Readout	pointer head: query/key projections → scaled dot-product → softmax directly over the option tokens. The probabilities are the output.
Inference	prefill-only, block-causal question isolation (each question sees the state, never a sibling question's answer) — the full battery in one forward pass
Calibration	temperature T=1.7411, fitted on 8,420 held-out calibration records (calibration/temperature.json), applied at load

Output battery

The battery is fixed — it is exactly what the checkpoint was trained on.

Field	Type	Meaning
verdict	choice	benign / suspicious / malicious
exploit_evidence	noul	p(yes): direct evidence of an exploit attempt?
recon_only	noul	p(yes): scan/recon only, no exploit attempt?
severity	score 0–3	4 ordered levels: benign → low → elevated → critical

Training data

luispoveda93/sharky-triage-states — per-flow states rendered as hex bytes (not parsed prose):

• 49,745 from rdpahalavan/UNSW-NB15 Packet-Bytes (v1 corpus)
• 23,814 additional held-out UNSW files (expansion)
• 8,324 real-world DAPT-2020 enterprise-traffic flows (per-flow labels matched from CICFlowMeter CSVs)

The combined v2 training pool is 75,017 records (md5-hash splits per class; the 2,446-state locked split is excluded from all training). IPs are anonymized to stable per-flow tokens (IP_A, IP_B) to prevent dataset-specific shortcuts.

Training

This checkpoint is a warm-start continuation of the v1 checkpoint (luispoveda93/minicpm4-pcap-kev-v1, adapter sha256 a19aa880…c66c939), with the pointer head carried over frozen (head_lr=0):

• lr 4e-5, weight decay 0.01, 2 epochs (150,160 records seen = 75,017 × 2), 9,386 steps, effective batch 16 (4 × accum 4)
• ordinal objective: ranked probability score (no label smoothing, focal or Brier terms)
• bf16 compute / fp32 weights; ~3.6 h wall on a single CUDA device
• 0 rejected / 0 truncated records

Evaluation (never-trained sets)

Calibrated results (T=1.7411), from sharky-workbench/phase6/eval_v1v2.json:

Set	n	Accuracy	Macro-F1	ECE	Conf. error (p≥0.9 & wrong)
Locked split	2,446	0.9865	0.9846	0.0064	0.45%
Held-out UNSW files (F6+F16, OOD)	2,100	0.9905	0.9898	0.0053	0.29%
Real-world DAPT-2020 (OOD hold)	3,674	0.9848	0.9830	0.0041	0.30%

• Locked-split per-class F1: benign 0.9995 · suspicious 0.9732 · malicious 0.9810
• Option-order flips: 0/300
• Gate numbers (measured in the demo Space's evaluation): benign auto-close 40% coverage @ 99.9% precision, malicious escalate ~32% coverage @ 99.1% precision

Benchmark results (phase7) — kev-family head-to-head

The phase7 benchmark plan of record (sharky-workbench/phase7/benchmark_plan.md, rev 2) has been executed. All arms run through kev's own predictor stack (kev.predictors.LocalPredictor) on the same battery and state strings: the locked split (n=2,446) and the real-world DAPT-2020 OOD hold (n=3,674), calibrated verdict probabilities at each arm's fitted temperature. Sharky's temperature re-audit refit landed on exactly T=1.7411 — zero drift vs the card value. Bootstrap CIs (record-paired, 5,000 resamples) from phase7/eval_bootstrap_p0.json; full artifacts in sharky-workbench/phase7/ (leaderboard.md, eval_pcap_headtohead.json, perrecord_*.json, temp_refit.json, manifests.json).

Arm	locked acc	locked mF1	locked ECE	DAPT-OOD acc	DAPT mF1	DAPT ECE
sharky-0.5B (this model)	0.9865	0.9846	0.0064	0.9848	0.9830	0.0041
untrained-head control (same checkpoint, LoRA zeroed)	0.4350	0.2848	0.5361	0.4543	0.2110	0.5415
kev-0.5b zero-shot	0.4007	0.1907	0.2182	0.4584	0.2095	0.1879
kev-0.6b zero-shot	0.4007	0.1907	0.4366	0.4584	0.2095	0.3780
kev-4b zero-shot	pending	pending	pending	pending	pending	pending

• Paired bootstrap (delta = sharky − arm, 95% CI): every baseline sits 0.53–0.59 below sharky on both sets — control +0.5515 [+0.5315, +0.5711] (locked), kev-0.5b +0.5857 [+0.5662, +0.6051], kev-0.6b +0.5860 [+0.5666, +0.6051]; all CIs exclude zero by a wide margin.
• Untrained-head control collapses to chance (0.435 / 0.454): the triage LoRA does essentially all of the work; the MiniCPM backbone contributes no zero-shot triage ability.
• Zero-shot kev baselines collapse to a constant benign answer: argmax benign on 100% of all 6,120 records (a distribution-level check on raw predictor probabilities rules out a harness artifact — kev-0.6b answers benign with mean p=0.83). No kev/Jev model has published pcap numbers; zero-shot transfer of the served kev checkpoints to this battery fails outright. kev-0.6b keeps its role as the indirect Jev anchor (its published head-to-head: 0.805 in-dist / 0.598 OOD vs Jev's 0.845 / 0.855 — neither measured on pcap).
• Gate curves (sharky, calibrated): benign auto-close 40.1% coverage @ 99.9% precision (locked — independently reproduces the demo-Space gate numbers) and 41.8% @ 100% at τ=0.99 (DAPT); max auto-close coverage subject to ≤5% error: 45.8% (DAPT, precision 0.984). Escalate (suspicious+malicious) precision ≥ 0.987 on both sets. The kev-4b gate-discipline comparison (34% at 5% error) closes when the kev-4b arm lands.
• kev-4b arm: pending — five consecutive GPU jobs failed at container init (MOUNT_ATTR_IDMAP on the NVIDIA MPS binary) across a10g-small, l4x1 and t4-small; the failure is image-independent and platform-side. The harness and command are ready (ARMS=kev-4b); this row and the gate comparison fill in when GPU infra heals.
• Jev API head-to-head: deferred — no API access; the protocol is preserved in the plan's appendix and runs unchanged when access lands. P2 (shared public suites, sst2/ag_news/banking77) is likewise still open.

---

Usage (serving)

Served with kev's TypeSafe-compatible POST /v1/systemone contract. The live demo Space (luispoveda93/sharky) exposes both a Gradio UI (pcap upload + playground) and the API; typesafe-sdk clients work by pointing base_url at the Space.

curl -X POST https://luispoveda93-sharky.hf.space/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{"model":"sharky-0.5B","state":"### Capture summary\n3 packets, TCP IP_A:445 -> IP_B:1337, mean payload 32B\n\n### Packet hex (up to 64 bytes each)\n01: <hex>\n02: <hex>","questions":{"verdict":{"type":"choice","criteria":{"benign":"routine traffic with no hostile behaviour","suspicious":"reconnaissance/scanning without an exploit attempt","malicious":"direct evidence of an exploit attempt or malicious payload"}}}}'

GET /v1/models returns the serving model card (run, base, device, dtype, temperature, prefix-cache stats).

Deployment pattern (the gate, not the decider)

Following the Jev field-study lessons: confident verdicts act, uncertain ones route to deeper analysis — thresholds live in application code, never in a prompt.

• benign at p≥0.8 → auto-close
• otherwise → human review
• malicious at p≥0.8 → alert, human-confirmed

Honest caveats

• Research model, not a production security tool. Labels derive from UNSW-NB15 (lab-generated 2015 traffic, imperfect labels, ~97% benign background) plus DAPT-2020; expect degraded accuracy on distributions neither dataset covers.
• Synthetic hand-built pcaps are out-of-distribution and can produce inverted verdicts (measured on the v1 checkpoint); treat any verdict on unfamiliar traffic as human-review input only.
• The SUSPICIOUS↔MALICIOUS boundary is the dominant error mode: 32 of the 33 locked-split errors (from the confusion matrix: 12 suspicious→malicious, 20 malicious→suspicious, plus 1 benign→malicious).
• Provenance: trained as minicpm4-pcap-kev-v2 and renamed to sharky-0.5B (weights identical); the minicpm4-pcap-kev-v1/-v2 repos remain for lineage, and the old pcap-kev-triage demo Space now redirects to luispoveda93/sharky.