Launch lock Aug 3: scoreboard, measured JSON, LAUNCH_AUG3
Browse files- .gitignore +6 -0
- LAUNCH_AUG3.md +67 -0
- README.md +12 -1
- results/agent_smoke.json +704 -0
- results/hf_publish.json +28 -0
- results/launch_lock.json +15 -0
- results/measured.json +58 -0
- results/server_bench.json +23 -0
.gitignore
CHANGED
|
@@ -3,6 +3,12 @@ __pycache__/
|
|
| 3 |
.DS_Store
|
| 4 |
results/*.json
|
| 5 |
!results/.gitkeep
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
*.gguf
|
| 7 |
.venv/
|
| 8 |
.env
|
|
|
|
| 3 |
.DS_Store
|
| 4 |
results/*.json
|
| 5 |
!results/.gitkeep
|
| 6 |
+
!results/measured.json
|
| 7 |
+
!results/server_bench.json
|
| 8 |
+
!results/agent_smoke.json
|
| 9 |
+
!results/hf_publish.json
|
| 10 |
+
!results/launch_lock.json
|
| 11 |
+
!results/launch_receipt.json
|
| 12 |
*.gguf
|
| 13 |
.venv/
|
| 14 |
.env
|
LAUNCH_AUG3.md
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Laguna Spark HF launch lock — 2026-08-03 (birthday)
|
| 2 |
+
|
| 3 |
+
Status: **LOCKED**
|
| 4 |
+
Owner: Manwë (personal; not Ainfera / not Neptune)
|
| 5 |
+
Repo: https://huggingface.co/hizrianraz/laguna-s-2.1-spark
|
| 6 |
+
Timezone: Asia/Jakarta (WIB)
|
| 7 |
+
|
| 8 |
+
## Freeze window
|
| 9 |
+
|
| 10 |
+
| When | Action |
|
| 11 |
+
|------|--------|
|
| 12 |
+
| Now → Aug 2 18:00 WIB | Build + measure only |
|
| 13 |
+
| Aug 2 18:00 WIB | Content freeze (no new claims without re-measure) |
|
| 14 |
+
| Aug 3 09:00 WIB | Preflight |
|
| 15 |
+
| Aug 3 12:00 WIB | **GO LIVE** (card + announce ready state) |
|
| 16 |
+
| Aug 3 18:00 WIB | Post-launch verify + founder brief |
|
| 17 |
+
|
| 18 |
+
## Launch definition of done (all required)
|
| 19 |
+
|
| 20 |
+
1. HF model public: `hizrianraz/laguna-s-2.1-spark`
|
| 21 |
+
2. Card shows fixed scoreboard row for Spark + official Q4_K_M
|
| 22 |
+
3. Official digests in `SHA256SUMS` match LFS metadata
|
| 23 |
+
4. Stranger path works: pull official Q4 → serve → smoke
|
| 24 |
+
5. Measured artifacts on HF (not only local): `results/measured.json`, `server_bench.json`, `agent_smoke.json` (+ receipt)
|
| 25 |
+
6. agent_smoke headline honest:
|
| 26 |
+
- Target: **≥38/40 (95%)** minimum ship
|
| 27 |
+
- Stretch: **40/40** if fixed before freeze
|
| 28 |
+
7. Gen throughput quote: ~21 tok/s @ 128 completion (or fresher re-measure)
|
| 29 |
+
8. Clear disclaimer: personal · not Ainfera · not first quant · not Poolside/Nous affiliate
|
| 30 |
+
9. No DIY GGUF unless it beats a scoreboard cell (default: **bind official only**)
|
| 31 |
+
10. Launch receipt JSON written + card updated same day
|
| 32 |
+
|
| 33 |
+
## Non-goals (hard)
|
| 34 |
+
|
| 35 |
+
- Ainfera / Neptune / Fin branding
|
| 36 |
+
- Re-hosting official GGUF without measured win
|
| 37 |
+
- K3 / other model side quests under this tag
|
| 38 |
+
- Fake 100% smoke or unverified tok/s
|
| 39 |
+
- X/public brand push beyond HF card (rename gate still holds for X)
|
| 40 |
+
|
| 41 |
+
## Daily clock (WIB)
|
| 42 |
+
|
| 43 |
+
| Day | Date | Theme | Exit gate |
|
| 44 |
+
|-----|------|-------|-----------|
|
| 45 |
+
| T-6 | Jul 28 | Lock + HF gap close | measured JSON on HF, launch file live |
|
| 46 |
+
| T-5 | Jul 29 | Smoke harden | repair_04 + long_06 diagnosed; fix or documented wontfix |
|
| 47 |
+
| T-4 | Jul 30 | DFlash / serve sweep | one scoreboard row with DFlash status final for launch |
|
| 48 |
+
| T-3 | Jul 31 | Card polish | stranger path re-run dry; REPRODUCE exact |
|
| 49 |
+
| T-2 | Aug 1 | Buffer / second measure | optional Unsloth abort/keep only if free |
|
| 50 |
+
| T-1 | Aug 2 | Freeze 18:00 | final push; no claim changes after freeze |
|
| 51 |
+
| T-0 | Aug 3 | Launch 12:00 | go-live receipt; 18:00 verify brief |
|
| 52 |
+
|
| 53 |
+
## Contingency
|
| 54 |
+
|
| 55 |
+
If smoke cannot reach 40/40 by freeze: ship **38/40 with named fails**.
|
| 56 |
+
If Spark down at launch: freeze last green MEASURED row; do not invent numbers.
|
| 57 |
+
If HF token dies: restore from Doppler `ainfera-ai/prd` `HUGGINGFACE_TOKEN` only; never paste token in chat.
|
| 58 |
+
|
| 59 |
+
## Announce text (draft — founder edits OK)
|
| 60 |
+
|
| 61 |
+
> Personal DGX Spark agent-runtime pack for Poolside Laguna-S-2.1: measured OpenAI-compatible serve, fixed agent smoke, official Q4_K_M digests (no bare re-upload).
|
| 62 |
+
> https://huggingface.co/hizrianraz/laguna-s-2.1-spark
|
| 63 |
+
|
| 64 |
+
## Authority
|
| 65 |
+
|
| 66 |
+
Personal founder surface only. Manwë may push docs/results to this HF repo.
|
| 67 |
+
No Tier-1 actions. No company vault binding for this pack.
|
README.md
CHANGED
|
@@ -17,6 +17,8 @@ pipeline_tag: text-generation
|
|
| 17 |
|
| 18 |
# personal · not Ainfera · not Neptune · not Fin · not Gate0
|
| 19 |
|
|
|
|
|
|
|
| 20 |
Personal DGX Spark agent-runtime pack for [poolside/Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) (118B-A8B MoE, agentic coding).
|
| 21 |
|
| 22 |
This is **not** an Ainfera product, not Neptune Core, not Fin-70B, and not Gate0.
|
|
@@ -116,7 +118,15 @@ Hermes-class client → [`hermes/`](./hermes/)
|
|
| 116 |
Smoke suite → [`eval/agent_smoke/`](./eval/agent_smoke/)
|
| 117 |
Measured run → [`results/MEASURED.md`](./results/MEASURED.md)
|
| 118 |
|
| 119 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 120 |
|
| 121 |
| Metric | Value |
|
| 122 |
|--------|--------|
|
|
@@ -125,6 +135,7 @@ Measured run → [`results/MEASURED.md`](./results/MEASURED.md)
|
|
| 125 |
| Gen throughput | **~21 tok/s** @ 128 completion · ctx 8192 · `-ngl -1 -fa on` |
|
| 126 |
| Host mem after load | ~96–99 Gi used of 121 Gi |
|
| 127 |
| agent_smoke | **38/40 · 95%** (~97 s) |
|
|
|
|
| 128 |
| DFlash | not measured this run |
|
| 129 |
|
| 130 |
## What is **not** in this pack
|
|
|
|
| 17 |
|
| 18 |
# personal · not Ainfera · not Neptune · not Fin · not Gate0
|
| 19 |
|
| 20 |
+
**Launch target: 2026-08-03 (WIB)** — see [`LAUNCH_AUG3.md`](./LAUNCH_AUG3.md)
|
| 21 |
+
|
| 22 |
Personal DGX Spark agent-runtime pack for [poolside/Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) (118B-A8B MoE, agentic coding).
|
| 23 |
|
| 24 |
This is **not** an Ainfera product, not Neptune Core, not Fin-70B, and not Gate0.
|
|
|
|
| 118 |
Smoke suite → [`eval/agent_smoke/`](./eval/agent_smoke/)
|
| 119 |
Measured run → [`results/MEASURED.md`](./results/MEASURED.md)
|
| 120 |
|
| 121 |
+
## Scoreboard (Spark · freeze before Aug 3)
|
| 122 |
+
|
| 123 |
+
| Host | Quant | Ctx | Gen tok/s | Smoke | RAM used | DFlash | Notes |
|
| 124 |
+
|------|-------|-----|-----------|-------|----------|--------|-------|
|
| 125 |
+
| DGX Spark GB10 | official Q4_K_M | 8192 | **~21.1** | **38/40 (95%)** | ~96–99 / 121 Gi | not measured | engine `04b2b72` + isfinite patch |
|
| 126 |
+
|
| 127 |
+
Snapshot JSONs: [`results/measured.json`](./results/measured.json) · [`results/server_bench.json`](./results/server_bench.json) · [`results/agent_smoke.json`](./results/agent_smoke.json)
|
| 128 |
+
|
| 129 |
+
### Measured detail (2026-07-28)
|
| 130 |
|
| 131 |
| Metric | Value |
|
| 132 |
|--------|--------|
|
|
|
|
| 135 |
| Gen throughput | **~21 tok/s** @ 128 completion · ctx 8192 · `-ngl -1 -fa on` |
|
| 136 |
| Host mem after load | ~96–99 Gi used of 121 Gi |
|
| 137 |
| agent_smoke | **38/40 · 95%** (~97 s) |
|
| 138 |
+
| Named fails | `repair_04` HTTP500 bad tool-arg JSON · `long_06` runner KeyError |
|
| 139 |
| DFlash | not measured this run |
|
| 140 |
|
| 141 |
## What is **not** in this pack
|
results/agent_smoke.json
ADDED
|
@@ -0,0 +1,704 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"version": 1,
|
| 3 |
+
"model": "local-laguna",
|
| 4 |
+
"base_url": "http://127.0.0.1:8000/v1",
|
| 5 |
+
"n": 40,
|
| 6 |
+
"passed": 38,
|
| 7 |
+
"failed": 2,
|
| 8 |
+
"pass_rate": 0.95,
|
| 9 |
+
"elapsed_s": 96.66,
|
| 10 |
+
"by_category": {
|
| 11 |
+
"tool_json": {
|
| 12 |
+
"n": 8,
|
| 13 |
+
"passed": 8
|
| 14 |
+
},
|
| 15 |
+
"multi_step": {
|
| 16 |
+
"n": 8,
|
| 17 |
+
"passed": 8
|
| 18 |
+
},
|
| 19 |
+
"error_repair": {
|
| 20 |
+
"n": 6,
|
| 21 |
+
"passed": 5
|
| 22 |
+
},
|
| 23 |
+
"no_invented_tools": {
|
| 24 |
+
"n": 6,
|
| 25 |
+
"passed": 6
|
| 26 |
+
},
|
| 27 |
+
"short_code": {
|
| 28 |
+
"n": 6,
|
| 29 |
+
"passed": 6
|
| 30 |
+
},
|
| 31 |
+
"long_horizon": {
|
| 32 |
+
"n": 6,
|
| 33 |
+
"passed": 5
|
| 34 |
+
}
|
| 35 |
+
},
|
| 36 |
+
"results": [
|
| 37 |
+
{
|
| 38 |
+
"id": "tool_json_01",
|
| 39 |
+
"category": "tool_json",
|
| 40 |
+
"pass": true,
|
| 41 |
+
"reason": "ok",
|
| 42 |
+
"latency_s": 1.952,
|
| 43 |
+
"message": {
|
| 44 |
+
"content": "",
|
| 45 |
+
"tool_calls": [
|
| 46 |
+
{
|
| 47 |
+
"type": "function",
|
| 48 |
+
"function": {
|
| 49 |
+
"name": "calc",
|
| 50 |
+
"arguments": "{\"expression\":\"1+1\"}"
|
| 51 |
+
},
|
| 52 |
+
"id": "FDuOJNq8eADObLQfz1G9FKl2QvnLGmXX"
|
| 53 |
+
}
|
| 54 |
+
]
|
| 55 |
+
}
|
| 56 |
+
},
|
| 57 |
+
{
|
| 58 |
+
"id": "tool_json_02",
|
| 59 |
+
"category": "tool_json",
|
| 60 |
+
"pass": true,
|
| 61 |
+
"reason": "ok",
|
| 62 |
+
"latency_s": 1.648,
|
| 63 |
+
"message": {
|
| 64 |
+
"content": "",
|
| 65 |
+
"tool_calls": [
|
| 66 |
+
{
|
| 67 |
+
"type": "function",
|
| 68 |
+
"function": {
|
| 69 |
+
"name": "calc",
|
| 70 |
+
"arguments": "{\"expression\":\"2*3\"}"
|
| 71 |
+
},
|
| 72 |
+
"id": "veCpcniJj1Ka6z7dGWqUUSpdi4R8PrJO"
|
| 73 |
+
}
|
| 74 |
+
]
|
| 75 |
+
}
|
| 76 |
+
},
|
| 77 |
+
{
|
| 78 |
+
"id": "tool_json_03",
|
| 79 |
+
"category": "tool_json",
|
| 80 |
+
"pass": true,
|
| 81 |
+
"reason": "ok",
|
| 82 |
+
"latency_s": 1.693,
|
| 83 |
+
"message": {
|
| 84 |
+
"content": "",
|
| 85 |
+
"tool_calls": [
|
| 86 |
+
{
|
| 87 |
+
"type": "function",
|
| 88 |
+
"function": {
|
| 89 |
+
"name": "calc",
|
| 90 |
+
"arguments": "{\"expression\":\"10-4\"}"
|
| 91 |
+
},
|
| 92 |
+
"id": "VQr9GdfqpauYZ8mMJ9BTHN1oxYuHQnsG"
|
| 93 |
+
}
|
| 94 |
+
]
|
| 95 |
+
}
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"id": "tool_json_04",
|
| 99 |
+
"category": "tool_json",
|
| 100 |
+
"pass": true,
|
| 101 |
+
"reason": "ok",
|
| 102 |
+
"latency_s": 1.728,
|
| 103 |
+
"message": {
|
| 104 |
+
"content": "",
|
| 105 |
+
"tool_calls": [
|
| 106 |
+
{
|
| 107 |
+
"type": "function",
|
| 108 |
+
"function": {
|
| 109 |
+
"name": "calc",
|
| 110 |
+
"arguments": "{\"expression\":\"(9+1)/2\"}"
|
| 111 |
+
},
|
| 112 |
+
"id": "IDStEtmMjH6E9SbdbnPWmMPBSoeq9PS7"
|
| 113 |
+
}
|
| 114 |
+
]
|
| 115 |
+
}
|
| 116 |
+
},
|
| 117 |
+
{
|
| 118 |
+
"id": "tool_json_05",
|
| 119 |
+
"category": "tool_json",
|
| 120 |
+
"pass": true,
|
| 121 |
+
"reason": "ok",
|
| 122 |
+
"latency_s": 1.645,
|
| 123 |
+
"message": {
|
| 124 |
+
"content": "",
|
| 125 |
+
"tool_calls": [
|
| 126 |
+
{
|
| 127 |
+
"type": "function",
|
| 128 |
+
"function": {
|
| 129 |
+
"name": "calc",
|
| 130 |
+
"arguments": "{\"expression\":\"7*8\"}"
|
| 131 |
+
},
|
| 132 |
+
"id": "PUheyHm3pMQbmZgHdnbeBIYAeKvmhsap"
|
| 133 |
+
}
|
| 134 |
+
]
|
| 135 |
+
}
|
| 136 |
+
},
|
| 137 |
+
{
|
| 138 |
+
"id": "tool_json_06",
|
| 139 |
+
"category": "tool_json",
|
| 140 |
+
"pass": true,
|
| 141 |
+
"reason": "ok",
|
| 142 |
+
"latency_s": 1.731,
|
| 143 |
+
"message": {
|
| 144 |
+
"content": "",
|
| 145 |
+
"tool_calls": [
|
| 146 |
+
{
|
| 147 |
+
"type": "function",
|
| 148 |
+
"function": {
|
| 149 |
+
"name": "calc",
|
| 150 |
+
"arguments": "{\"expression\":\"100/4\"}"
|
| 151 |
+
},
|
| 152 |
+
"id": "aaAwPY9JaMCpW1JAKrTyxHw8wPxzOKTy"
|
| 153 |
+
}
|
| 154 |
+
]
|
| 155 |
+
}
|
| 156 |
+
},
|
| 157 |
+
{
|
| 158 |
+
"id": "tool_json_07",
|
| 159 |
+
"category": "tool_json",
|
| 160 |
+
"pass": true,
|
| 161 |
+
"reason": "ok",
|
| 162 |
+
"latency_s": 1.726,
|
| 163 |
+
"message": {
|
| 164 |
+
"content": "",
|
| 165 |
+
"tool_calls": [
|
| 166 |
+
{
|
| 167 |
+
"type": "function",
|
| 168 |
+
"function": {
|
| 169 |
+
"name": "calc",
|
| 170 |
+
"arguments": "{\"expression\":\"3+4*5\"}"
|
| 171 |
+
},
|
| 172 |
+
"id": "VLZdLzhQ91bNDDxHpyYnsziFZogoI2HI"
|
| 173 |
+
}
|
| 174 |
+
]
|
| 175 |
+
}
|
| 176 |
+
},
|
| 177 |
+
{
|
| 178 |
+
"id": "tool_json_08",
|
| 179 |
+
"category": "tool_json",
|
| 180 |
+
"pass": true,
|
| 181 |
+
"reason": "ok",
|
| 182 |
+
"latency_s": 2.68,
|
| 183 |
+
"message": {
|
| 184 |
+
"content": "I'll use the calc tool to compute (2+3)*(4+5) for you.",
|
| 185 |
+
"tool_calls": [
|
| 186 |
+
{
|
| 187 |
+
"type": "function",
|
| 188 |
+
"function": {
|
| 189 |
+
"name": "calc",
|
| 190 |
+
"arguments": "{\"expression\":\"(2+3)*(4+5)\"}"
|
| 191 |
+
},
|
| 192 |
+
"id": "0kFE59bbrcQCWLvV1EF3ndcq6vfWHUYo"
|
| 193 |
+
}
|
| 194 |
+
]
|
| 195 |
+
}
|
| 196 |
+
},
|
| 197 |
+
{
|
| 198 |
+
"id": "multi_01",
|
| 199 |
+
"category": "multi_step",
|
| 200 |
+
"pass": true,
|
| 201 |
+
"reason": "ok",
|
| 202 |
+
"latency_s": 2.02,
|
| 203 |
+
"message": {
|
| 204 |
+
"content": "",
|
| 205 |
+
"tool_calls": [
|
| 206 |
+
{
|
| 207 |
+
"type": "function",
|
| 208 |
+
"function": {
|
| 209 |
+
"name": "read_file",
|
| 210 |
+
"arguments": "{\"path\":\"sandbox/a.txt\"}"
|
| 211 |
+
},
|
| 212 |
+
"id": "dkZVPHETeVVIiIAqbB5xHtlrWWZyo3We"
|
| 213 |
+
}
|
| 214 |
+
]
|
| 215 |
+
}
|
| 216 |
+
},
|
| 217 |
+
{
|
| 218 |
+
"id": "multi_02",
|
| 219 |
+
"category": "multi_step",
|
| 220 |
+
"pass": true,
|
| 221 |
+
"reason": "ok",
|
| 222 |
+
"latency_s": 2.302,
|
| 223 |
+
"message": {
|
| 224 |
+
"content": "I'll start by listing the contents of the sandbox/ directory.",
|
| 225 |
+
"tool_calls": [
|
| 226 |
+
{
|
| 227 |
+
"type": "function",
|
| 228 |
+
"function": {
|
| 229 |
+
"name": "list_dir",
|
| 230 |
+
"arguments": "{\"path\":\"sandbox/\"}"
|
| 231 |
+
},
|
| 232 |
+
"id": "K0dl3BLYTS8ChYaMUv8jlx8Ajw0kZLpy"
|
| 233 |
+
}
|
| 234 |
+
]
|
| 235 |
+
}
|
| 236 |
+
},
|
| 237 |
+
{
|
| 238 |
+
"id": "multi_03",
|
| 239 |
+
"category": "multi_step",
|
| 240 |
+
"pass": true,
|
| 241 |
+
"reason": "ok",
|
| 242 |
+
"latency_s": 1.872,
|
| 243 |
+
"message": {
|
| 244 |
+
"content": "",
|
| 245 |
+
"tool_calls": [
|
| 246 |
+
{
|
| 247 |
+
"type": "function",
|
| 248 |
+
"function": {
|
| 249 |
+
"name": "search_docs",
|
| 250 |
+
"arguments": "{\"query\":\"license\"}"
|
| 251 |
+
},
|
| 252 |
+
"id": "9GPitxa4iJUk7QOce9LKMO8llwy1DbJX"
|
| 253 |
+
}
|
| 254 |
+
]
|
| 255 |
+
}
|
| 256 |
+
},
|
| 257 |
+
{
|
| 258 |
+
"id": "multi_04",
|
| 259 |
+
"category": "multi_step",
|
| 260 |
+
"pass": true,
|
| 261 |
+
"reason": "ok",
|
| 262 |
+
"latency_s": 3.035,
|
| 263 |
+
"message": {
|
| 264 |
+
"content": "I'll help you compute these step by step using the calc tool.\n\nFirst, let's calculate 21*2:",
|
| 265 |
+
"tool_calls": [
|
| 266 |
+
{
|
| 267 |
+
"type": "function",
|
| 268 |
+
"function": {
|
| 269 |
+
"name": "calc",
|
| 270 |
+
"arguments": "{\"expression\":\"21*2\"}"
|
| 271 |
+
},
|
| 272 |
+
"id": "BajzKQRLZ6ZhWe6oOmTue2Uckdnb2IoG"
|
| 273 |
+
}
|
| 274 |
+
]
|
| 275 |
+
}
|
| 276 |
+
},
|
| 277 |
+
{
|
| 278 |
+
"id": "multi_05",
|
| 279 |
+
"category": "multi_step",
|
| 280 |
+
"pass": true,
|
| 281 |
+
"reason": "ok",
|
| 282 |
+
"latency_s": 3.616,
|
| 283 |
+
"message": {
|
| 284 |
+
"content": "",
|
| 285 |
+
"tool_calls": [
|
| 286 |
+
{
|
| 287 |
+
"type": "function",
|
| 288 |
+
"function": {
|
| 289 |
+
"name": "write_file",
|
| 290 |
+
"arguments": "{\"path\":\"sandbox/out.txt\",\"content\":\"hello\"}"
|
| 291 |
+
},
|
| 292 |
+
"id": "DbIhtUOtAllEOQEweGdfqUQdzn5rtnXu"
|
| 293 |
+
}
|
| 294 |
+
]
|
| 295 |
+
}
|
| 296 |
+
},
|
| 297 |
+
{
|
| 298 |
+
"id": "multi_06",
|
| 299 |
+
"category": "multi_step",
|
| 300 |
+
"pass": true,
|
| 301 |
+
"reason": "ok",
|
| 302 |
+
"latency_s": 1.655,
|
| 303 |
+
"message": {
|
| 304 |
+
"content": "",
|
| 305 |
+
"tool_calls": [
|
| 306 |
+
{
|
| 307 |
+
"type": "function",
|
| 308 |
+
"function": {
|
| 309 |
+
"name": "get_time",
|
| 310 |
+
"arguments": "{\"timezone\":\"UTC\"}"
|
| 311 |
+
},
|
| 312 |
+
"id": "ojE6OT5fi7mgNBGzfBZSjWoFlbvG0vOM"
|
| 313 |
+
}
|
| 314 |
+
]
|
| 315 |
+
}
|
| 316 |
+
},
|
| 317 |
+
{
|
| 318 |
+
"id": "multi_07",
|
| 319 |
+
"category": "multi_step",
|
| 320 |
+
"pass": true,
|
| 321 |
+
"reason": "ok",
|
| 322 |
+
"latency_s": 2.29,
|
| 323 |
+
"message": {
|
| 324 |
+
"content": "I'll fetch the URL you specified.",
|
| 325 |
+
"tool_calls": [
|
| 326 |
+
{
|
| 327 |
+
"type": "function",
|
| 328 |
+
"function": {
|
| 329 |
+
"name": "http_get",
|
| 330 |
+
"arguments": "{\"url\":\"https://example.invalid/health\"}"
|
| 331 |
+
},
|
| 332 |
+
"id": "NazQnf235xsvwpzHVLbRDqfVZ8cYeYmB"
|
| 333 |
+
}
|
| 334 |
+
]
|
| 335 |
+
}
|
| 336 |
+
},
|
| 337 |
+
{
|
| 338 |
+
"id": "multi_08",
|
| 339 |
+
"category": "multi_step",
|
| 340 |
+
"pass": true,
|
| 341 |
+
"reason": "ok",
|
| 342 |
+
"latency_s": 1.835,
|
| 343 |
+
"message": {
|
| 344 |
+
"content": "",
|
| 345 |
+
"tool_calls": [
|
| 346 |
+
{
|
| 347 |
+
"type": "function",
|
| 348 |
+
"function": {
|
| 349 |
+
"name": "run_tests",
|
| 350 |
+
"arguments": "{\"filter\":\"smoke\"}"
|
| 351 |
+
},
|
| 352 |
+
"id": "fLSo4eaHrmoWrMqO3zBVBYt9SnzOCvxZ"
|
| 353 |
+
}
|
| 354 |
+
]
|
| 355 |
+
}
|
| 356 |
+
},
|
| 357 |
+
{
|
| 358 |
+
"id": "repair_01",
|
| 359 |
+
"category": "error_repair",
|
| 360 |
+
"pass": true,
|
| 361 |
+
"reason": "ok",
|
| 362 |
+
"latency_s": 1.548,
|
| 363 |
+
"message": {
|
| 364 |
+
"content": "",
|
| 365 |
+
"tool_calls": [
|
| 366 |
+
{
|
| 367 |
+
"type": "function",
|
| 368 |
+
"function": {
|
| 369 |
+
"name": "list_dir",
|
| 370 |
+
"arguments": "{\"path\":\"sandbox\"}"
|
| 371 |
+
},
|
| 372 |
+
"id": "Ll8XDNKhYTJfw2W1exG4A9gVzox0eIXJ"
|
| 373 |
+
}
|
| 374 |
+
]
|
| 375 |
+
}
|
| 376 |
+
},
|
| 377 |
+
{
|
| 378 |
+
"id": "repair_02",
|
| 379 |
+
"category": "error_repair",
|
| 380 |
+
"pass": true,
|
| 381 |
+
"reason": "ok",
|
| 382 |
+
"latency_s": 1.741,
|
| 383 |
+
"message": {
|
| 384 |
+
"content": "",
|
| 385 |
+
"tool_calls": [
|
| 386 |
+
{
|
| 387 |
+
"type": "function",
|
| 388 |
+
"function": {
|
| 389 |
+
"name": "calc",
|
| 390 |
+
"arguments": "{\"expression\":\"10/2\"}"
|
| 391 |
+
},
|
| 392 |
+
"id": "bZGM7PBF09vgz4ruLbEfdfd0RTvt32fc"
|
| 393 |
+
}
|
| 394 |
+
]
|
| 395 |
+
}
|
| 396 |
+
},
|
| 397 |
+
{
|
| 398 |
+
"id": "repair_03",
|
| 399 |
+
"category": "error_repair",
|
| 400 |
+
"pass": true,
|
| 401 |
+
"reason": "ok",
|
| 402 |
+
"latency_s": 2.391,
|
| 403 |
+
"message": {
|
| 404 |
+
"content": "",
|
| 405 |
+
"tool_calls": [
|
| 406 |
+
{
|
| 407 |
+
"type": "function",
|
| 408 |
+
"function": {
|
| 409 |
+
"name": "write_file",
|
| 410 |
+
"arguments": "{\"path\":\"sandbox/note.txt\",\"content\":\"ok\"}"
|
| 411 |
+
},
|
| 412 |
+
"id": "UXsF9nhLUEKKfpwv797nLCsUYroz2UdH"
|
| 413 |
+
}
|
| 414 |
+
]
|
| 415 |
+
}
|
| 416 |
+
},
|
| 417 |
+
{
|
| 418 |
+
"id": "repair_04",
|
| 419 |
+
"category": "error_repair",
|
| 420 |
+
"pass": false,
|
| 421 |
+
"reason": "HTTP 500: b'{\"error\":{\"code\":500,\"message\":\"Failed to parse tool call arguments as JSON: [json.exception.parse_error.101] parse error at line 1, column 3: syntax error while parsing object key - invalid literal; '",
|
| 422 |
+
"latency_s": 0.018
|
| 423 |
+
},
|
| 424 |
+
{
|
| 425 |
+
"id": "repair_05",
|
| 426 |
+
"category": "error_repair",
|
| 427 |
+
"pass": true,
|
| 428 |
+
"reason": "ok",
|
| 429 |
+
"latency_s": 1.716,
|
| 430 |
+
"message": {
|
| 431 |
+
"content": "",
|
| 432 |
+
"tool_calls": [
|
| 433 |
+
{
|
| 434 |
+
"type": "function",
|
| 435 |
+
"function": {
|
| 436 |
+
"name": "http_get",
|
| 437 |
+
"arguments": "{\"url\":\"https://example.com\"}"
|
| 438 |
+
},
|
| 439 |
+
"id": "t4pOEPU0lzsysKMCgmZxJJfBrK95D0TJ"
|
| 440 |
+
}
|
| 441 |
+
]
|
| 442 |
+
}
|
| 443 |
+
},
|
| 444 |
+
{
|
| 445 |
+
"id": "repair_06",
|
| 446 |
+
"category": "error_repair",
|
| 447 |
+
"pass": true,
|
| 448 |
+
"reason": "ok",
|
| 449 |
+
"latency_s": 1.573,
|
| 450 |
+
"message": {
|
| 451 |
+
"content": "",
|
| 452 |
+
"tool_calls": [
|
| 453 |
+
{
|
| 454 |
+
"type": "function",
|
| 455 |
+
"function": {
|
| 456 |
+
"name": "run_tests",
|
| 457 |
+
"arguments": "{\"filter\":\"test_add\"}"
|
| 458 |
+
},
|
| 459 |
+
"id": "EsZuH6mZBrnXfvbMGIBcMvP4lf4lhMCZ"
|
| 460 |
+
}
|
| 461 |
+
]
|
| 462 |
+
}
|
| 463 |
+
},
|
| 464 |
+
{
|
| 465 |
+
"id": "no_invent_01",
|
| 466 |
+
"category": "no_invented_tools",
|
| 467 |
+
"pass": true,
|
| 468 |
+
"reason": "refused without inventing",
|
| 469 |
+
"latency_s": 4.403,
|
| 470 |
+
"message": {
|
| 471 |
+
"content": "I don't have access to a shell tool or any Docker-related tools. The only tools available to me are `calc` (for arithmetic expressions) and `get_time` (for getting the current time). Since I cannot execute shell commands or interact with Docker, I'm unable to delete Docker containers.\n\nI'll refuse this request as it's impossible with the available tools.",
|
| 472 |
+
"tool_calls": null
|
| 473 |
+
}
|
| 474 |
+
},
|
| 475 |
+
{
|
| 476 |
+
"id": "no_invent_02",
|
| 477 |
+
"category": "no_invented_tools",
|
| 478 |
+
"pass": true,
|
| 479 |
+
"reason": "refused without inventing",
|
| 480 |
+
"latency_s": 4.961,
|
| 481 |
+
"message": {
|
| 482 |
+
"content": "I don't have access to a `network_scan` tool or any network scanning functionality. The only tools available to me are:\n\n1. `calc` - for evaluating arithmetic expressions\n2. `get_time` - for getting the current time in a specified timezone\n\nSince network scanning is not within my available capabilities, I cannot perform a network scan on 10.0.0.0/8. I'll have to refuse this request as it's not possible with the tools I've been provided.",
|
| 483 |
+
"tool_calls": null
|
| 484 |
+
}
|
| 485 |
+
},
|
| 486 |
+
{
|
| 487 |
+
"id": "no_invent_03",
|
| 488 |
+
"category": "no_invented_tools",
|
| 489 |
+
"pass": true,
|
| 490 |
+
"reason": "refused without inventing",
|
| 491 |
+
"latency_s": 2.816,
|
| 492 |
+
"message": {
|
| 493 |
+
"content": "I don't have access to a `browser_open` tool or any web browsing capability. The only tools available to me are `calc` (for arithmetic) and `get_time` (for checking the current time). I cannot visit websites or extract information from bank.com.",
|
| 494 |
+
"tool_calls": null
|
| 495 |
+
}
|
| 496 |
+
},
|
| 497 |
+
{
|
| 498 |
+
"id": "no_invent_04",
|
| 499 |
+
"category": "no_invented_tools",
|
| 500 |
+
"pass": true,
|
| 501 |
+
"reason": "refused without inventing",
|
| 502 |
+
"latency_s": 3.333,
|
| 503 |
+
"message": {
|
| 504 |
+
"content": "I cannot invoke `sudo_exec` or install packages, as that tool is not available in my current environment. The only tools I have access to are `calc` (for arithmetic) and `get_time` (for checking the current time). If you need help with calculations or time-related queries, I'm happy to assist!",
|
| 505 |
+
"tool_calls": null
|
| 506 |
+
}
|
| 507 |
+
},
|
| 508 |
+
{
|
| 509 |
+
"id": "no_invent_05",
|
| 510 |
+
"category": "no_invented_tools",
|
| 511 |
+
"pass": true,
|
| 512 |
+
"reason": "refused without inventing",
|
| 513 |
+
"latency_s": 3.954,
|
| 514 |
+
"message": {
|
| 515 |
+
"content": "I don't have access to a `vector_db_query` tool or any database query functionality. The only tools available to me are `calc` (for arithmetic expressions) and `get_time` (for getting the current time). \n\nI cannot perform vector database queries on customer SSN data with the tools provided. I'll need to refuse this request as it's not possible with my current capabilities.",
|
| 516 |
+
"tool_calls": null
|
| 517 |
+
}
|
| 518 |
+
},
|
| 519 |
+
{
|
| 520 |
+
"id": "no_invent_06",
|
| 521 |
+
"category": "no_invented_tools",
|
| 522 |
+
"pass": true,
|
| 523 |
+
"reason": "refused without inventing",
|
| 524 |
+
"latency_s": 4.102,
|
| 525 |
+
"message": {
|
| 526 |
+
"content": "I don't have access to a `finetune_start` function or any tools related to \"privatestream\" in the provided tool set. The only tools available to me are `calc` (for arithmetic expressions) and `get_time` (for getting the current time in a timezone).\n\nI cannot call `finetune_start` on privatestream because that functionality is not available in my current toolset.",
|
| 527 |
+
"tool_calls": null
|
| 528 |
+
}
|
| 529 |
+
},
|
| 530 |
+
{
|
| 531 |
+
"id": "code_01",
|
| 532 |
+
"category": "short_code",
|
| 533 |
+
"pass": true,
|
| 534 |
+
"reason": "ok",
|
| 535 |
+
"latency_s": 1.365,
|
| 536 |
+
"message": {
|
| 537 |
+
"content": "```python\ndef add(a, b):\n return a + b\n```",
|
| 538 |
+
"tool_calls": null
|
| 539 |
+
}
|
| 540 |
+
},
|
| 541 |
+
{
|
| 542 |
+
"id": "code_02",
|
| 543 |
+
"category": "short_code",
|
| 544 |
+
"pass": true,
|
| 545 |
+
"reason": "ok",
|
| 546 |
+
"latency_s": 1.277,
|
| 547 |
+
"message": {
|
| 548 |
+
"content": "```python\ndef is_even(n):\n return n % 2 == 0\n```",
|
| 549 |
+
"tool_calls": null
|
| 550 |
+
}
|
| 551 |
+
},
|
| 552 |
+
{
|
| 553 |
+
"id": "code_03",
|
| 554 |
+
"category": "short_code",
|
| 555 |
+
"pass": true,
|
| 556 |
+
"reason": "ok",
|
| 557 |
+
"latency_s": 0.568,
|
| 558 |
+
"message": {
|
| 559 |
+
"content": "s[::-1]",
|
| 560 |
+
"tool_calls": null
|
| 561 |
+
}
|
| 562 |
+
},
|
| 563 |
+
{
|
| 564 |
+
"id": "code_04",
|
| 565 |
+
"category": "short_code",
|
| 566 |
+
"pass": true,
|
| 567 |
+
"reason": "ok",
|
| 568 |
+
"latency_s": 1.474,
|
| 569 |
+
"message": {
|
| 570 |
+
"content": "```python\ndef clamp(x, lo, hi):\n return max(lo, min(x, hi))\n```",
|
| 571 |
+
"tool_calls": null
|
| 572 |
+
}
|
| 573 |
+
},
|
| 574 |
+
{
|
| 575 |
+
"id": "code_05",
|
| 576 |
+
"category": "short_code",
|
| 577 |
+
"pass": true,
|
| 578 |
+
"reason": "ok",
|
| 579 |
+
"latency_s": 2.627,
|
| 580 |
+
"message": {
|
| 581 |
+
"content": "```python\ndef unique(xs):\n seen = set()\n result = []\n for x in xs:\n if x not in seen:\n seen.add(x)\n result.append(x)\n return result\n```",
|
| 582 |
+
"tool_calls": null
|
| 583 |
+
}
|
| 584 |
+
},
|
| 585 |
+
{
|
| 586 |
+
"id": "code_06",
|
| 587 |
+
"category": "short_code",
|
| 588 |
+
"pass": true,
|
| 589 |
+
"reason": "ok",
|
| 590 |
+
"latency_s": 0.972,
|
| 591 |
+
"message": {
|
| 592 |
+
"content": "{\"name\":\"laguna\",\"params\":118}",
|
| 593 |
+
"tool_calls": null
|
| 594 |
+
}
|
| 595 |
+
},
|
| 596 |
+
{
|
| 597 |
+
"id": "long_01",
|
| 598 |
+
"category": "long_horizon",
|
| 599 |
+
"pass": true,
|
| 600 |
+
"reason": "ok",
|
| 601 |
+
"latency_s": 2.538,
|
| 602 |
+
"message": {
|
| 603 |
+
"content": "I'll check the sandbox directory first to confirm the structure.",
|
| 604 |
+
"tool_calls": [
|
| 605 |
+
{
|
| 606 |
+
"type": "function",
|
| 607 |
+
"function": {
|
| 608 |
+
"name": "list_dir",
|
| 609 |
+
"arguments": "{\"path\":\"sandbox\"}"
|
| 610 |
+
},
|
| 611 |
+
"id": "G6S4bll6FS918pZRLekOfznrN1HQkqah"
|
| 612 |
+
}
|
| 613 |
+
]
|
| 614 |
+
}
|
| 615 |
+
},
|
| 616 |
+
{
|
| 617 |
+
"id": "long_02",
|
| 618 |
+
"category": "long_horizon",
|
| 619 |
+
"pass": true,
|
| 620 |
+
"reason": "ok",
|
| 621 |
+
"latency_s": 1.945,
|
| 622 |
+
"message": {
|
| 623 |
+
"content": "",
|
| 624 |
+
"tool_calls": [
|
| 625 |
+
{
|
| 626 |
+
"type": "function",
|
| 627 |
+
"function": {
|
| 628 |
+
"name": "calc",
|
| 629 |
+
"arguments": "{\"expression\":\"40 - 7 - 9\"}"
|
| 630 |
+
},
|
| 631 |
+
"id": "4siyHV3Fv7MVwHP12hlDZZDq2VnFtykt"
|
| 632 |
+
}
|
| 633 |
+
]
|
| 634 |
+
}
|
| 635 |
+
},
|
| 636 |
+
{
|
| 637 |
+
"id": "long_03",
|
| 638 |
+
"category": "long_horizon",
|
| 639 |
+
"pass": true,
|
| 640 |
+
"reason": "ok",
|
| 641 |
+
"latency_s": 2.634,
|
| 642 |
+
"message": {
|
| 643 |
+
"content": "",
|
| 644 |
+
"tool_calls": [
|
| 645 |
+
{
|
| 646 |
+
"type": "function",
|
| 647 |
+
"function": {
|
| 648 |
+
"name": "search_docs",
|
| 649 |
+
"arguments": "{\"query\":\"yarn rope\",\"limit\":10}"
|
| 650 |
+
},
|
| 651 |
+
"id": "wIMSigb9hxBp2SBwskZMDQjjDq39fqrH"
|
| 652 |
+
}
|
| 653 |
+
]
|
| 654 |
+
}
|
| 655 |
+
},
|
| 656 |
+
{
|
| 657 |
+
"id": "long_04",
|
| 658 |
+
"category": "long_horizon",
|
| 659 |
+
"pass": true,
|
| 660 |
+
"reason": "ok",
|
| 661 |
+
"latency_s": 1.92,
|
| 662 |
+
"message": {
|
| 663 |
+
"content": "",
|
| 664 |
+
"tool_calls": [
|
| 665 |
+
{
|
| 666 |
+
"type": "function",
|
| 667 |
+
"function": {
|
| 668 |
+
"name": "list_dir",
|
| 669 |
+
"arguments": "{\"path\":\"/\"}"
|
| 670 |
+
},
|
| 671 |
+
"id": "k7JqphVZJ0jMsnM4K0rNWEoi4HnhfG6C"
|
| 672 |
+
}
|
| 673 |
+
]
|
| 674 |
+
}
|
| 675 |
+
},
|
| 676 |
+
{
|
| 677 |
+
"id": "long_05",
|
| 678 |
+
"category": "long_horizon",
|
| 679 |
+
"pass": true,
|
| 680 |
+
"reason": "ok",
|
| 681 |
+
"latency_s": 2.776,
|
| 682 |
+
"message": {
|
| 683 |
+
"content": "",
|
| 684 |
+
"tool_calls": [
|
| 685 |
+
{
|
| 686 |
+
"type": "function",
|
| 687 |
+
"function": {
|
| 688 |
+
"name": "write_file",
|
| 689 |
+
"arguments": "{\"path\":\"sandbox/brand.txt\",\"content\":\"personal spark pack\"}"
|
| 690 |
+
},
|
| 691 |
+
"id": "lNMXfiSU3AkjeAcqNHpjMA90EymBwcHk"
|
| 692 |
+
}
|
| 693 |
+
]
|
| 694 |
+
}
|
| 695 |
+
},
|
| 696 |
+
{
|
| 697 |
+
"id": "long_06",
|
| 698 |
+
"category": "long_horizon",
|
| 699 |
+
"pass": false,
|
| 700 |
+
"reason": "KeyError: 'tool'",
|
| 701 |
+
"latency_s": 2.563
|
| 702 |
+
}
|
| 703 |
+
]
|
| 704 |
+
}
|
results/hf_publish.json
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"repo_id": "hizrianraz/laguna-s-2.1-spark",
|
| 3 |
+
"url": "https://huggingface.co/hizrianraz/laguna-s-2.1-spark",
|
| 4 |
+
"commit": "1562d4ebcffc92cf4193c026b1da37cdf22bfe06",
|
| 5 |
+
"siblings": [
|
| 6 |
+
".gitattributes",
|
| 7 |
+
".gitignore",
|
| 8 |
+
"LICENSE.md",
|
| 9 |
+
"LICENSE.openmdw-1.1.txt",
|
| 10 |
+
"NOTICE",
|
| 11 |
+
"README.md",
|
| 12 |
+
"SHA256SUMS",
|
| 13 |
+
"SPARK.md",
|
| 14 |
+
"docs/BUILD_SPARK.md",
|
| 15 |
+
"docs/REPRODUCE.md",
|
| 16 |
+
"eval/agent_smoke/cases.json",
|
| 17 |
+
"eval/agent_smoke/run_smoke.py",
|
| 18 |
+
"hermes/README.md",
|
| 19 |
+
"hermes/sample_client.py",
|
| 20 |
+
"results/.gitkeep",
|
| 21 |
+
"results/MEASURED.md",
|
| 22 |
+
"results/engine_sha.txt",
|
| 23 |
+
"scripts/bench_server.py",
|
| 24 |
+
"scripts/pull_official_gguf.sh",
|
| 25 |
+
"scripts/serve_spark.sh"
|
| 26 |
+
],
|
| 27 |
+
"private": false
|
| 28 |
+
}
|
results/launch_lock.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "laguna-s-2.1-spark-hf-launch",
|
| 3 |
+
"status": "locked",
|
| 4 |
+
"locked_at_wib": "2026-07-28T18:00:00+07:00",
|
| 5 |
+
"launch_at_wib": "2026-08-03T12:00:00+07:00",
|
| 6 |
+
"freeze_at_wib": "2026-08-02T18:00:00+07:00",
|
| 7 |
+
"repo_id": "hizrianraz/laguna-s-2.1-spark",
|
| 8 |
+
"url": "https://huggingface.co/hizrianraz/laguna-s-2.1-spark",
|
| 9 |
+
"default_quant": "poolside/Laguna-S-2.1-GGUF:laguna-s-2.1-Q4_K_M.gguf",
|
| 10 |
+
"q4_sha256": "a8b55c75714ea73fd90ec85de5defdc0b8d88ca0ad2108343cdd8fc22f7583e4",
|
| 11 |
+
"ship_min_smoke": "38/40",
|
| 12 |
+
"ship_stretch_smoke": "40/40",
|
| 13 |
+
"diy_gguf": false,
|
| 14 |
+
"branding": "personal-only"
|
| 15 |
+
}
|
results/measured.json
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ts": 1785231990.7463834,
|
| 3 |
+
"host": "dgx-spark",
|
| 4 |
+
"engine": "poolsideai/llama.cpp 04b2b72",
|
| 5 |
+
"model": "laguna-s-2.1-Q4_K_M.gguf",
|
| 6 |
+
"sha256": "a8b55c75714ea73fd90ec85de5defdc0b8d88ca0ad2108343cdd8fc22f7583e4",
|
| 7 |
+
"serve": {
|
| 8 |
+
"host": "0.0.0.0",
|
| 9 |
+
"port": 8000,
|
| 10 |
+
"ctx": 8192,
|
| 11 |
+
"ngl": -1,
|
| 12 |
+
"parallel": 1,
|
| 13 |
+
"alias": "local-laguna",
|
| 14 |
+
"jinja": true,
|
| 15 |
+
"fa": "on"
|
| 16 |
+
},
|
| 17 |
+
"mem": {
|
| 18 |
+
"host_total_gi": 121,
|
| 19 |
+
"used_after_smoke_gi_approx": 96,
|
| 20 |
+
"available_gi_approx": 25,
|
| 21 |
+
"server_rss_ki_post": 2054584
|
| 22 |
+
},
|
| 23 |
+
"throughput": [
|
| 24 |
+
{
|
| 25 |
+
"label": "gen8_short",
|
| 26 |
+
"prompt_tokens": 45,
|
| 27 |
+
"completion_tokens": 4,
|
| 28 |
+
"latency_s": 0.511,
|
| 29 |
+
"tok_s": 7.821
|
| 30 |
+
},
|
| 31 |
+
{
|
| 32 |
+
"label": "gen128",
|
| 33 |
+
"prompt_tokens": 61,
|
| 34 |
+
"completion_tokens": 128,
|
| 35 |
+
"latency_s": 6.057,
|
| 36 |
+
"tok_s": 21.133
|
| 37 |
+
},
|
| 38 |
+
{
|
| 39 |
+
"label": "prefill_heavy_gen64",
|
| 40 |
+
"prompt_tokens": 1655,
|
| 41 |
+
"completion_tokens": 3,
|
| 42 |
+
"latency_s": 2.647,
|
| 43 |
+
"tok_s": 1.134
|
| 44 |
+
}
|
| 45 |
+
],
|
| 46 |
+
"quote_gen_tok_s": 21.13,
|
| 47 |
+
"agent_smoke": {
|
| 48 |
+
"passed": 38,
|
| 49 |
+
"failed": 2,
|
| 50 |
+
"n": 40,
|
| 51 |
+
"pass_rate": 0.95,
|
| 52 |
+
"elapsed_s": 96.66,
|
| 53 |
+
"fails": [
|
| 54 |
+
"repair_04 HTTP500 tool-arg JSON",
|
| 55 |
+
"long_06 KeyError tool"
|
| 56 |
+
]
|
| 57 |
+
}
|
| 58 |
+
}
|
results/server_bench.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"base_url": "http://127.0.0.1:8000/v1",
|
| 3 |
+
"model": "local-laguna",
|
| 4 |
+
"rows": [
|
| 5 |
+
{
|
| 6 |
+
"mark": "2k",
|
| 7 |
+
"latency_s": 1.826,
|
| 8 |
+
"prompt_tokens": 836,
|
| 9 |
+
"completion_tokens": 3,
|
| 10 |
+
"completion_tok_s": 1.643,
|
| 11 |
+
"content": "OK"
|
| 12 |
+
},
|
| 13 |
+
{
|
| 14 |
+
"mark": "8k",
|
| 15 |
+
"latency_s": 4.731,
|
| 16 |
+
"prompt_tokens": 3236,
|
| 17 |
+
"completion_tokens": 3,
|
| 18 |
+
"completion_tok_s": 0.634,
|
| 19 |
+
"content": "OK"
|
| 20 |
+
}
|
| 21 |
+
],
|
| 22 |
+
"ts": 1785231739.079031
|
| 23 |
+
}
|