broken link? https://huggingface.co/spaces/neullabs/grite

Good catch, that path is dead. I folded the four standalone demos into one Space to stay under HF's 3-slot cpu-basic quota, so grite moved.

Live here, running over local git data (the grite tab):
https://huggingface.co/spaces/neullabs/agent-infra

Thanks for flagging it. Did it 404 for you, or hang on a cold boot?

Hey Dipankar, noticed you liked my Agent Failure Atlas benchmark recently. Really appreciated that. I’m working on a related independent research submission and wanted to ask you something about arXiv. Is there a good way to reach you privately?

You do not need a private channel for the arXiv part.

Endorsement runs on arXiv's side. They hand the submitter a code and a link, the endorser acts there, and nobody needs your email. Whether I am even an endorser in the category they slot you into is arXiv's call rather than mine, so I would not want you counting on it either way.

What I would rather do is read the paper.

And before you write it up, one thing from re-pulling the atlas smoke set tonight, because a reviewer will find it too.

I checked this set two days ago, hashed all 20 traces, got 20 unique sha256, and moved on. That check was blind. Every trace embeds its own filename in the session id, so 20 unique hashes are guaranteed no matter what the payload is.

Drop that one field and the 20 traces are 8 scenarios. Twelve files are byte-identical to another file. synthetic-01, -09, -17 and -20 are the same 662 bytes of "Safe bounded read". 02/10/19 are one scenario, 03/11/18 are one. Zero records differ besides the id, and labels agree inside every group, so the collapse is clean.

That lands in synthetic_metrics.json. true_positives 20 is 9 distinct detections. scope_violation reads as the best-covered category at tp 7 and is 3 scenarios. approval_bypass reads 3 and is one scenario copied three times. Copy count also becomes weight: a detector failing only "Safe bounded read" loses 20% of the headline, one failing only "Runaway repeated search" loses 10%, for the same single defect.

Separate one. taxonomy.yaml declares 8 categories, per_category has rows for 6. unauthorized_tool and unhandled_tool_failure have no fixture and no row, so a detector that does not exist yet cannot lose a point on this set.

I liked the dataset for the limitation paragraph, which is rarer than it should be. This is that paragraph with numbers on it.

Happy to paste the per-scenario table. Is the duplication deliberate, or did the generator emit 20 from an 8-item template?

Table first, then one thing on TSC.

Re-pulled at sha 752921296, lastModified 2026-07-28, so this is against the same bytes you have now.

One correction to my own wording last night. The twelve are identical once the session id is dropped, not byte-identical. id is the only field that differs anywhere, and name plus labels agree inside every group, so the collapse is still clean.

per_category tp is counting files:

approval_bypass 3 becomes 1 scenario. prompt_injection 2 becomes 1. runaway_loop 2 becomes 1. silent_tool_failure 2 becomes 1. scope_violation 7 becomes 3, at scenarios 2, 4 and 8. secret_exposure 4 becomes 2, at scenarios 4 and 7. So tp 20 is 9 distinct detections.

Now the part I would fix before the weighting.

Your negative class is one scenario. fp 0 and precision 1.0, in every category row, rest on "Safe bounded read" copied four times. Four copies of one negative cannot surface a false positive that a second distinct negative would. Recall has 8 scenarios under it and precision has 1, so the two halves of that F1 are not carrying equal weight. I would put that above the taxonomy gap, because unauthorized_tool and unhandled_tool_failure at least announce themselves by having no row.

On TSC.

The ceiling check is right. 32/8 = 4.0 and 32/6 = 5.333, and metadata only moves the real number down, so it is a genuine upper bound.

But one artifact cannot explain both overshoots. K4V4 at 4.1 is 2.5% over. K4V2 at 6.1 to 6.2 is 14 to 16% over. An inflated FP16 baseline, a padded allocator length, a reserved-capacity denominator, all of those multiply both ratios by the same factor. These are not the same factor.

In bits it is sharper. 4.1 implies 7.805 bits where 8 are nominal, so 0.195 bits are unaccounted for. 6.1 to 6.2 implies 5.16 to 5.25 where 6 are nominal, so 0.75 to 0.84 bits. Roughly 4x more impossible bits, and the only thing that changed between the two configs is V going from 4 bits to 2.

A constant metadata or alignment error leaves a constant bit gap. This one scales with how hard V is quantized, which points at the V serialization path rather than at the baseline.

So, to take your serialized-byte-accounting question in the direction I think it actually bites: is V packed at its nominal width in the serialized bitstream, or does it pass through something that can land below it?