debuxbench_v1.json

{
 "tag": "ckpt80",
 "model": "/Volumes/ASAFE/strix/qwen38-27b-4bit",
 "adapter": "/Users/mont127/debux/adapters_run1/ckpt_20",
 "rows": [
  {
   "id": "insufficient_breach_suspicion__t2",
   "scenario": "insufficient_breach_suspicion",
   "shape": "insufficient",
   "want": "INSUFFICIENT",
   "got": "NEED",
   "correct": false,
   "tokens": 345,
   "finish": "stop",
   "output": "REASONING:\n- the file exists at /usr/local/bin/sysupdate but is not registered with dpkg, so it was not installed via the system package manager\n- the file is ~4.2 MB and was modified on 2026-09-20, which is a future date relative to now, so the mtime is either spoofed or the system clock is wrong\n- no ssh session around 03:0x, so the file was not placed via an interactive ssh login at that time\n- the process is running as root and making an outbound connection to 185.220.101.7:443, which is consistent with a malware beacon or a legitimate update agent\n- the name 'sysupdate' is not a standard Linux system binary, which raises suspicion but does not confirm compromise\n- the future mtime and the unregistered file are strong indicators of tampering or a malicious implant, but not yet conclusive without inspecting the binary itself\n\nNEED:\n  file /usr/local/bin/sysupdate\n  sha256sum /usr/local/bin/sysupdate\n  stat /usr/local/bin/sysupdate\n  ls -la /proc/8841/exe\n  cat /proc/8841/cmdline | tr '\\0' ' '\n  ss -tnp | grep 8841\n  journalctl -u sysupdate --no-pager | tail -50\n  systemctl status sysupdate 2>/dev/null || echo 'no systemd unit'\n  ls -la /etc/systemd/system/ | grep -i sysupdate"
  },
  {
   "id": "insufficient_intermittent_gone_now__t1",
   "scenario": "insufficient_intermittent_gone_now",
   "shape": "insufficient",
   "want": "INSUFFICIENT",
   "got": "DIAGNOSIS",
   "correct": false,
   "tokens": 268,
   "finish": "stop",
   "output": "REASONING:\n- the interface shows zero errors, zero drops, and zero overrun on both RX and TX, and the link never flapped (no kernel messages)\n- this completely rules out a local NIC, driver, or cable problem on this host\n- since the local side was clean and the network team says their side was clean, the loss was not a full link down but intermittent packet loss\n- the only remaining candidates are: a transient fault on the remote host's own interface, a switch port buffer overflow on the path, or a one-way loss on the remote side that the network team's monitoring did not capture\n- the 40-minute duration and self-clearing nature is consistent with a buffer overflow or a flapping remote interface that the network team's coarse monitoring missed\nDIAGNOSIS: the packet loss was not on this host's interface; it was either on the remote host's interface or a transient buffer overflow on a switch port in the path that the network team's monitoring did not capture\nFIX: check the remote host's interface counters and dmesg for the same time window, and ask the network team to pull per-port drop and error counters from the specific switch ports on the path for that 40-minute window rather than just a general \"clean\" report"
  },
  {
   "id": "insufficient_intermittent_slow_no_data__t1",
   "scenario": "insufficient_intermittent_slow_no_data",
   "shape": "insufficient",
   "want": "INSUFFICIENT",
   "got": "NEED",
   "correct": false,
   "tokens": 115,
   "finish": "stop",
   "output": "REASONING:\n- the load average of 1.20 is low and does not indicate CPU saturation, so the app server itself is not the bottleneck\n- the access log shows a single request with no timing data, so it does not reveal slow response times or which endpoint is slow\n- the user's report of intermittent slowness is not yet tied to any metric, so the root cause is not yet established\nNEED:\n  sudo journalctl -u nginx.service --since '1 hour ago' | grep -i 'slow'"
  },
  {
   "id": "insufficient_memory_leak_claim__t1",
   "scenario": "insufficient_memory_leak_claim",
   "shape": "insufficient",
   "want": "INSUFFICIENT",
   "got": "NEED",
   "correct": false,
   "tokens": 163,
   "finish": "stop",
   "output": "REASONING:\n- the process has been running for 3 days and its current RSS (2.2GB) is nearly identical to its all-time high (VmHWM 2.2GB), which shows the memory has been growing and never released\n- the jump from 900MB at deploy to 2.2GB over 3 days is a sustained increase, not a one-time spike\n- but this alone does not confirm a leak: a service can legitimately grow to a new steady state (e.g., cache warming, connection pool expansion) and then plateau\n- the critical missing piece is whether the growth is still ongoing or has stabilized\nNEED:\n  watch -n 30 'grep VmRSS /proc/1902/status'"
  },
  {
   "id": "search_needed_cloud_quota_semantics__t2",
   "scenario": "search_needed_cloud_quota_semantics",
   "shape": "search_needed",
   "want": "SEARCH",
   "got": "DIAGNOSIS",
   "correct": false,
   "tokens": 313,
   "finish": "stop",
   "output": "REASONING:\n- the user's observation of throttling before hitting the 200 concurrency limit is consistent with the existence of a separate, lower burst limit that applies before the account's reserved concurrency is fully engaged or before the adaptive auto-scaling can react to a sudden spike\n- the 200 figure is the default *reserved* concurrency per function if not explicitly set, but the *account-level* default is 1000, and the *burst* behavior is governed by the difference between the current in-flight executions and the reserved limit\n- the specific burst limit value is not a static number but a dynamic calculation based on the function's configuration and the region's current capacity\n- the exact burst limit for a specific function in eu-west-1 depends on whether the function has a reserved concurrency set, and if so, what that value is\n- the user's question about a \"separate burst limit\" is based on a misunderstanding of how the 200 figure relates to the actual throttling mechanism\nDIAGNOSIS: the throttling is not caused by a separate, fixed burst limit but by the function's reserved concurrency being set to 200, which is the default if not explicitly configured, and the traffic spike exceeded this limit before the account's adaptive auto-scaling could increase the available concurrency\nFIX: check the function's reserved concurrency setting and either increase it or remove it to allow the function to scale up to the account's default of 1000 concurrent executions"
  },
  {
   "id": "search_needed_new_release_behaviour__t2",
   "scenario": "search_needed_new_release_behaviour",
   "shape": "search_needed",
   "want": "SEARCH",
   "got": "NEED",
   "correct": false,
   "tokens": 198,
   "finish": "stop",
   "output": "REASONING:\n- the error text 'deprecated keyring' on 24.04 is the signature of the apt_key module being removed or its keyring path being deprecated in favor of signed-by in sources.list.d\n- ansible-core 2.16 does not ship the deb822_repository module; that module was introduced in ansible-core 2.17\n- so on 22.04 hosts with ansible-core 2.16, deb822_repository is unavailable and apt_key still works\n- on 24.04 hosts, apt_key is broken/deprecated and deb822_repository is needed but missing from 2.16\n- the only way to satisfy both targets with a single playbook is to upgrade ansible-core to 2.17+ on the control node, which provides deb822_repository and removes the apt_key dependency entirely\nNEED:\n  ansible --version"
  },
  {
   "id": "search_not_needed_http2_error__t0",
   "scenario": "search_not_needed_http2_error",
   "shape": "search_not_needed",
   "want": "NEED",
   "got": null,
   "correct": false,
   "tokens": 400,
   "finish": "length",
   "output": "REASONING:\n- the error is a client-side report that the server's HTTP/2 framing was malformed or that the connection was reset mid-stream\n- this is almost never a browser bug; it is almost always a server-side or intermediate-cause issue\n- the most common real-world causes are:\n  - server-side HTTP/2 implementation bug or misconfiguration (e.g., nginx with an old version, Apache with mod_http2, a load balancer)\n  - an intermediate proxy or load balancer that does not fully support HTTP/2 and corrupts frames\n  - a server that sends a GOAWAY or RST_STREAM with an error code the client does not expect\n  - a server that closes the connection abruptly without a proper GOAWAY\n  - a server that sends a frame larger than the advertised MAX_FRAME_SIZE\n  - a server that sends a HEADERS frame with an invalid stream ID\n  - a server that sends a DATA frame on a stream that has already been half-closed\n  - a server that sends a WINDOW_UPDATE that exceeds the flow control window\n  - a server that sends a SETTINGS frame with an invalid value\n  - a server that sends a PING that is not acknowledged\n  - a server that sends a RST_STREAM with an error code that is not defined in the spec\n  - a server that sends a GOAWAY with a last stream ID that is not a valid stream ID\n  - a server that sends a frame with an invalid length\n  - a server that sends a frame with an invalid type\n  - a server that sends a frame with an invalid flags\n  - a server that sends a frame with an invalid stream ID\n  - a server that sends a frame with an invalid padding\n  - a server that sends a frame with an invalid priority\n  - a server that sends a frame with an invalid weight\n  - a server that sends a"
  },
  {
   "id": "search_not_needed_selinux_denial__t0",
   "scenario": "search_not_needed_selinux_denial",
   "shape": "search_not_needed",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 190,
   "finish": "stop",
   "output": "REASONING:\n- confirmed: file exists, mode 644, nginx returns 403\n- need to rule out: SELinux denial (Rocky 9 enforces by default), directory traversal perms, open_file_cache, autoindex off, mime type block, worker user mismatch\n- need to see: actual error line in /var/log/nginx/error.log, SELinux state, directory chain perms, nginx.conf relevant sections\nNEED:\n  sudo tail -n 20 /var/log/nginx/error.log\n  sudo getenforce\n  sudo ls -laZ /var/www/html/ | head\n  sudo namei -l /var/www/html/<the-file>\n  sudo grep -n 'user' /etc/nginx/nginx.conf\n  sudo grep -n 'autoindex\\|open_file_cache\\|location' /etc/nginx/conf.d/*.conf /etc/nginx/nginx.conf"
  },
  {
   "id": "acl_vs_mode_confusion__t0",
   "scenario": "acl_vs_mode_confusion",
   "shape": "misleading",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 143,
   "finish": "stop",
   "output": "REASONING:\n- the file mode is -rw-r-----: owner root has rw, group analysts has r, others none\n- alice is in analysts, so group read should grant her access\n- but the trailing + means an ACL is present, and ACLs override the group bits\n- the ls output does not show the actual ACL entries, so the group-class permission that applies to alice is unknown\n- an ACL entry could explicitly deny alice, or the group-class entry could be more restrictive than the r bit suggests\n- need to see the actual ACL to know what alice's effective permission is\n\nNEED:\n  getfacl /srv/data/report.csv\n  id alice"
  },
  {
   "id": "ipv6_preferred_broken__t0",
   "scenario": "ipv6_preferred_broken",
   "shape": "misleading",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 400,
   "finish": "length",
   "output": "REASONING:\n- the 20+ second timeout signature on an otherwise healthy API almost always points to a network path issue rather than application logic, since app-level slowness would be consistent rather than intermittent\n- the client-network variance is the strongest clue: it means the problem is in the path between specific clients and the server, not in the server itself\n- the most common cause of this exact pattern is a stale or half-open connection in an intermediate device (load balancer, firewall, NAT) that silently drops the connection while both ends still think it's alive, so the client's next request on that connection hangs until the TCP keepalive or the client's own timeout fires\n- other candidates to keep open: a load balancer with an idle timeout shorter than the client's reuse window, a firewall doing stateful inspection that drops idle connections, or a client-side proxy\n- need to confirm whether the slowness is on connection establishment (handshake) or on the response after the request is sent, and whether it correlates with connection reuse\nNEED:\n  curl -v -o /dev/null -w 'time_namelookup=%{time_namelookup}\\ntime_connect=%{time_connect}\\ntime_appconnect=%{time_appconnect}\\ntime_starttransfer=%{time_starttransfer}\\ntime_total=%{time_total}\\n' https://your-api.example.com/health\n  curl -v -o /dev/null -w 'time_connect=%{time_connect}\\ntime_starttransfer=%{time_starttransfer}\\ntime_total=%{time_total}\\n' https://your-api.example.com/health\n  curl -v -o /dev/null -w 'time_connect=%{time_connect}\\ntime_starttransfer=%{time_starttransfer}\\ntime_total=%{time_total}\\n' https://your-api.example.com/health\n  ss -tn state established | head -20"
  },
  {
   "id": "ntp_vs_timezone__t0",
   "scenario": "ntp_vs_timezone",
   "shape": "misleading",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 191,
   "finish": "stop",
   "output": "REASONING:\n- the claim is that the system clock is 2 hours behind real time, which would make every timestamp in the logs appear 2 hours in the past\n- need to confirm the system clock is actually wrong before touching NTP\n- need to check whether the time source is UTC or local time, since a 2-hour offset could be a timezone misconfiguration rather than actual clock drift\n- need to check the NTP service state and last sync time to see if it is running, stopped, or silently failing\nNEED:\n  date\n  timedatectl\n  systemctl status chronyd 2>/dev/null || systemctl status ntpd 2>/dev/null || systemctl status systemd-timesyncd 2>/dev/null\n  chronyc tracking 2>/dev/null || ntpq -p 2>/dev/null || timedatectl show-timesync --all 2>/dev/null"
  },
  {
   "id": "acl_vs_mode_confusion__t2",
   "scenario": "acl_vs_mode_confusion",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "DIAGNOSIS",
   "correct": true,
   "tokens": 206,
   "finish": "stop",
   "output": "REASONING:\n- the file has an ACL (the '+' in ls and the getfacl output)\n- alice has an explicit named-user ACL entry: user:alice:--- which grants no access\n- in POSIX ACLs, a named-user entry takes precedence over the group entry, so the group::r-- and mask::r-- are never consulted for alice\n- the mask is r--, which would cap group access anyway, but the named-user entry already denies everything\n- so the permissions look fine for the group, but alice is explicitly overridden to nothing\n\nDIAGNOSIS: an explicit named-user ACL entry user:alice:--- overrides her group membership and denies all access to /srv/data/report.csv\nFIX:\n  sudo setfacl -m u:alice:r /srv/data/report.csv\n  sudo setfacl -m m:r /srv/data/report.csv\n  ls -l /srv/data/report.csv\n  sudo getfacl /srv/data/report.csv"
  },
  {
   "id": "cron_path__t2",
   "scenario": "cron_path",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "DIAGNOSIS",
   "correct": true,
   "tokens": 248,
   "finish": "stop",
   "output": "REASONING:\n- the error is a PATH lookup failure, not a script logic error\n- cron runs with a minimal environment (typically PATH=/usr/bin:/bin)\n- the script works manually because the interactive shell has a richer PATH that includes the directory holding pg_dump\n- this is the classic cron-vs-interactive-shell PATH mismatch, not a permissions or scheduling issue\nDIAGNOSIS: cron's minimal PATH does not include the directory where pg_dump lives, so the script cannot find it when invoked by cron\nFIX:\n  add an explicit PATH at the top of /home/ops/backup.sh, before any command that depends on it:\n  export PATH=/usr/local/bin:/usr/bin:/bin\n  (or better, call pg_dump by its absolute path, e.g. /usr/local/bin/pg_dump, which is immune to PATH entirely)\n  verify the real location first with: which pg_dump   (run in the same shell where the manual run succeeds)\n  then re-test under cron's environment to confirm:\n  env -i /bin/sh -c 'PATH=/usr/local/bin:/usr/bin:/bin /home/ops/backup.sh'"
  },
  {
   "id": "docker_layer_disk__t2",
   "scenario": "docker_layer_disk",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "NEED",
   "correct": false,
   "tokens": 144,
   "finish": "stop",
   "output": "REASONING:\n- the three dangling images alone are ~11.8GB, but the user says there are 39 more, so the total dangling image size could be much larger\n- the running containers (web, worker) and the exited migrator are small in count but their volumes and logs could still be large\n- need to see the full picture: total docker disk usage, the full list of dangling images, and volume sizes before deleting anything\nNEED:\n  docker system df\n  docker images -f \"dangling=true\" --format '{{.Size}}' | sort -h | tail -20\n  docker volume ls\n  docker system df -v"
  },
  {
   "id": "ipv6_preferred_broken__t3",
   "scenario": "ipv6_preferred_broken",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "NEED",
   "correct": false,
   "tokens": 261,
   "finish": "stop",
   "output": "REASONING:\n- the link-local address fe80::1 is not a valid target for a general reachability test, so the 100% loss here is expected and tells us nothing about the actual API host\n- the user's own description already narrows the problem: intermittent 20s delays on https, worse from some client networks than others, which is the classic signature of a path-dependent issue rather than a server-side one\n- the 20s figure is a strong hint toward a timeout that is being hit and then retried, rather than genuine slow processing\n- the need to distinguish server-side slowness from client-path slowness is the first fork in the road, and the existing output does not resolve it\nNEED:\n  curl -v -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}\\n' https://<api-host>/  --run this from the server itself and from one of the slow client machines, 3-5 times each\n  ss -ltnp | grep ':443'\n  cat /proc/sys/net/ipv4/tcp_slow_start_after_idle"
  },
  {
   "id": "journald_filling_disk__t3",
   "scenario": "journald_filling_disk",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "NEED",
   "correct": false,
   "tokens": 151,
   "finish": "stop",
   "output": "REASONING:\n- the log lines show UFW blocking inbound scans on ports 23 and 2375, which are common targets for botnets\n- these kernel messages are written to /var/log/kern.log or /var/log/syslog, and at high scan rates they can accumulate quickly\n- 1GB/day on an idle box is consistent with sustained logging of blocked packets rather than application data\n- need to confirm which file under /var is actually growing before concluding it's the firewall log\nNEED:\n  sudo find /var -type f -size +10M -exec ls -lh {} \\; 2>/dev/null | sort -k5 -h | tail -20"
  },
  {
   "id": "k8s_imagepullbackoff_auth__t2",
   "scenario": "k8s_imagepullbackoff_auth",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "NEED",
   "correct": false,
   "tokens": 154,
   "finish": "stop",
   "output": "REASONING:\n- the ServiceAccount object is unrelated to image pulling; it only governs API access and token mounting\n- ImagePullBackOff means the kubelet could not fetch the image, but the pasted object gives no image name, registry, or error message\n- without the actual pull error, the cause could be a bad tag, wrong registry, missing credentials, or a network block, and none of these can be confirmed from this object alone\nNEED:\n  kubectl -n prod get pod -l app=billing -o jsonpath='{.items[0].spec.containers[*].image}'\n  kubectl -n prod describe pod -l app=billing | sed -n '/Events:/,$p'"
  },
  {
   "id": "k8s_oomkilled_limit__t2",
   "scenario": "k8s_oomkilled_limit",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "NEED",
   "correct": false,
   "tokens": 232,
   "finish": "stop",
   "output": "REASONING:\n- the current logs show no application error, which means the process is not crashing with an unhandled exception or error message before the container exits\n- the pod is in CrashLoopBackOff, which means the container is repeatedly starting and stopping\n- the resource usage (310m CPU, 248Mi memory) is low and unlikely to be the cause of an OOM kill\n- the most likely cause is that the container is exiting with a non-zero status code before the application has a chance to log an error, or the application is exiting cleanly but the liveness/readiness probe is failing\n- need to check the previous container's logs to see what happened before the crash\n- need to check the pod's events to see if there are any probe failures or other issues\n- need to check the exit code of the previous container to understand why it's restarting\n\nNEED:\n  kubectl logs worker-6f7b9c4d8-p2qwl --previous\n  kubectl describe pod worker-6f7b9c4d8-p2qwl"
  },
  {
   "id": "k8s_pending_unschedulable__t2",
   "scenario": "k8s_pending_unschedulable",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "DIAGNOSIS",
   "correct": true,
   "tokens": 400,
   "finish": "length",
   "output": "REASONING:\n- the pod's own requests are small (1 cpu / 512Mi) and the node's memory is only 65% requested, so memory is not the blocker\n- the node's CPU is at 95% of its allocatable (3800m of 4000m) and its memory is at 65%\n- a pod that never gets a node assigned (no node assigned at all) is the signature of the scheduler finding no node that can satisfy the request, not of a node that is tainted or cordoned in a way that would still show a reason\n- the only resource that is nearly fully committed on this node is CPU, and the pod needs 1 cpu of headroom that this node does not have\n- this is the classic overcommit-of-CPU-requests situation: the node looks fine on memory but its CPU requests are already almost entirely allocated, so the scheduler cannot place a new 1-cpu pod on it\nDIAGNOSIS: the node's CPU requests are already ~95% allocated, leaving no room for this pod's 1-cpu request, so the scheduler cannot find a schedulable node and the pod stays Pending with no node assigned\nFIX:\n  - confirm the reason with: kubectl describe pod <pod> | grep -A5 Events  (look for '0/'N' nodes are available: N Insufficient cpu')\n  - short term: free CPU on that node by scaling down or deleting an existing CPU-hungry pod, or move the job to a node with CPU headroom\n  - longer term: right-size existing pods' CPU requests (many are likely over-requesting) or add a node with more CPU so the cluster has real CPU headroom\n  - if this is a batch job that should not block on CPU, consider a lower CPU request or a node pool with"
  },
  {
   "id": "k8s_readiness_probe_fail__t2",
   "scenario": "k8s_readiness_probe_fail",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "NEED",
   "correct": false,
   "tokens": 316,
   "finish": "stop",
   "output": "REASONING:\n- the app is alive and the HTTP server is up: the /healthz request got a 500 in 2ms rather than timing out, so the process and its listener are fine\n- the 500 comes from the health check itself, which depends on the database: the error shows the app cannot reach postgres at 10.96.44.12:5432 with connection refused\n- connection refused at the Service IP means nothing is listening on that port right now, which is why the readiness probe fails and the pod is never marked Ready\n- a pod that never becomes Ready is removed from the Service's endpoints, which is exactly why the Service shows zero endpoints and why curling the Service from inside the cluster times out instead of getting a response\n- so the timeout is not a network or app problem at this layer; it is the Service having no healthy backends because the app's own health check is failing on the database dependency\n- the open question is why the database is refusing connections: the postgres pod itself may be down or not yet ready, or the app may be pointing at the wrong database Service\n\nNEED:\n  kubectl get pods -n <namespace> | grep -i postgres\n  kubectl get svc -n <namespace> | grep -i postgres\n  kubectl get endpoints -n <namespace> | grep -i postgres\n  kubectl logs -n <namespace> deploy/<postgres-deployment> --tail=50"
  },
  {
   "id": "keepalive_lb_resets__t2",
   "scenario": "keepalive_lb_resets",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "NEED",
   "correct": false,
   "tokens": 207,
   "finish": "stop",
   "output": "REASONING:\n- the error is a RST on an idle connection, not a timeout close, so something is actively killing the socket rather than the app timing out\n- the load balancer is the only stateful middlebox between client and this backend, and its idle timeout is the classic cause of this exact signature\n- need to see what the LB's idle timeout actually is and whether the app itself has a shorter keepalive that could be racing it\nNEED:\n  ss -tnpi | grep 10.0.4.5:52344\n  cat /proc/sys/net/ipv4/tcp_keepalive_time /proc/sys/net/ipv4/tcp_keepalive_intvl /proc/sys/net/ipv4/tcp_keepalive_probes\n  systemctl show nginx | grep -i timeout   # or whatever the app's service unit is\n  journalctl -u nginx --since '1 hour ago' | grep -i 'idle\\|timeout\\|reset'"
  },
  {
   "id": "ntp_vs_timezone__t2",
   "scenario": "ntp_vs_timezone",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "DIAGNOSIS",
   "correct": true,
   "tokens": 244,
   "finish": "stop",
   "output": "REASONING:\n- the query output shows the system clock is only 42 microseconds off from the NTP reference, which is well within normal tolerance and directly contradicts a 2-hour drift\n- the reference time itself is stamped September 24, 2026, which is in the future relative to the current real-world date, so the box's clock is actually ahead of true wall clock time, not behind\n- the user's premise that the clock is 2 hours behind is not supported by the query data, so the real issue is likely something else entirely, such as the application's own timestamping logic, a timezone misconfiguration, or the application reading from a different time source than the system clock\nDIAGNOSIS: the system clock is not the problem, it is actually 42 microseconds fast and the reference time is in the future, so the 2-hour discrepancy the user is seeing in the logs is almost certainly a timezone or application-level timestamping issue rather than an NTP drift\nFIX: check the application's timezone configuration and how it generates timestamps, and verify the system timezone with timedatectl status, since the clock itself is healthy"
  },
  {
   "id": "pam_account_expired__t2",
   "scenario": "pam_account_expired",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "NEED",
   "correct": false,
   "tokens": 116,
   "finish": "stop",
   "output": "REASONING:\n- the password and account are not expired, so the denial is not a credential validity issue\n- the user says nothing changed, but something did change today that blocks login after the password is accepted\n- need to see the actual denial message and the auth logs to find what is rejecting the session\nNEED:\n  sudo journalctl -u sshd.service -n 50 --no-pager\n  sudo tail -n 50 /var/log/auth.log\n  sudo tail -n 50 /var/log/secure"
  },
  {
   "id": "proxy_strips_real_ip__t2",
   "scenario": "proxy_strips_real_ip",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "DIAGNOSIS",
   "correct": true,
   "tokens": 400,
   "finish": "length",
   "output": "REASONING:\n- the app's own access log shows 127.0.0.1 for every request, which is the loopback address\n- the note explicitly says the app uses request.remote_addr and does not parse proxy headers\n- this is the classic signature of an app sitting behind a reverse proxy (nginx, haproxy, cloud LB) without being configured to trust and read the real client IP from X-Forwarded-For or similar headers\n- because the app never looks at the proxy headers, it can't distinguish clients and any rate limiting keyed on the app's own remote_addr is effectively keyed on 127.0.0.1 for everyone\n- this is not a network or firewall issue; the real client IPs are almost certainly arriving at the proxy and being forwarded, but the app discards them\nDIAGNOSIS: the application is behind a reverse proxy but is not configured to read the real client IP from proxy headers, so it only ever sees the proxy's loopback address\nFIX:\n  1) confirm the proxy is actually forwarding the real client IP - check the proxy's own access log for the same request and look for the real client IP and an X-Forwarded-For header\n  2) in the app, enable trusted proxy / real IP handling so it reads X-Forwarded-For instead of the socket peer, e.g.:\n     - nginx: add real_ip_header X-Forwarded-For; and real_ip_recursive on; in the proxy block, and make sure proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; is present\n     - app side: set the framework's trusted proxy / real IP option (e.g. trust proxy, real_ip_header, or the equivalent) so request.remote_addr resolves to the client rather than 127.0.0.1\n  3) re-test"
  },
  {
   "id": "rabbitmq_queue_backlog__t2",
   "scenario": "rabbitmq_queue_backlog",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "DIAGNOSIS",
   "correct": true,
   "tokens": 356,
   "finish": "stop",
   "output": "REASONING:\n- the channel listing shows each consumer has a prefetch of 20,000 and roughly 15,670 unacknowledged messages, so the broker has already delivered its entire prefetch window to each consumer and is now blocked from sending more until those are acked or nacked\n- the worker log shows a single order taking 105 seconds before a DB query timeout and retry, which means the consumer's processing loop is effectively serialized on a slow database call rather than the broker being the bottleneck\n- because the consumer never acks the 15,670 in-flight messages (they are stuck on the same slow DB path), the prefetch window never drains, so the queue keeps accumulating even though the consumers are connected\nDIAGNOSIS: the consumers are not draining because their prefetch of 20,000 is far too large for the actual processing rate, and the underlying database is timing out on queries, so the in-flight messages never get acked and the broker stops delivering new ones\nFIX:\n  1. immediately lower the prefetch on the existing consumers to something realistic for the current DB latency, e.g. 10 to 50, so the broker can start delivering to the faster consumers and the slow ones stop hoarding the entire queue\n  2. fix the database timeout at the root, either by restoring the database's health or by adding a circuit breaker in the worker that nacks with requeue=false and routes to a dead letter queue instead of retrying against a dead database\n  3. after the prefetch is reduced and the database is healthy, watch the unacknowledged count drop to near zero before raising prefetch again"
  },
  {
   "id": "readonly_filesystem__t2",
   "scenario": "readonly_filesystem",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "NEED",
   "correct": false,
   "tokens": 185,
   "finish": "stop",
   "output": "REASONING:\n- the dmesg shows an ATA exception followed by a SATA link re-negotiation to 6.0 Gbps, which is a classic signature of a disk controller or link error that can cause the kernel to remount the filesystem read-only as a protective measure\n- SMART shows no reallocated sectors or pending sectors, so the disk itself is healthy and not the source of the error\n- the 'Read-only file system' error on every write confirms the mount is now read-only, not that the disk is physically dead\n- the most likely cause is that the kernel detected an I/O error or timeout and automatically remounted the filesystem read-only to prevent data corruption\nNEED: mount | grep -v 'proc\\|sys\\|dev' && dmesg | grep -i 'read-only\\|remount\\|error' | tail -20"
  },
  {
   "id": "search_needed_cloud_quota_semantics__t1",
   "scenario": "search_needed_cloud_quota_semantics",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "NEED",
   "correct": false,
   "tokens": 260,
   "finish": "stop",
   "output": "REASONING:\n- the function has a hard cap of 200 concurrent executions (ReservedConcurrentExecutions)\n- the account has 1000 total, but only 800 are unreserved, so this function can never exceed 200\n- the throttling errors are consistent with hitting that 200-function cap, not the account-wide 1000\n- the spike pushed this function's concurrency to its own reserved ceiling, which looks like a small number compared to the account limit but is the actual binding constraint\nNEED:\n  aws lambda get-function --function-name <function-name> --query 'Configuration.ReservedConcurrentExecutions'\n  aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name ConcurrentExecutions --dimensions Name=FunctionName,Value=<function-name> --start-time <spike-start> --end-time <spike-end> --period 60 --statistics Maximum\n  aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Throttles --dimensions Name=FunctionName,Value=<function-name> --start-time <spike-start> --end-time <spike-end> --period 60 --statistics Sum"
  },
  {
   "id": "search_needed_new_release_behaviour__t1",
   "scenario": "search_needed_new_release_behaviour",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "DIAGNOSIS",
   "correct": true,
   "tokens": 374,
   "finish": "stop",
   "output": "REASONING:\n- the error is not a keyring format issue but a missing binary: 'apt-key' not found\n- apt-key was removed from the apt package in Ubuntu 24.04 (and Debian 12) because it is deprecated and insecure\n- the playbook's add_repo_key task still calls apt_key, which shells out to the apt-key binary, so it fails\n- the modern replacement is to place the GPG key directly into /etc/apt/keyrings/ and reference it in the sources file with signed-by=\nDIAGNOSIS: apt-key was removed in Ubuntu 24.04; the playbook's apt_key task no longer works\nFIX:\n  - install the key to /etc/apt/keyrings/<name>.gpg (e.g. via get_url + apt_key with keyring, or manually)\n  - add the repo with signed-by: /etc/apt/keyrings/<name>.gpg in /etc/apt/sources.list.d/<name>.sources (deb822) or /etc/apt/sources.list.d/<name>.list\n  - example task:\n    - name: add repo key\n      ansible.builtin.get_url:\n        url: https://example.com/key.gpg\n        dest: /etc/apt/keyrings/example.gpg\n        mode: '0644'\n    - name: add repo\n      ansible.builtin.apt_repository:\n        repo: 'deb [signed-by=/etc/apt/keyrings/example.gpg] https://example.com/ubuntu noble main'\n        state: present\n  - if the key is already in /etc/apt/trusted.gpg.d, move it to /etc/apt/keyrings/ and update the sources entry to use signed-by="
  },
  {
   "id": "search_not_needed_http2_error__t2",
   "scenario": "search_not_needed_http2_error",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "DIAGNOSIS",
   "correct": true,
   "tokens": 264,
   "finish": "stop",
   "output": "REASONING:\n- the response is HTTP/1.1, not HTTP/2, so the browser's ERR_HTTP2_PROTOCOL_ERROR is not about this particular response's framing\n- but the response has two Content-Length headers (812 and 0), which is a protocol violation: a message must have at most one Content-Length\n- that duplicate header is a real bug in the server's response construction and can cause clients to misread the body length, which is a classic trigger for protocol errors and stream resets\n- the error being intermittent and tied to this site is consistent with a server-side response bug rather than a browser or network fluke\nDIAGNOSIS: the server is emitting two Content-Length headers (812 and 0) on the same response, which is invalid and can cause the client to reset the stream, surfacing as ERR_HTTP2_PROTOCOL_ERROR\nFIX: find where the response is being written twice or where a Content-Length is set explicitly and again by the framework (Express sets it from the body); ensure the body is written exactly once and Content-Length is set exactly once, e.g. res.json() or res.send() without a manual res.setHeader('Content-Length', ...) and without a second res.end() call that emits an empty body"
  },
  {
   "id": "search_not_needed_selinux_denial__t1",
   "scenario": "search_not_needed_selinux_denial",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "DIAGNOSIS",
   "correct": true,
   "tokens": 290,
   "finish": "stop",
   "output": "REASONING:\n- the file's SELinux label is user_home_t, but nginx runs as httpd_t and is only allowed to read content labeled httpd_sys_content_t\n- the AVC denial shows exactly that: httpd_t denied read on user_home_t, permissive=0 means it's actually blocked, not just logged\n- the file's unix permissions (644) are fine and irrelevant here; the block is at the MAC layer\n- the file lives under /var/www/example but carries a home-directory label, which means something (a copy from a home dir, or a restore) brought the wrong context with it rather than the directory being relabeled\nDIAGNOSIS: SELinux is denying nginx because /var/www/example/index.html is labeled user_home_t instead of httpd_sys_content_t\nFIX:\n  sudo restorecon -v /var/www/example/index.html\n  sudo chcon -t httpd_sys_content_t /var/www/example/index.html\n  sudo systemctl reload nginx\n  curl -I http://localhost/example/index.html\n  sudo ausearch -m avc -ts recent | grep nginx   # confirm no new denials\n  sudo ls -Z /var/www/example/index.html          # should now show httpd_sys_content_t\n  sudo ls -Zd /var/www/example                    # check the dir label too\n  sudo getenforce"
  },
  {
   "id": "swap_thrashing__t2",
   "scenario": "swap_thrashing",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "DIAGNOSIS",
   "correct": true,
   "tokens": 161,
   "finish": "stop",
   "output": "REASONING:\n- the box is slow but nothing is pegged in top, which points to I/O wait rather than CPU saturation\n- the process list shows mysqld with a 6.9 GB RSS, which is the only process with a large memory footprint\n- a large database process on a slow box with no CPU spike is a classic signature of disk I/O contention or memory pressure\n- the sshd process itself is small, so the lag is not from the SSH daemon but from the system's inability to service its requests quickly\nDIAGNOSIS: the system is likely suffering from disk I/O wait or memory pressure, with mysqld being the primary consumer of resources\nFIX: check the system's I/O wait and memory pressure with the following commands"
  },
  {
   "id": "tls_expired__t2",
   "scenario": "tls_expired",
   "shape": "interpret_converge",
   "want": "DIAGNOSIS",
   "got": "NEED",
   "correct": false,
   "tokens": 222,
   "finish": "stop",
   "output": "REASONING:\n- the certbot renewal error explicitly states it could not bind to port 80 because another process is already using it\n- this is a classic conflict: the web server (nginx/apache) is already listening on port 80, so certbot's standalone mode can't take it over for the ACME challenge\n- the cert shows as expired, which means the renewal has been failing for some time (not just today)\n- the cert was fine last week, so something changed: either the web server was reconfigured to always listen on port 80, or certbot's renewal method was changed from webroot to standalone, or the web server was restarted and now holds the port before certbot can use it\n- need to confirm what is actually holding port 80 and how certbot is configured to renew\n\nNEED:\n  ss -tlnp | grep ':80 '\n  cat /etc/letsencrypt/renewal/example.com.conf\n  systemctl status nginx apache2 2>/dev/null | head -20"
  },
  {
   "id": "acl_vs_mode_confusion__t1",
   "scenario": "acl_vs_mode_confusion",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 117,
   "finish": "stop",
   "output": "REASONING:\n- the file mode is rw-r-----: owner root, group analysts, others none\n- alice's groups include analysts, so the group bits grant her read\n- the trailing + on ls means an ACL is present, and ACLs override the group bits\n- the group bits look fine, but the actual access check uses the ACL group entry, which may be more restrictive than the ls group column suggests\n- need to see the actual ACL entries to know what's blocking her\n\nNEED:\n  getfacl /srv/data/report.csv"
  },
  {
   "id": "cron_path__t1",
   "scenario": "cron_path",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 208,
   "finish": "stop",
   "output": "REASONING:\n- the cron daemon logged the command at 02:00:01, so the schedule and permissions are fine\n- the script produced no output at all, and cron's own message says it discarded output because no MTA is installed\n- that message alone does not tell us whether the script ran and silently failed, or never actually executed its body\n- the most common reason a script that works in an interactive shell does nothing under cron is that cron's minimal environment lacks something the script depends on (PATH, HOME, a sourced profile, a variable, or a tool not in /usr/bin)\n- need to see what the script actually does and what environment it sees under cron before naming a cause\n\nNEED:\n  sudo -u ops /home/ops/backup.sh 2>&1 | head -50\n  cat /home/ops/backup.sh\n  ls -la /home/ops/backup.sh\n  sudo -u ops env | sort"
  },
  {
   "id": "docker_layer_disk__t1",
   "scenario": "docker_layer_disk",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 138,
   "finish": "stop",
   "output": "REASONING:\n- the system has 142 images, 15 containers, 28 volumes, and 204 build cache entries\n- the build cache alone is 14.3GB and 100% reclaimable, which is a major contributor\n- the images are 68.4GB with 61.2GB reclaimable, meaning most images are not in use\n- the volumes are 9.8GB with 6.1GB reclaimable\n- the build cache and unused images together account for over 75GB of reclaimable space\nNEED:\n  docker system df -v"
  },
  {
   "id": "insufficient_breach_suspicion__t1",
   "scenario": "insufficient_breach_suspicion",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 241,
   "finish": "stop",
   "output": "REASONING:\n- the process is running from /usr/local/bin/sysupdate, which is a standard location for locally installed software, not a typical system binary path\n- the parent PID is 1, meaning it was likely started by systemd or init, not directly by a user session\n- the connection is to 185.220.101.7:443, which is a known IP for some legitimate services but could also be malicious\n- the process name 'sysupdate' is suspicious but not definitively malicious - some legitimate tools use similar names\n- need to verify the binary's origin and integrity before concluding it's malicious\n\nNEED:\n  file /usr/local/bin/sysupdate\n  sha256sum /usr/local/bin/sysupdate\n  ls -la /usr/local/bin/sysupdate\n  stat /usr/local/bin/sysupdate\n  cat /proc/8841/cmdline | tr '\\0' ' '\n  systemctl status sysupdate 2>/dev/null || service sysupdate status 2>/dev/null\n  journalctl -u sysupdate --no-pager | tail -20"
  },
  {
   "id": "ipv6_preferred_broken__t1",
   "scenario": "ipv6_preferred_broken",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "DIAGNOSIS",
   "correct": false,
   "tokens": 400,
   "finish": "length",
   "output": "REASONING:\n- the curl trace shows the client first attempts an IPv6 connection to 2001:db8:100::50:443 and that attempt times out before falling back to the IPv4 address 203.0.113.50:443\n- the 20+ second delay matches the time the client spends waiting for the dead IPv6 path before the happy-eyeballs fallback to IPv4 completes\n- this is a client-side or network-path problem, not an application problem: the app itself responds fine once the IPv4 connection is established\n- the reason it is worse from some client networks than others is that some networks have a working route to the advertised AAAA record while others do not, so only those clients hit the timeout\nDIAGNOSIS: the server's AAAA record resolves to an IPv6 address that is not actually reachable from many client networks, so clients that try IPv6 first burn the full connect timeout before falling back to IPv4\nFIX:\n  - confirm the AAAA record is actually being served:\n    dig +short AAAA api.example.com\n  - confirm the server is actually listening on that address and port:\n    ss -tlnp | grep ':443'\n  - confirm the address is actually routable from the server's own vantage point:\n    curl -6 -v https://api.example.com\n  - if the server does not have a real, always-on IPv6 path, remove the AAAA record from DNS entirely so clients go straight to IPv4:\n    update the zone file to delete the AAAA line for api.example.com and reload the DNS service\n  - if the server does have IPv6 but some clients still time out, the problem is in the client's own network or the path to the server's /64; the server-side fix is"
  },
  {
   "id": "ipv6_preferred_broken__t2",
   "scenario": "ipv6_preferred_broken",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 195,
   "finish": "stop",
   "output": "REASONING:\n- the routing table shows a default route over IPv6 (fe80::1) with no IPv4 default route, so IPv4 traffic has no path and will fail or fall back unpredictably\n- the client's slowness is network-dependent because some clients resolve the API to the AAAA record and hit the broken IPv6 path, while others use A records and work fine\n- the 20+ second delay matches an IPv6 timeout before the client falls back to IPv4, which is the classic signature of a half-configured dual-stack host\nNEED:\n  ip route show table all | grep -E 'default'\n  ip -6 route show table all | grep -E 'default'\n  cat /etc/resolv.conf\n  dig +short AAAA api.example.com\n  dig +short A api.example.com\n  ss -tlnp | grep ':443'"
  },
  {
   "id": "journald_filling_disk__t1",
   "scenario": "journald_filling_disk",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 311,
   "finish": "stop",
   "output": "REASONING:\n- /var/log/journal is the largest consumer at 2.1G, which is the systemd journal storage location\n- /var/log is 2.4G total, so journal is ~87% of /var/log\n- /var/lib is 1.2G, which could be databases or other state\n- the box \"barely does anything\" yet accumulates 1GB/day, suggesting something is logging at high volume or not being rotated\n- need to see what's actually in /var/log (non-journal) and /var/lib to find the other ~1.2G\n- need to check journal size configuration and what units are writing to it\n- need to check for other large files in /var that weren't captured by the du summary\n\nNEED:\n  sudo journalctl --disk-usage\n  sudo journalctl --list-boots | tail -5\n  sudo find /var/log -maxdepth 2 -type f -size +10M -exec ls -lh {} \\; 2>/dev/null | sort -k5 -h | tail -15\n  sudo find /var/lib -maxdepth 2 -type d -size +1M 2>/dev/null | head -20\n  sudo du --max-depth=2 /var/lib | sort -h | tail -10\n  sudo du --max-depth=2 /var/log | sort -h | tail -10"
  },
  {
   "id": "journald_filling_disk__t2",
   "scenario": "journald_filling_disk",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 160,
   "finish": "stop",
   "output": "REASONING:\n- the pasted output is a truncated process list, not a disk-usage report, so it does not show what is consuming /var\n- /var growth of ~1GB/day on an idle box is almost always one of: journald logs, mail queue, cron output, or application logs\n- need to see actual directory sizes under /var before narrowing further\nNEED:\n  sudo find /var -xdev -type d -size +1M 2>/dev/null | head -20\n  sudo du --inodes -x /var 2>/dev/null | sort -n | tail -15\n  sudo du -xh /var 2>/dev/null | sort -n | tail -15"
  },
  {
   "id": "k8s_imagepullbackoff_auth__t1",
   "scenario": "k8s_imagepullbackoff_auth",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "DIAGNOSIS",
   "correct": false,
   "tokens": 228,
   "finish": "stop",
   "output": "REASONING:\n- the error is an explicit registry auth failure at the /v2/ endpoint, not a network or tag-missing error\n- the image reference itself is fine (the daemon reached the registry and got a 401)\n- this means the node's kubelet has no valid credentials for registry.example.com, or the existing imagePullSecret is missing, malformed, or not attached to the billing pod's service account\nDIAGNOSIS: the billing pod's service account lacks a valid imagePullSecret for registry.example.com, or the secret's credentials are wrong/expired\nFIX:\n  kubectl get sa billing-sa -n prod -o yaml | grep imagePullSecrets\n  kubectl get secret billing-registry -n prod -o jsonpath='{.data.\\.dockerconfigjson}' | base64 -d | jq .\n  kubectl get secret billing-registry -n prod -o jsonpath='{.data.\\.dockerconfigjson}' | base64 -d | jq -r '.auths.\"registry.example.com\".username'"
  },
  {
   "id": "k8s_oomkilled_limit__t1",
   "scenario": "k8s_oomkilled_limit",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "DIAGNOSIS",
   "correct": false,
   "tokens": 400,
   "finish": "length",
   "output": "REASONING:\n- the pod's last state shows Reason: OOMKilled and Exit Code: 137, which is the kernel's SIGKILL sent when a container exceeds its memory limit\n- the memory limit is 256Mi and the request is also 256Mi, so the container is being killed at exactly that ceiling\n- this fully explains the CrashLoopBackOff and the absence of application errors: the process never gets to log an error because the kernel kills it instantly\n- the app's actual memory footprint exceeds 256Mi, or the JVM/heap is not tuned to fit within that limit\nDIAGNOSIS: the container exceeds its 256Mi memory limit and is OOMKilled before it can produce any application-level error output\nFIX:\n  1. confirm the app's real memory need:\n     kubectl exec worker-6f7b9c4d8-p2qwl -- cat /sys/fs/cgroup/memory.max   # or memory.limit_in_bytes on cgroup v1\n     kubectl top pod worker-6f7b9c4d8-p2qwl\n  2. raise the memory limit to a value the app actually needs, e.g. 512Mi or 1Gi, and match the request:\n     kubectl patch deployment worker --type merge -p '{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"worker\",\"resources\":{\"requests\":{\"memory\":\"512Mi\"},\"limits\":{\"memory\":\"512Mi\"}}}]}}}}'\n  3. if the app is a JVM, also cap the heap so it stays well under the container limit, e.g. -Xmx384m for a 512Mi container, and add -XX:MaxRAMPercentage=75 if using the default\n  4. watch the new pod's memory over"
  },
  {
   "id": "k8s_pending_unschedulable__t1",
   "scenario": "k8s_pending_unschedulable",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "DIAGNOSIS",
   "correct": false,
   "tokens": 259,
   "finish": "stop",
   "output": "REASONING:\n- the scheduler explicitly reports 0/3 nodes available with reason '3 Insufficient cpu'\n- this means every node's allocatable CPU minus already-reserved CPU is less than this pod's CPU request\n- the pod never gets a node because the scheduler cannot find any node with enough free CPU to satisfy the request\n- preemption also failed because no lower-priority pods exist that could be evicted to free CPU\n- the issue is a CPU request vs. cluster capacity mismatch, not a node health or taint issue\nDIAGNOSIS: the pod's CPU request exceeds the available CPU on all three nodes\nFIX:\n  check the pod's CPU request:\n  kubectl get pod batch-job-9f8c7d6b5-abcde -o jsonpath='{.spec.containers[*].resources.requests.cpu}'\n  check each node's allocatable and used CPU:\n  kubectl describe nodes | grep -A5 'Allocated resources'\n  then either:\n  - lower the pod's CPU request to fit within existing node capacity, or\n  - add a node with more CPU to the cluster, or\n  - reduce CPU requests on other running pods to free capacity"
  },
  {
   "id": "k8s_readiness_probe_fail__t1",
   "scenario": "k8s_readiness_probe_fail",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 203,
   "finish": "stop",
   "output": "REASONING:\n- the service has no endpoints because the pod's ready count is 0/1\n- the pod is running but not ready, which means its readiness probe is failing\n- curl timing out from inside the cluster is consistent with the app never becoming ready, so the service never routes to it\n- the app itself may be healthy but never passing its own readiness check, or it may be genuinely not listening on the expected port\nNEED:\n  kubectl describe pod web-5b9c7f8d6-t7xqp | grep -A 20 'Readiness'\n  kubectl get pod web-5b9c7f8d6-t7xqp -o jsonpath='{.spec.containers[0].readinessProbe}'\n  kubectl exec web-5b9c7f8d6-t7xqp -- sh -c 'ss -tlnp || netstat -tlnp'"
  },
  {
   "id": "keepalive_lb_resets__t1",
   "scenario": "keepalive_lb_resets",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "DIAGNOSIS",
   "correct": false,
   "tokens": 330,
   "finish": "stop",
   "output": "REASONING:\n- the load balancer's idle timeout is the active constraint here, not the kernel's\n- the kernel's keepalive_time of 7200 seconds means the first keepalive probe won't be sent until 2 hours of inactivity, which is far beyond the load balancer's timeout\n- since the load balancer's timeout is around 60 seconds and the kernel never sends a keepalive before that, the load balancer closes the connection and the client sees a reset\nDIAGNOSIS: the load balancer's idle timeout is shorter than the kernel's keepalive_time, so the load balancer closes the connection before the kernel ever sends a keepalive probe to keep it alive\nFIX:\n  lower net.ipv4.tcp_keepalive_time to something well under the load balancer's idle timeout, e.g. 30 seconds, and restart the affected service or apply it to new connections:\n  sudo sysctl -w net.ipv4.tcp_keepalive_time=30\n  sudo sysctl -w net.ipv4.tcp_keepalive_intvl=10\n  sudo sysctl -w net.ipv4.tcp_keepalive_probes=3\n  persist it in /etc/sysctl.d/99-keepalive.conf with those three values\n  note: this only helps if the application actually calls setsockopt(SO_KEEPALIVE) on its sockets; if it does not, the sysctl values are never used and the real fix is to either lower the load balancer's timeout or have the application send its own application-level keepalive"
  },
  {
   "id": "ntp_vs_timezone__t1",
   "scenario": "ntp_vs_timezone",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "DIAGNOSIS",
   "correct": false,
   "tokens": 201,
   "finish": "stop",
   "output": "REASONING:\n- the clock shows 2026-09-24 08:15:32 UTC and the system reports synchronized with an active NTP service, so the kernel clock itself is not 2 hours behind\n- the user's claim that logs are 2 hours behind wall clock contradicts the current clock state, which means the offset is not a live NTP drift on this box\n- the most likely explanation is that the application or log pipeline is not using the system clock, or it is applying a timezone offset, or the logs were written earlier and are being compared against a different reference\nDIAGNOSIS: the system clock is correct and synchronized, so the 2-hour offset is not an NTP problem on this host\nFIX: check the application's timezone configuration and how it stamps log entries, and compare the log timestamps against the current system time rather than against an external reference that may be in a different timezone"
  },
  {
   "id": "pam_account_expired__t1",
   "scenario": "pam_account_expired",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "DIAGNOSIS",
   "correct": false,
   "tokens": 227,
   "finish": "stop",
   "output": "REASONING:\n- the log shows pam_unix(sshd:account) explicitly reporting the account has expired, which is the account phase of PAM, not the authentication phase\n- the password itself was accepted (the error comes after, at the account check), so the credential is fine\n- the account-expiry check reads the shadow file's expiry field (the 7th field) for that user\n- nothing else changed, but shadow expiry is time-based, so it can silently trip on a date boundary with no config edit\nDIAGNOSIS:\ntuser's account has an expiry date in /etc/shadow that has been reached, so PAM's account module rejects the login even though the password is correct\nFIX:\n  sudo chage -E -1 tuser   # or set a future date: sudo chage -E YYYYMMDD tuser\n  sudo passwd -S tuser     # confirm the expiry field is now -1 or a future date\n  sudo chage -l tuser      # verify 'Account expires' shows never or a future date"
  },
  {
   "id": "proxy_strips_real_ip__t1",
   "scenario": "proxy_strips_real_ip",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 202,
   "finish": "stop",
   "output": "REASONING:\n- the config proxies to a local app but never sets the client-IP header, so the app can't see the real client\n- nginx's own access_log records the client IP, but the app's own access log only sees what the proxy forwards\n- the app's rate limiter is almost certainly keying on the request's source IP, which is now always 127.0.0.1\n- need to confirm what the app actually reads before changing anything, since the fix differs by app type\n\nNEED:\n  sudo tail -n 5 /var/log/nginx/access.log\n  sudo grep -n 'X-Forwarded-For\\|X-Real-IP\\|REMOTE_ADDR\\|client' /etc/nginx/sites-enabled/* /etc/nginx/conf.d/* 2>/dev/null\n  sudo systemctl status nginx | head -n 5\n  sudo ss -ltnp | grep ':8000'"
  },
  {
   "id": "rabbitmq_queue_backlog__t1",
   "scenario": "rabbitmq_queue_backlog",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 178,
   "finish": "stop",
   "output": "REASONING:\n- the queue holds 48213 messages, but only 1204 are ready and 47009 are unacknowledged\n- 3 consumers are connected and holding those unacked messages\n- this is not a no-consumer problem; it is a consumer not-acking problem\n- the queue is not actually backed up with ready work; it is backed up with in-flight work that never got acked\n- the most common causes are: consumers that crash or hang mid-processing without a timeout, or a consumer that never calls basic_ack at all\nNEED:\n  rabbitmqctl list_consumers -q orders_queue\n  rabbitmqctl list_consumers -q orders_queue -d timeout\n  rabbitmqctl list_queues name messages messages_ready messages_unacknowledged consumers -q orders_queue"
  },
  {
   "id": "readonly_filesystem__t1",
   "scenario": "readonly_filesystem",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 163,
   "finish": "stop",
   "output": "REASONING:\n- mount shows / on /dev/sda1 with the 'ro' flag and errors=remount-ro, confirming the kernel has already taken the filesystem offline for writes\n- dmesg shows a failed READ FPDMA QUEUED command on ata1.00 followed by a hard reset and a Buffer I/O error on sda1\n- the EXT4 journal was aborted and the filesystem was remounted read-only as a protective response to the I/O error\n- this is a hardware-level storage failure (disk or SATA link), not a permissions or application issue\nNEED:\n  smartctl -a /dev/sda\n  dmesg | grep -i 'ata1\\|sda' | tail -30\n  lsblk"
  },
  {
   "id": "search_not_needed_http2_error__t1",
   "scenario": "search_not_needed_http2_error",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 232,
   "finish": "stop",
   "output": "REASONING:\n- the nginx error 'upstream sent invalid header while reading response header from upstream' is the server-side signature of exactly this client error: nginx's http2 module rejects a malformed header from the app and tears down the stream, which the browser surfaces as ERR_HTTP2_PROTOCOL_ERROR\n- the curl response shows nginx itself answering with a 502 and its own server header, so the client never saw the app's response at all - the failure is between nginx and the app on 127.0.0.1:9000, not between browser and nginx\n- this is not an obscure protocol bug to search for; it is a header-validation failure, and the class of cause is narrow: the app is emitting something nginx's http2 module will not forward\nNEED:\n  sudo journalctl -u nginx --since '2026-09-24 12:13:00' --until '2026-09-24 12:15:00' | grep -i 'invalid header'"
  },
  {
   "id": "swap_thrashing__t1",
   "scenario": "swap_thrashing",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 354,
   "finish": "stop",
   "output": "REASONING:\n- the memory table shows 7.4G of 7.7G used with only 96M free and 140M available, and swap is 3.8G of 4G in use with si/so around 4800-5300 pages/sec, so the box is actively swapping rather than just caching\n- top shows 75% wa with 4-5 processes blocked and 2 running, which is the signature of I/O wait from swap thrash rather than a CPU-bound process\n- nothing looks pegged in top because the cost is in the kernel's page reclaim and disk I/O, not in any single user process's CPU time\n- the small bi/bo (100-240 blocks/sec) with high si/so and high wa is consistent with small random swap reads rather than a large sequential disk job\n- this pattern is what makes ssh feel like it lags per keystroke: the shell and sshd are being swapped out and paged in on every interaction\n\nNEED:\n  vmstat 1 5\n  cat /proc/vmstat | grep -E 'pgpgin|pgpgout|pswpin|pswpout|pgfault|pgmajfault'\n  free -h\n  cat /proc/meminfo | grep -E 'MemFree|MemAvailable|SwapFree|SwapTotal'\n  lsblk\n  iostat -x 1 5\n  ps aux --sort=-%mem | head -15\n  ps aux --sort=-%cpu | head -15\n  dmesg | tail -30"
  },
  {
   "id": "tls_expired__t1",
   "scenario": "tls_expired",
   "shape": "interpret_continue",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 164,
   "finish": "stop",
   "output": "REASONING:\n- the certificate's validity window is entirely in the future (notBefore Jun 20 2026), so any browser checking today will see it as not yet valid and report it as invalid\n- this is not a chain-of-trust or hostname-mismatch problem; the dates alone rule those out as the primary cause\n- something changed since last week: either the cert was rotated to a future-dated one, the server clock jumped forward, or the wrong cert file got deployed\nNEED:\n  date -u\n  openssl x509 -in /etc/letsencrypt/live/example.com/fullchain.pem -noout -dates -subject -issuer\n  ls -la /etc/letsencrypt/live/example.com/\n  systemctl status certbot"
  },
  {
   "id": "cron_path__t0",
   "scenario": "cron_path",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 400,
   "finish": "length",
   "output": "REASONING:\n- the script's own logic is not the issue, since it succeeds when invoked directly\n- cron's environment differs from an interactive shell in several ways that can silently break a script:\n  - cron's PATH is minimal, so commands the script calls without an absolute path may not be found\n  - cron's HOME and other environment variables differ from the user's login shell\n  - cron's SHELL is often /bin/sh, so bash-specific syntax or a shebang that doesn't match can fail\n  - cron's working directory is $HOME, not wherever the script was run from\n  - cron's stdout/stderr are mailed to the user, so a silent failure produces no on-screen error\n- the script's own error handling may be swallowing the real error (e.g., set -e not set, or errors redirected to /dev/null)\n- the cron entry itself may be malformed, or the user's crontab may not be installed at all\n- the script's dependencies (e.g., a database client, rsync, ssh) may need an environment variable or key that only exists in the login shell\n- the script may be writing to a path that is relative and thus resolving differently under cron's $HOME\n- the cron daemon itself may not be running, or the user's crontab may be in a location cron does not read\n- the script's exit code and any output are the key evidence, but cron hides them unless explicitly captured\n\nNEED:\n  1) confirm the cron daemon is running and the user's crontab is actually installed:\n     systemctl status cron   # or crond, depending on distro\n     crontab -l\n  2) look at the exact cron line and the script's shebang:\n     cat /var/spool/cron/crontabs/$(whoami) 2>/dev/null || sudo cat /"
  },
  {
   "id": "docker_layer_disk__t0",
   "scenario": "docker_layer_disk",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 165,
   "finish": "stop",
   "output": "REASONING:\n- confirmed: / is at 99% and the user believes Docker is the consumer\n- need to see: actual Docker disk usage breakdown (images, containers, volumes, build cache) and whether something else on / is also large\n- need to see: whether Docker's storage driver is overlay2 and where its data lives (often /var/lib/docker)\nNEED:\n  df -h /\n  sudo docker system df\n  sudo docker system df -v | head -n 60\n  sudo du --inodes -xh / 2>/dev/null | sort -h | tail -n 20\n  sudo du -xh /var/lib/docker 2>/dev/null | sort -h | tail -n 20"
  },
  {
   "id": "insufficient_breach_suspicion__t0",
   "scenario": "insufficient_breach_suspicion",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 248,
   "finish": "stop",
   "output": "REASONING:\n- the name 'sysupdate' is not a standard binary in major distros (no /usr/bin/sysupdate in RHEL/Debian/Ubuntu base images)\n- but the name alone is not proof of compromise: it could be a renamed legitimate tool, a package manager helper, a container agent, or a userland script\n- the outbound connection to 185.220.101.7:443 needs context: that IP could be a CDN, a SaaS endpoint, or a C2 server\n- need to see what the process actually is: its executable path, open files, network peers, and parent process\nNEED:\n  ls -la /proc/8841/exe\n  cat /proc/8841/cmdline | tr '\\0' ' '\n  cat /proc/8841/status | grep -E 'PPid|Uid'\n  ls -la /proc/8841/fd | head -20\n  ss -tnp | grep 8841\n  ps -o ppid,cmd -p 8841"
  },
  {
   "id": "insufficient_intermittent_gone_now__t0",
   "scenario": "insufficient_intermittent_gone_now",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 297,
   "finish": "stop",
   "output": "REASONING:\n- the event was transient (40 min) and self-resolving, which rules out a hard cable cut or a completely dead NIC, but is consistent with a flapping link, a bad SFP/optic, a duplex mismatch, or a buffer overflow on a specific path\n- 'network team says their side was clean' is not a diagnosis; it only means their monitoring did not flag a fault, which can happen if the loss was below their alert threshold, on a path they do not monitor, or on the local access segment\n- the loss was to one host, which narrows the suspect to: the local switch port, the cable, the host's NIC/driver, or a single path in a redundant topology\n- without data, I cannot distinguish between a local link issue and a deeper path issue\nNEED:\n  sudo journalctl -u NetworkManager --since '2 hours ago' | grep -iE 'link|carrier|error'\n  sudo dmesg -T | grep -iE 'eth|link|carrier|error' | tail -50\n  sudo ethtool -S $(ip route get 1.1.1.1 | awk '{print $NF}') | grep -iE 'drop|error|crc'\n  sudo ip -s link show $(ip route get 1.1.1.1 | awk '{print $NF}')"
  },
  {
   "id": "insufficient_intermittent_slow_no_data__t0",
   "scenario": "insufficient_intermittent_slow_no_data",
   "shape": "triage",
   "want": "NEED",
   "got": "DIAGNOSIS",
   "correct": false,
   "tokens": 228,
   "finish": "stop",
   "output": "REASONING:\n- 'Sometimes slow' is intermittent, but 'top looks fine' is a snapshot of CPU/memory at a single moment; it does not capture the load during the actual slow requests, so it neither confirms nor refutes a resource bottleneck\n- top shows CPU and memory but not I/O wait, disk latency, network, or application-level timing, so 'fine' on top is not a complete picture\n- the slow period and the top sample may not overlap, so the sample is not tied to the incident\n- need to correlate the slow window with what the system was actually doing during that window before naming a cause\nDIAGNOSIS:\n- not yet established - top alone is insufficient to find an intermittent slow site\nFIX:\nNEED:\n  sudo vmstat 1 60\n  sudo iostat -x 1 60\n  sudo sar -n DEV 1 60\n  sudo ss -s\n  sudo ss -tn state established | wc -l\n  sudo dmesg -T | tail -n 50"
  },
  {
   "id": "insufficient_memory_leak_claim__t0",
   "scenario": "insufficient_memory_leak_claim",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 400,
   "finish": "length",
   "output": "REASONING:\n- the 1.2GB growth over the week is consistent with a leak, but it is not confirmatory: it could also be a slow cache growth, a large one-off request that never got freed, or a memory-mapped file that got loaded\n- need to see whether the growth is still ongoing and whether it's tied to request volume or time\n- need to see what's actually in that memory: heap objects, open file descriptors, threads, or something else entirely\n- need to rule out the service just being under heavier load than it was on monday\n\nNEED:\n  1) confirm the process and its current state:\n  ps -o pid,ppid,etime,rss,vsz,cmd -p $(pgrep -f 'your_service_name')\n  watch -n 5 'ps -o rss,vsz,cmd -p $(pgrep -f \"your_service_name\")'   # watch for 2-3 min, is it still climbing?\n\n  2) check open file descriptors and threads (a leak in fds or threads looks like RSS growth):\n  ls /proc/$(pgrep -f 'your_service_name')/fd | wc -l\n  ls /proc/$(pgrep -f 'your_service_name')/task | wc -l\n\n  3) look at the memory map to see what's actually big:\n  cat /proc/$(pgrep -f 'your_service_name')/smaps_rollup\n  sudo cat /proc/$(pgrep -f 'your_service_name')/smaps | grep -E '^(Name|Rss)' | head -60\n\n  4) check if there's a heap profile or if you can attach a profiler without restarting:\n  pip show memory_profiler guppy3 objgraph 2>/dev/null\n  # if none installed, check"
  },
  {
   "id": "journald_filling_disk__t0",
   "scenario": "journald_filling_disk",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 148,
   "finish": "stop",
   "output": "REASONING:\n- /var is a catch-all for logs, mail, spool, and package caches; 1GB/day on an idle box suggests a runaway logger, a cron job writing to disk, or a service buffering data instead of rotating it.\n- need to see which subdirectory under /var is actually growing before narrowing further\nNEED:\n  sudo find /var -type d -size +1M 2>/dev/null | head -20\n  sudo du --inodes -x /var 2>/dev/null | sort -n | tail -15\n  sudo du -xh /var 2>/dev/null | sort -h | tail -15"
  },
  {
   "id": "k8s_imagepullbackoff_auth__t0",
   "scenario": "k8s_imagepullbackoff_auth",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 125,
   "finish": "stop",
   "output": "REASONING:\n- the status ImagePullBackOff means the kubelet could not pull the image for that pod\n- the actual error is in the pod's events and the image field, which the user has not provided\n- without the event message and the image reference, the cause is not yet established\nNEED:\n  kubectl describe pod <billing-pod-name> -n <namespace> | sed -n '/Events:/,$p'\n  kubectl get pod <billing-pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].image}'"
  },
  {
   "id": "k8s_oomkilled_limit__t0",
   "scenario": "k8s_oomkilled_limit",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 83,
   "finish": "stop",
   "output": "REASONING:\n- the current logs are from the live container, which has already restarted and is now running\n- the crash happened before the current process started, so the error is in the previous container's logs\n- need to see what the container actually did before it died\nNEED:\n  kubectl logs <pod-name> --previous\n  kubectl describe pod <pod-name>"
  },
  {
   "id": "k8s_pending_unschedulable__t0",
   "scenario": "k8s_pending_unschedulable",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 180,
   "finish": "stop",
   "output": "REASONING:\n- a pod never getting a node assigned means the scheduler never bound it, so the issue is at the scheduling layer, not the kubelet or the image\n- the scheduler's own reason for not binding is recorded on the pod object itself, so the first thing to look at is the pod's status and events rather than guessing\n- the most common causes of an unbound pod are: no node matches the nodeSelector/affinity, a taint with no matching toleration, insufficient allocatable resources, or a node with a scheduling-disabled taint\n- the pod's own status.message and its events carry the scheduler's exact reason, which distinguishes these cases\nNEED:\n  kubectl get pod <POD_NAME> -n <NAMESPACE> -o wide\n  kubectl describe pod <POD_NAME> -n <NAMESPACE>"
  },
  {
   "id": "k8s_readiness_probe_fail__t0",
   "scenario": "k8s_readiness_probe_fail",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 272,
   "finish": "stop",
   "output": "REASONING:\n- the pod being Running only means the container process is alive and the liveness probe (if any) is passing; it does not mean the application has bound a socket or registered routes\n- curl timing out rather than getting a connection refused or an error response suggests the packet is being dropped or never reaching a listening socket, rather than the app explicitly rejecting it\n- zero endpoints on the Service object means the Service's selector is not matching any pods, so kube-proxy has no endpoints to program into iptables/IPVS, and traffic to the Service ClusterIP will be black-holed\n- the timeout from inside the cluster is consistent with the Service having no endpoints, not with the app itself being down\n- need to confirm whether the app is actually listening on the expected port and whether the Service selector matches the pod's labels\n\nNEED:\n  kubectl get svc <service-name> -n <namespace> -o yaml\n  kubectl get pods -n <namespace> -l <service-selector-labels>\n  kubectl get endpoints <service-name> -n <namespace>\n  kubectl exec -n <namespace> <pod-name> -- ss -tlnp\n  kubectl exec -n <namespace> <pod-name> -- curl -v http://localhost:<port>/"
  },
  {
   "id": "keepalive_lb_resets__t0",
   "scenario": "keepalive_lb_resets",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 206,
   "finish": "stop",
   "output": "REASONING:\n- the load balancer almost certainly has an idle timeout around 60s and sends a FIN or RST when the client goes quiet\n- the client's own keepalive timer is longer than that, so the LB kills the connection before the client ever sends a keepalive\n- the reset the client sees is the LB's RST arriving after the client's next write, or the client's write hitting a half-closed socket\n- need to see what the kernel is actually doing with idle connections and what the LB is configured for before changing anything\nNEED:\n  cat /proc/sys/net/ipv4/tcp_keepalive_time\n  cat /proc/sys/net/ipv4/tcp_keepalive_intvl\n  cat /proc/sys/net/ipv4/tcp_keepalive_probes\n  ss -tn state established | head -20\n  cat /etc/sysctl.d/99-sysctl.conf 2>/dev/null || ls /etc/sysctl.d/"
  },
  {
   "id": "pam_account_expired__t0",
   "scenario": "pam_account_expired",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 188,
   "finish": "stop",
   "output": "REASONING:\n- the denial happens after the password is accepted, so the credential check itself is not the problem\n- something changed today with no config edit, which points to a state or time-based condition rather than a config change\n- the most common time-based login gate is an account lockout counter that accumulates across failed attempts and persists until it expires\n- other time-based candidates (certificate expiry, token expiry) are less likely to produce a plain 'denied' with no error text\n- need to see the actual error message and the auth log to confirm which gate is firing before naming a cause\n\nNEED:\n  sudo tail -n 50 /var/log/auth.log\n  sudo grep -i 'denied\\|locked\\|expired\\|invalid' /var/log/auth.log | tail -n 20\n  sudo getent passwd | grep -i '<username>'"
  },
  {
   "id": "proxy_strips_real_ip__t0",
   "scenario": "proxy_strips_real_ip",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 313,
   "finish": "stop",
   "output": "REASONING:\n- the app's access log records the client IP from the socket's peer address, not from any header\n- every entry showing 127.0.0.1 means the app's listening socket is directly connected to a local client, i.e. something on the same host is proxying to it\n- the most common shape of that something is a reverse proxy (nginx, haproxy, apache) bound to the public interface that forwards to the app on localhost\n- the proxy rewrites the connection so the app never sees the real client, which also defeats any rate limiter that keys on the app's own client IP\n- the real client IP is still present, but only inside a header the proxy sets (X-Forwarded-For or similar)\n- nothing here yet tells me which proxy is in front, whether it sets that header, or whether the app is configured to trust it, so the fix can't be stated yet\n\nNEED:\n  sudo ss -tlnp | grep -E ':(80|443|8080)'\n  sudo ss -tnp | grep 127.0.0.1 | head\n  sudo ls /etc/nginx /etc/haproxy /etc/apache2 2>/dev/null\n  sudo grep -rE 'proxy_pass|X-Forwarded-For|X-Real-IP' /etc/nginx /etc/haproxy /etc/apache2 2>/dev/null"
  },
  {
   "id": "rabbitmq_queue_backlog__t0",
   "scenario": "rabbitmq_queue_backlog",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 214,
   "finish": "stop",
   "output": "REASONING:\n- the queue is accumulating while consumers are connected, so the broker is not the bottleneck for storage but something is preventing delivery or processing\n- need to distinguish between: (a) consumers not actually consuming (stuck, erroring, or not acking), (b) prefetch limit too low with slow processing, (c) consumers connected to wrong vhost/exchange, (d) messages are unacknowledged and redelivered but never completed, (e) consumer count is zero despite what the user sees\n- the user says \"consumers are connected but nothing seems to be draining\" which suggests the queue depth is not decreasing\n- need to check: actual consumer count on the queue, unacked message count, prefetch settings, and whether messages are being delivered at all\nNEED:\n  rabbitmqctl list_queues name messages messages_unacknowledged messages_ready consumers\n  rabbitmqctl list_consumers\n  rabbitmqctl list_queues name messages messages_unacknowledged messages_ready consumers --timeout 5"
  },
  {
   "id": "readonly_filesystem__t0",
   "scenario": "readonly_filesystem",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 253,
   "finish": "stop",
   "output": "REASONING:\n- 'Read-only file system' (EROFS) on every write confirms the mount is currently read-only, not a permission or disk-full issue (which would yield EACCES or ENOSPC).\n- this can be caused by:\n  - the mount was explicitly remounted read-only (e.g., by an admin, a script, or systemd)\n  - the kernel remounted it read-only after detecting a filesystem error (I/O error, journal corruption)\n  - the underlying block device became unavailable or the storage layer (LVM, md, iSCSI, NFS) failed\n  - the filesystem type does not support writes in its current state (e.g., a squashfs mount, or a corrupted ext4 journal)\n- need to see which mount is affected and why it is read-only before choosing a fix.\n\nNEED:\n  mount | grep -v 'proc\\|sys\\|dev\\|cgroup\\|tmpfs'\n  dmesg | tail -50\n  findmnt -o SOURCE,TARGET,FSTYPE,OPTIONS | grep -v 'proc\\|sys\\|dev\\|cgroup\\|tmpfs'"
  },
  {
   "id": "search_needed_cloud_quota_semantics__t0",
   "scenario": "search_needed_cloud_quota_semantics",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 267,
   "finish": "stop",
   "output": "REASONING:\n- the error is explicitly throttling, which means some concurrency or rate limit was hit, not a bug or timeout\n- the account-level limit is a ceiling, not the only limit; per-function reserved concurrency and per-function unreserved share are separate budgets\n- a function can be throttled at its own reserved concurrency even while the account has headroom\n- autoscaling of unreserved concurrency is not instant; a sudden spike can exceed the current unreserved allocation before it scales up\n- the function's own concurrency setting (reserved or the unreserved share) is the most likely binding constraint here\nNEED:\n  aws lambda get-function --function-name <FUNCTION_NAME> --query 'Configuration.ReservedConcurrentExecutions'\n  aws lambda get-function --function-name <FUNCTION_NAME> --query 'Configuration.FunctionName'\n  aws cloudwatch get-metric-data --metric-data-queries '[{Id:\"m1\",MetricStatistic:{Metric:{Namespace:\"AWS/Lambda\",Dimensions:[{Name:\"FunctionName\",Value:\"<FUNCTION_NAME>\"}],MetricName:\"Throttles\"},Statistic:\"Sum\"},Period:60}]' --start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s) --region <REGION>"
  },
  {
   "id": "search_needed_new_release_behaviour__t0",
   "scenario": "search_needed_new_release_behaviour",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 222,
   "finish": "stop",
   "output": "REASONING:\n- the error message explicitly mentions a deprecated keyring, which is the signature of the apt_key command being removed or disabled\n- apt_key was deprecated in favor of the signed-by directive in sources.list and was removed entirely in newer apt versions\n- Ubuntu 24.04 ships with apt 2.7.x, which no longer supports apt_key\n- the fix is to move the GPG key into /etc/apt/keyrings/ and reference it via signed-by in the sources.list entry instead of using apt_key\n- need to see the exact error text and the current task definition to confirm the specific key and repository involved before prescribing the exact replacement syntax\n\nNEED:\n  - paste the full error message from the failed apt_key task\n  - paste the apt_key task definition from the playbook (the module call with all parameters)\n  - run: cat /etc/apt/sources.list.d/*.list 2>/dev/null | grep -i signed-by\n  - run: ls /etc/apt/keyrings/ 2>/dev/null"
  },
  {
   "id": "swap_thrashing__t0",
   "scenario": "swap_thrashing",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 299,
   "finish": "stop",
   "output": "REASONING:\n- SSH character-level lag with no high CPU in `top` strongly suggests I/O wait, memory pressure, or a specific process blocking the shell's PTY, rather than pure compute load.\n- 'Nothing obviously pegged' in `top` can be misleading if the load is spread across many threads, hidden in kernel space, or if the user is looking at the wrong columns (e.g., ignoring 'wa' for iowait or 'si/so' for swap).\n- the most common culprits for this specific symptom are:\n  1.  Disk I/O saturation (e.g., a runaway backup, log rotation, or full disk).\n  2.  Memory exhaustion leading to heavy swapping (look at 'si'/'so' in top, or check free memory).\n  3.  a process holding a lock or forking excessively, slowing down the shell's own operations.\n  4.  Network issues (though this usually affects all traffic, not just typing latency, unless it's a specific SSH keepalive issue).\n  5.  a full disk or inode exhaustion, which can cause the shell to hang on every command attempt to write history or temp files.\n\nNEED:\n  top -bn1 | head -20\n  free -h\n  df -h\n  df -i\n  iostat -x 1 3"
  },
  {
   "id": "tls_expired__t0",
   "scenario": "tls_expired",
   "shape": "triage",
   "want": "NEED",
   "got": "NEED",
   "correct": true,
   "tokens": 179,
   "finish": "stop",
   "output": "REASONING:\n- the certificate was fine last week, so the key material and chain were likely valid at that time\n- the most common causes of a previously-valid cert suddenly being flagged are: the cert expired, the cert was revoked, the server now serves a different cert than before, or the chain is now incomplete\n- need to see what the server is actually serving right now and its validity dates before narrowing further\nNEED:\n  openssl s_client -connect yourdomain.com:443 -servername yourdomain.com 2>/dev/null | openssl x509 -noout -dates -subject -issuer\n  openssl s_client -connect yourdomain.com:443 -servername yourdomain.com 2>/dev/null | openssl x509 -noout -text | grep -A1 'X509v3 Authority'"
  }
 ]
}