soul0101/telco-track-b-submission

Telco Troubleshooting Agentic Challenge — Track B Submission

Private artifact bundle for the Track B (Phase 2) submission of the Telco Troubleshooting Agentic Challenge.

This repo holds a single archive, code.zip, conforming to the competition SUBMISSION.md spec.

What's inside code.zip

track_b_submission/
├── README.md
├── run.sh / run.py                # entry point
├── requirements.txt
├── .env.example                   # backend configuration template
├── src/
│   ├── agent.py                   # Qwen critique-and-refine wrapper
│   ├── llm_client.py              # vLLM + OpenRouter client
│   ├── harness.py                 # classify, test loader, evidence reader
│   ├── network_model.py           # per-question network state
│   ├── parsers.py                 # vendor CLI parsers
│   ├── solver_fault.py            # fault root-cause solver
│   ├── solver_path.py             # L3 path solver
│   ├── solver_topo.py             # L2 forwarding-path solver
│   ├── llm_fault.py               # fault prompt + validator + taxonomy
│   ├── runtime_logger.py          # per-problem runtime decorator (spec)
│   └── sandbox_client.py          # live /get_devices_list and /execute
├── data/
│   ├── test.json                  # Phase 2 questions (100)
│   └── vendors.json               # static device roster + vendor map
├── models/
│   └── deploy.sh                  # vLLM launch script (Qwen)
└── result/                        # produced by `bash run.sh`
    ├── README.md
    ├── results.csv
    ├── traces.json
    └── runtime.json

Approach (generic, topology-agnostic agent)

The solver makes no assumptions about device names, question phrasing, or topology. For each scenario:

1. Device discovery (3-tier, fault-tolerant): POST /ip/api/agent/get_devices_list → if unavailable, crawl the topology from the devices named in the question (BFS via LLDP neighbours / interface descriptions, validating each candidate by probing it) → static known-roster only as a last resort. All calls are Bearer-authenticated and serialized (one in-flight request per token; ≤500 calls/question).
2. Understanding (LLM): a router extracts the task type + parameters as structured JSON, with a roster-aware deterministic backstop (framing-agnostic; no regex classifier).
3. Deterministic compute engines do the actual reasoning — grounded entirely in live CLI output, so the LLM cannot hallucinate the result:
   • fault — taxonomy detectors (shutdown / routing-walk / OSPF / BGP / VXLAN / L3VPN / MAC / loopback / host-info), no cross-question baseline;
   • path — hop-by-hop L3 trace over routing tables (recovers faulted/locked tables), with L2-access insertion;
   • link/topology — reciprocity-resolved adjacency with canonical port names (from display current-configuration, bandwidth suffixes stripped).
4. Format rendering (LLM): the structured result is rendered into exactly the format the question specifies (separators, interface-name style, reason spelling), using only the computed facts.
5. Grounding gate + never-blank fallback: any answer line referencing a non-roster device is dropped; a grounded ReAct agent is the last resort.

Defaults are the straightforward ideal: live server, no cache, serial. Validated on the Phase-1 ground-truth set (fault/path/link-restoration) across two unseen topology families.

The agent produces the three competition-required outputs:

• results.csv — one final prediction per scenario
• traces.json — every completion produced
• runtime.json — per-problem wall time via the spec runtime_logger

How to run (summary)

unzip code.zip && cd track_b_submission
pip install -r requirements.txt
cp .env.example .env

# 1. Point the agent at your Qwen server (any OpenAI-compatible /v1 endpoint)
export LLM_BASE_URL=http://localhost:8001/v1   # or a remote Qwen host
export LLM_MODEL=Qwen3.5-35B-A3B
#   (models/deploy.sh is an optional convenience launcher for a local vLLM)

# 2. Point the agent at the organizer sandbox + your player token
export TRACK_B_SANDBOX_URL=https://124.71.227.61   # or https://120.46.145.77
export TRACK_B_API_TOKEN=ip-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# 3. Run on all 100 problems
bash run.sh --input data/test.json --output result
# Both flags default to the paths shown above, so this is equivalent:
bash run.sh

Device CLI output is fetched live per run (no device cache ships, and the cache is per-run only). The client serializes calls (one concurrent request per token) and stays under the 500-call-per-question budget. A full 100-scenario run is therefore serial and takes a few hours; use bash run.sh --ids 1,40,80 or --limit N for smoke tests.

See track_b_submission/README.md inside the archive for full details.