# X03 — Observability

**Spec version:** v1.0
**Depends on:** X04 (config), stdlib, prometheus_client (optional)
**Depended on by:** every module that does I/O

---

## 1. Responsibility

Provides logging, metrics, tracing, and self-diagnostics. No module imports `logging` directly; they import `get_logger(__name__)` from this module.

---

## 2. File layout

```
hearthnet/observability/
├── __init__.py        # exports: get_logger, metrics, trace, doctor
├── logging.py         # structured JSON logging
├── metrics.py         # Prometheus-compatible counters/histograms
├── tracing.py         # per-request trace IDs + ring buffer
└── doctor.py          # self-diagnostics
```

---

## 3. Logging

### 3.1 Public API

```python
# hearthnet.observability.logging

def configure(config: ObservabilityConfig) -> None:
    """Install handlers, formatters, rotation. Idempotent. Call once at startup."""

def get_logger(name: str) -> Logger:
    """Return a stdlib logger configured to emit JSON lines.
       Convention: name = module's __name__ (e.g. 'hearthnet.bus.router')."""

class JsonFormatter(logging.Formatter):
    """Renders LogRecords as one-line JSON: ts, level, logger, msg, **extras."""
```

### 3.2 Conventions

- Use `extra=` to attach structured fields: `log.info("routed", extra={"capability": "llm.chat", "to": node_id, "ms": 12})`
- Never `f"log message {variable}"` for production diagnostics; use structured fields instead
- Log levels:
  - `debug` — internal state, only useful with `--debug`
  - `info` — meaningful protocol events (manifest received, capability registered, peer joined)
  - `warning` — recoverable problem (signature failed, peer unreachable, quarantine)
  - `error` — unexpected failure (exception caught, service crash)
- Exceptions: always `log.exception("what happened", extra={...})` — captures traceback automatically
- Rate-limit noisy warnings: `RateLimitedLogger` wrapper, log at most once per second per (logger, message_key)

### 3.3 On-disk format

```json
{"ts":"2026-05-26T08:14:22.281Z","level":"info","logger":"hearthnet.bus.router","msg":"routed","trace_id":"01HXR...","capability":"llm.chat","to":"7H4G-...","ms":12}
```

One line per event. Files rotate daily at midnight UTC. Retention: `LOG_RETENTION_DAYS = 14` from constants.

---

## 4. Metrics

### 4.1 Public API

```python
# hearthnet.observability.metrics

def configure(config: ObservabilityConfig) -> None:
    """Set up registries, start the metrics endpoint if enabled."""

# Counter / histogram / gauge factory functions:
def counter(name: str, doc: str, labels: list[str] = []) -> Counter
def histogram(name: str, doc: str, labels: list[str] = [], buckets: list[float] | None = None) -> Histogram
def gauge(name: str, doc: str, labels: list[str] = []) -> Gauge

# Convenience for "everything else returns None when metrics disabled":
def disabled() -> bool
```

### 4.2 Standard metric set

```
hearthnet_requests_total{capability, result}                     counter
hearthnet_request_duration_ms{capability, quantile}              histogram
hearthnet_active_streams{capability}                             gauge
hearthnet_nodes_online{community}                                gauge
hearthnet_event_log_size{community}                              gauge
hearthnet_event_log_lamport_head{community}                      gauge
hearthnet_emergency_mode{state}                                  gauge   // 0 or 1
hearthnet_blob_storage_bytes                                     gauge
hearthnet_llm_tokens_generated_total{model, backend}             counter
hearthnet_llm_concurrent{model}                                  gauge
hearthnet_capability_health_success_rate{capability, node}       gauge
hearthnet_rate_limited_total{capability, reason}                 counter
hearthnet_signature_failures_total{reason}                       counter
```

### 4.3 Scrape endpoint

`GET /metrics` on the transport server (port 7080). Plain text, Prometheus format. No auth — same trust domain as the rest of the bus.

---

## 5. Tracing

### 5.1 Public API

```python
# hearthnet.observability.tracing

class Trace:
    trace_id:   str
    capability: str
    started_at: float       # monotonic seconds
    spans:      list[Span]

@contextmanager
def span(name: str, **extras) -> Iterator[Span]:
    """Open a sub-span on the current trace. Auto-closes."""

def new_trace(capability: str) -> Trace:
    """Start a new trace (typically at the top of a capability handler)."""

def current_trace() -> Trace | None:
    """Get the trace attached to the current asyncio task."""

def attach(trace: Trace) -> None:
    """Attach a trace to the current task. Used by transport when it receives a request with an X-HearthNet-Request-Id."""

def detach() -> None:
    """End the trace; record to the ring buffer; emit done log."""

def get_recent(n: int = 100) -> list[Trace]:
    """Return last N completed traces from the ring buffer (used by /trace endpoint)."""
```

### 5.2 Storage

Ring buffer in memory, `TRACE_RING_BUFFER = 10000` from constants. Optionally exported to OpenTelemetry in Phase 2.

### 5.3 Trace IDs are ULIDs

ULIDs are used because they sort by time and need no separate timestamp field.

---

## 6. Doctor

### 6.1 Public API

```python
# hearthnet.observability.doctor

@dataclass
class CheckResult:
    name:    str
    ok:      bool
    detail:  str
    fix:     str | None

def run_all(config: Config, bus: CapabilityBus) -> list[CheckResult]:
    """Run every check; return list of results."""

def run_one(name: str, config: Config, bus: CapabilityBus) -> CheckResult:
    """Run a single named check."""

# Each check is a registered function:
def register(name: str, check: Callable[[Config, CapabilityBus], CheckResult]) -> None
```

### 6.2 Standard checks

| Name | Verifies |
|------|----------|
| `keys_present` | Device key file exists, has 0600 permissions |
| `keys_loadable` | Keys parse as Ed25519 |
| `community_present` | Community manifest exists |
| `event_log_writable` | SQLite open and writable |
| `mdns_socket` | mDNS socket can bind |
| `udp_multicast` | UDP discovery socket can bind |
| `transport_port` | Bus port is free or owned by us |
| `at_least_one_capability` | Bus has registered ≥ 1 capability |
| `disk_space` | Free space ≥ 1 GB |
| `clock_sanity` | System clock within ±60s of HTTP-reachable anchor (only when internet up) |
| `llm_backend_reachable` | At least one LLM backend responds |
| `recent_error_rate` | Last 100 traces have < 20% error rate |

### 6.3 CLI integration

`hearthnet doctor` runs `run_all`, prints a coloured report, exits non-zero on any failure. See [M12](../modules/M12-cli.md).

---

## 7. Tests

- `test_logger_writes_json_lines` — assert each line parses as JSON with expected fields
- `test_metrics_endpoint_format` — Prometheus text format conforms
- `test_trace_context_propagation` — `attach`/`detach` round-trips across `asyncio.gather`
- `test_doctor_all_pass_on_default_config` — `run_all` returns all-OK on fresh init
- `test_doctor_keys_missing` — failure case for `keys_present`

---

## 8. References

- Config: [X04 §3](X04-config.md)
- Trace IDs propagate via [CONTRACT §5.1](../CAPABILITY_CONTRACT.md) `X-HearthNet-Request-Id`
- Bus emits trace events: [M03 §5.6](../modules/M03-bus.md)