# How the Companion saves Claude API credit

This document explains the cost model and the three optimizations now baked
into the backend, so you understand what every chat costs and how to keep
that cost low as the user base grows.

## The mental model

Anthropic charges per token, both for what Claude reads (input) and what
Claude writes (output). A token is roughly ¾ of a word.

Sonnet 4.6 pricing (as of writing):

| Token type | Cost per million tokens |
|------------|------------------------|
| Standard input | $3.00 |
| Cache write (first time) | $3.75 (25% surcharge) |
| Cache read (subsequent calls) | $0.30 (10× cheaper) |
| Output | $15.00 |

There is **no session**. Every API call is stateless. The "conversation"
feeling is built by your frontend resending the chat history every turn.
That's where the bill grows.

## Optimization 1 — Prompt caching on system + context

**File:** `claude_engine.py`, the `system=[...]` block in `generate_response`.

We send TWO cache blocks at the front of every chat request:

1. **System prompt** — your instructions to Claude. Same on every call.
2. **Retrieved ALU context** — the JSON chunks the retrieval engine returns.
   Often overlaps between similar questions (a lot of students ask about
   admissions, so they all retrieve the admissions JSON).

Each block is marked `cache_control: ephemeral`, which tells Anthropic to
store the processed prefix for 5 minutes. The next request that arrives with
the SAME prefix pays $0.30/M instead of $3/M for that portion — **10× cheaper.**

The break-even is one cache hit per 5-minute window. Anything more is pure
savings.

**Logged metrics:** Every Claude response now logs `cache_read` and
`cache_write` token counts. Check the HF Space logs to see caching working:

```
[OK] Claude reply 412 chars | tokens in=58 out=312 cache_read=2483 cache_write=0
```

That line says: we paid full price for 58 input tokens (the question) and the
output, but got 2,483 cached tokens at 10× discount instead of full price.
Without caching that same call would have cost ~3× more on the input side.

## Optimization 2 — History capped at last 8 turns

**File:** `claude_engine.py`, `MAX_HISTORY_TURNS = 8` constant.

Before this change, every chat sent the ENTIRE conversation history to
Anthropic every turn. By message 10 that meant paying for 10 messages worth
of input. By message 20 it was 20 messages.

Now we cap at the most recent 8 turns (≈4 user + 4 AI). Older turns rarely
influence the next answer, and the cap stops the linear growth of input cost
on long sessions.

To tune: change `MAX_HISTORY_TURNS` at the top of `claude_engine.py`. Higher
= more context, more cost. Lower = cheaper but the chatbot forgets sooner.

## Optimization 3 — Knowledge base trimmed

**Done in the previous session.** The fabricated 180KB `comprehensive_knowledge.json`
and other quarantined files were removed from the brain loader. Less context
gets retrieved per query → less input cost per call.

## What a single chat now costs (rough estimate)

A 5-turn conversation about admissions:

| Turn | Action | Cost notes |
|------|--------|------------|
| 1 | First question | Full price: writes cache (~$0.012) |
| 2 | Follow-up | Cache hit on system + context (~$0.004) |
| 3 | Another follow-up | Cache hit (~$0.004) |
| 4 | Different topic | Partial cache hit, new context (~$0.008) |
| 5 | Follow-up to #4 | Cache hit (~$0.004) |

**Total: ~$0.032 for a 5-turn session.** With 1,000 active students having one
5-turn session per week, monthly Anthropic cost ≈ **$130/month**.

Without caching, the same usage would be 2–3× that.

## What you can still do to save more

Things I didn't implement automatically but you can flip on:

1. **Lower `max_tokens` to 600** — set `DEFAULT_MAX_TOKENS = 600` in
   `claude_engine.py`. Output is 5× more expensive than input. Most ALU
   answers fit in 600 tokens. Risk: very long answers get truncated.

2. **Add a Haiku router** — route obvious simple questions ("hi", "what's
   ALU's website?") to `claude-haiku-4-5-20251001`, which is 3× cheaper than
   Sonnet. I can add this when you're ready — it needs a classifier and
   testing.

3. **Front the chatbot with a rate limit** — cap each student to e.g. 30
   chats per day. Easy in your `app.py` middleware. Stops one student
   accidentally (or maliciously) burning through the quota.

4. **Set a hard monthly budget in Anthropic Console** — go to your billing
   page on console.anthropic.com and set a usage limit. If usage exceeds it,
   the API returns errors instead of charging more. This is the safety net.

## How to verify caching is actually working

After deploying, send 2-3 chat messages in quick succession (within 5
minutes). Check the HF Space logs. The first call should show:

```
cache_read=0 cache_write=2000+
```

The second call should show:

```
cache_read=2000+ cache_write=0
```

If you see `cache_read` staying at 0, the cache isn't matching — usually
because the system prompt or context changed between calls. Open an issue.