Phase 3: prompt caching, history cap, essay coach endpoint

COST_OPTIMIZATION.md

@@ -0,0 +1,129 @@
+# How the Companion saves Claude API credit
+
+This document explains the cost model and the three optimizations now baked
+into the backend, so you understand what every chat costs and how to keep
+that cost low as the user base grows.
+
+## The mental model
+
+Anthropic charges per token, both for what Claude reads (input) and what
+Claude writes (output). A token is roughly ¾ of a word.
+
+Sonnet 4.6 pricing (as of writing):
+
+| Token type | Cost per million tokens |
+|------------|------------------------|
+| Standard input | $3.00 |
+| Cache write (first time) | $3.75 (25% surcharge) |
+| Cache read (subsequent calls) | $0.30 (10× cheaper) |
+| Output | $15.00 |
+
+There is **no session**. Every API call is stateless. The "conversation"
+feeling is built by your frontend resending the chat history every turn.
+That's where the bill grows.
+
+## Optimization 1 — Prompt caching on system + context
+
+**File:** `claude_engine.py`, the `system=[...]` block in `generate_response`.
+
+We send TWO cache blocks at the front of every chat request:
+
+1. **System prompt** — your instructions to Claude. Same on every call.
+2. **Retrieved ALU context** — the JSON chunks the retrieval engine returns.
+   Often overlaps between similar questions (a lot of students ask about
+   admissions, so they all retrieve the admissions JSON).
+
+Each block is marked `cache_control: ephemeral`, which tells Anthropic to
+store the processed prefix for 5 minutes. The next request that arrives with
+the SAME prefix pays $0.30/M instead of $3/M for that portion — **10× cheaper.**
+
+The break-even is one cache hit per 5-minute window. Anything more is pure
+savings.
+
+**Logged metrics:** Every Claude response now logs `cache_read` and
+`cache_write` token counts. Check the HF Space logs to see caching working:
+
+```
+[OK] Claude reply 412 chars | tokens in=58 out=312 cache_read=2483 cache_write=0
+```
+
+That line says: we paid full price for 58 input tokens (the question) and the
+output, but got 2,483 cached tokens at 10× discount instead of full price.
+Without caching that same call would have cost ~3× more on the input side.
+
+## Optimization 2 — History capped at last 8 turns
+
+**File:** `claude_engine.py`, `MAX_HISTORY_TURNS = 8` constant.
+
+Before this change, every chat sent the ENTIRE conversation history to
+Anthropic every turn. By message 10 that meant paying for 10 messages worth
+of input. By message 20 it was 20 messages.
+
+Now we cap at the most recent 8 turns (≈4 user + 4 AI). Older turns rarely
+influence the next answer, and the cap stops the linear growth of input cost
+on long sessions.
+
+To tune: change `MAX_HISTORY_TURNS` at the top of `claude_engine.py`. Higher
+= more context, more cost. Lower = cheaper but the chatbot forgets sooner.
+
+## Optimization 3 — Knowledge base trimmed
+
+**Done in the previous session.** The fabricated 180KB `comprehensive_knowledge.json`
+and other quarantined files were removed from the brain loader. Less context
+gets retrieved per query → less input cost per call.
+
+## What a single chat now costs (rough estimate)
+
+A 5-turn conversation about admissions:
+
+| Turn | Action | Cost notes |
+|------|--------|------------|
+| 1 | First question | Full price: writes cache (~$0.012) |
+| 2 | Follow-up | Cache hit on system + context (~$0.004) |
+| 3 | Another follow-up | Cache hit (~$0.004) |
+| 4 | Different topic | Partial cache hit, new context (~$0.008) |
+| 5 | Follow-up to #4 | Cache hit (~$0.004) |
+
+**Total: ~$0.032 for a 5-turn session.** With 1,000 active students having one
+5-turn session per week, monthly Anthropic cost ≈ **$130/month**.
+
+Without caching, the same usage would be 2–3× that.
+
+## What you can still do to save more
+
+Things I didn't implement automatically but you can flip on:
+
+1. **Lower `max_tokens` to 600** — set `DEFAULT_MAX_TOKENS = 600` in
+   `claude_engine.py`. Output is 5× more expensive than input. Most ALU
+   answers fit in 600 tokens. Risk: very long answers get truncated.
+
+2. **Add a Haiku router** — route obvious simple questions ("hi", "what's
+   ALU's website?") to `claude-haiku-4-5-20251001`, which is 3× cheaper than
+   Sonnet. I can add this when you're ready — it needs a classifier and
+   testing.
+
+3. **Front the chatbot with a rate limit** — cap each student to e.g. 30
+   chats per day. Easy in your `app.py` middleware. Stops one student
+   accidentally (or maliciously) burning through the quota.
+
+4. **Set a hard monthly budget in Anthropic Console** — go to your billing
+   page on console.anthropic.com and set a usage limit. If usage exceeds it,
+   the API returns errors instead of charging more. This is the safety net.
+
+## How to verify caching is actually working
+
+After deploying, send 2-3 chat messages in quick succession (within 5
+minutes). Check the HF Space logs. The first call should show:
+
+```
+cache_read=0 cache_write=2000+
+```
+
+The second call should show:
+
+```
+cache_read=2000+ cache_write=0
+```
+
+If you see `cache_read` staying at 0, the cache isn't matching — usually
+because the system prompt or context changed between calls. Open an issue.

app.py

@@ -201,6 +201,44 @@ async def chat_claude(request: ClaudeChatRequest):
     }
 
 
+# ---------------------------------------------------------------------------
+# Phase 2: Application essay coach
+# ---------------------------------------------------------------------------
+
+class EssayCoachRequest(BaseModel):
+    essay: str
+    prompt: Optional[str] = None  # the essay question being answered, optional
+
+
+@app.post("/api/coach/essay")
+async def coach_essay(request: EssayCoachRequest):
+    """
+    Give feedback on an ALU application essay draft. Stateless — does NOT
+    use chat history or RAG. Each request is independent.
+    """
+    if not claude_engine.enabled:
+        raise HTTPException(
+            status_code=503,
+            detail="Essay coach unavailable — Claude is not configured on the server.",
+        )
+    if len(request.essay.strip()) < 50:
+        raise HTTPException(
+            status_code=400,
+            detail="Essay is too short — please share at least a paragraph.",
+        )
+    if len(request.essay) > 12000:
+        raise HTTPException(
+            status_code=400,
+            detail="Essay is too long — please keep drafts under 12,000 characters.",
+        )
+
+    feedback = claude_engine.coach_essay(request.essay, prompt=request.prompt)
+    if not feedback:
+        raise HTTPException(status_code=502, detail="Coach could not produce feedback.")
+
+    return {"feedback": feedback, "engine": "claude", "model": claude_engine.model}
+
+
 # ---------------------------------------------------------------------------
 # Phase 2: Opportunities for students (Tavily-backed)
 # ---------------------------------------------------------------------------

claude_engine.py

@@ -16,6 +16,11 @@ DEFAULT_MODEL = "claude-sonnet-4-6"
 DEFAULT_MAX_TOKENS = 1024
 DEFAULT_TEMPERATURE = 0.4  # lower than chat — we want faithful retrieval
 
+# How many recent turns to keep when forwarding history. Older turns rarely
+# influence the next answer and grow input cost linearly. 8 = ~4 user turns
+# + ~4 assistant turns, which covers natural follow-up patterns.
+MAX_HISTORY_TURNS = 8
+
 SYSTEM_PROMPT = """You are the ALU Student Companion, an AI assistant for African Leadership University.
 
 # Your single most important rule
@@ -108,15 +113,35 @@ class ClaudeEngine:
         return "\n\n---\n\n".join(lines)
 
     @staticmethod
-    def _to_anthropic_messages(history: List[Dict[str, str]], user_message: str) -> List[Dict[str, str]]:
-        """Convert frontend history into Anthropic's messages format."""
+    def _to_anthropic_messages(
+        history: List[Dict[str, str]],
+        user_message: str,
+        max_turns: int = MAX_HISTORY_TURNS,
+    ) -> List[Dict[str, str]]:
+        """
+        Convert frontend history into Anthropic's messages format.
+
+        Trims to the last `max_turns` turns to cap input cost on long chats.
+        Anthropic requires the messages array to start with a `user` turn,
+        so we drop a leading `assistant` if the trim landed on one.
+        """
         msgs: List[Dict[str, str]] = []
         for entry in history or []:
             role = entry.get("role")
             content = entry.get("content", "")
             if role in ("user", "assistant") and content:
                 msgs.append({"role": role, "content": content})
-        # Ensure last message is the new user turn.
+
+        # Trim to the most recent N turns BEFORE we append the new user message,
+        # so the new message is always kept.
+        if len(msgs) > max_turns:
+            msgs = msgs[-max_turns:]
+
+        # Anthropic rejects an array that starts with 'assistant' — drop it.
+        while msgs and msgs[0]["role"] != "user":
+            msgs.pop(0)
+
+        # Ensure the last message is the new user turn we're answering.
         if not msgs or msgs[-1]["role"] != "user" or msgs[-1]["content"] != user_message:
             msgs.append({"role": "user", "content": user_message})
         return msgs
@@ -143,42 +168,150 @@ class ClaudeEngine:
 
         context_block = self._format_context(context_docs or [])
 
-        # We send the context as the FIRST user turn so prompt caching can
-        # reuse the system prompt across requests (Anthropic caches in 5-min
-        # windows). The retrieved context changes per query, but the system
-        # prompt stays stable.
-        rag_preamble = (
-            f"Use the following context from the ALU knowledge base to answer "
-            f"the question. If the context is insufficient, say so honestly.\n\n"
-            f"=== CONTEXT ===\n{context_block}\n=== END CONTEXT ===\n\n"
-            f"Question: {query}"
-        )
-
-        messages = self._to_anthropic_messages(history or [], rag_preamble)
+        # Anthropic prompt caching: structured as TWO cache blocks in `system`.
+        #
+        # Block 1 — the system prompt — almost never changes, so it stays in
+        # cache for the full 5-minute window and serves every conversation.
+        #
+        # Block 2 — the retrieved ALU context — changes per query, but when
+        # two queries retrieve overlapping content (very common: lots of
+        # questions about admissions all surface the same JSON entries),
+        # Anthropic returns a cache hit and we pay 10x less ($0.30/M vs $3/M).
+        # If the context is unique, we pay a one-time 25% cache-write surcharge
+        # ($3.75/M), so the break-even is one repeat within 5 minutes.
+        #
+        # The user message is now just the question — small, cheap, never
+        # cached (it's different every time).
+        system_blocks = [
+            {
+                "type": "text",
+                "text": SYSTEM_PROMPT,
+                "cache_control": {"type": "ephemeral"},
+            },
+            {
+                "type": "text",
+                "text": (
+                    "Use the following context from the ALU knowledge base "
+                    "to answer the student's question. If the context does "
+                    "not contain the answer, say so honestly and point to "
+                    "the official source.\n\n"
+                    f"=== CONTEXT ===\n{context_block}\n=== END CONTEXT ==="
+                ),
+                "cache_control": {"type": "ephemeral"},
+            },
+        ]
+
+        messages = self._to_anthropic_messages(history or [], query)
 
         try:
             response = self.client.messages.create(
                 model=self.model,
                 max_tokens=max_tokens,
                 temperature=temperature,
+                system=system_blocks,
+                messages=messages,
+            )
+            # response.content is a list of content blocks; we want the text.
+            text_parts = [
+                block.text for block in response.content if getattr(block, "type", None) == "text"
+            ]
+            answer = "\n".join(text_parts).strip()
+
+            # Log token usage so you can audit caching behaviour. The cache
+            # read/write counts confirm whether prompt caching is paying off.
+            usage = getattr(response, "usage", None)
+            if usage:
+                logger.info(
+                    "[OK] Claude reply %d chars | tokens in=%d out=%d "
+                    "cache_read=%d cache_write=%d",
+                    len(answer),
+                    getattr(usage, "input_tokens", 0),
+                    getattr(usage, "output_tokens", 0),
+                    getattr(usage, "cache_read_input_tokens", 0) or 0,
+                    getattr(usage, "cache_creation_input_tokens", 0) or 0,
+                )
+            else:
+                logger.info(f"[OK] Claude reply {len(answer)} chars")
+
+            return answer or None
+        except Exception as e:
+            logger.error(f"[FAIL] Claude API error: {e}")
+            return None
+
+    # ------------------------------------------------------------------
+    # Application essay coach
+    # ------------------------------------------------------------------
+
+    ESSAY_COACH_SYSTEM = """You are an application-essay coach for African Leadership University (ALU). A student is sharing a draft of their application essay and wants feedback.
+
+ALU's selection criteria emphasise:
+- A clear sense of personal mission — what problem does the applicant want to solve?
+- Evidence of leadership, initiative, or impact in their school or community
+- Authentic voice — ALU is wary of generic, overly polished essays
+- Pan-African or global perspective
+- Resilience and reflection — what did they learn from setbacks?
+
+Your feedback style:
+- Be specific. Quote short phrases from the draft when commenting on them.
+- Lead with what works before what doesn't.
+- Give 3-5 concrete suggestions, ranked by importance.
+- Be honest. Don't tell a weak essay it's strong — students need real feedback to improve.
+- End with a one-line summary of the essay's biggest strength and its biggest opportunity.
+- Never rewrite the essay for them. Coach, don't ghostwrite.
+
+Format your response as:
+
+## What's working
+[2-3 bullet points with quoted evidence]
+
+## Where to improve
+[3-5 bullet points, ranked by importance, with specific suggestions]
+
+## Bottom line
+[One sentence on the biggest strength + one sentence on the biggest opportunity]
+"""
+
+    def coach_essay(
+        self,
+        essay_text: str,
+        prompt: Optional[str] = None,
+        max_tokens: int = 1500,
+    ) -> Optional[str]:
+        """
+        Give application-essay feedback. Optional `prompt` is the essay
+        question the student is answering — providing it improves the feedback.
+        Returns markdown feedback or None on failure.
+        """
+        if not self.enabled or not self.client:
+            return None
+        if not essay_text or len(essay_text.strip()) < 50:
+            return None  # Caller handles the "essay too short" case.
+
+        user_message = "Here is the student's draft essay.\n\n"
+        if prompt:
+            user_message += f"Essay prompt they're answering:\n{prompt.strip()}\n\n"
+        user_message += f"=== ESSAY DRAFT ===\n{essay_text.strip()}\n=== END DRAFT ==="
+
+        try:
+            response = self.client.messages.create(
+                model=self.model,
+                max_tokens=max_tokens,
+                temperature=0.5,
                 system=[
                     {
                         "type": "text",
-                        "text": SYSTEM_PROMPT,
+                        "text": self.ESSAY_COACH_SYSTEM,
                         "cache_control": {"type": "ephemeral"},
                     }
                 ],
-                messages=messages,
+                messages=[{"role": "user", "content": user_message}],
             )
-            # response.content is a list of content blocks; we want the text.
             text_parts = [
                 block.text for block in response.content if getattr(block, "type", None) == "text"
             ]
-            answer = "\n".join(text_parts).strip()
-            logger.info(f"[OK] Claude response generated ({len(answer)} chars)")
-            return answer or None
+            return "\n".join(text_parts).strip() or None
         except Exception as e:
-            logger.error(f"[FAIL] Claude API error: {e}")
+            logger.error(f"[FAIL] Essay coach error: {e}")
             return None
 
     def get_status(self) -> Dict[str, Any]: