# A1 — "Third Eye" | Claude Code Build Prompt
## Build Small Hackathon 2026 | Backyard AI Track

---

## Mission
Build a fully voice-driven accessibility app that lets a blind or low-vision person point their webcam at anything — a menu, medicine label, door sign, street scene — speak a question, and hear the answer back in their own language. Zero typing required at any point in the pipeline.

---

## Models (ONLY sponsor models — no exceptions)

| Role | Model ID | Params | Sponsor |
|---|---|---|---|
| Vision + OCR (PRIMARY) | `openbmb/MiniCPM-V-2_0` | 2.8B | OpenBMB |
| Vision + OCR (FALLBACK) | `openbmb/MiniCPM-V-4_5` | 8B | OpenBMB |
| Speech-to-text | `CohereLabs/cohere-transcribe-03-2026` | 2B | Cohere |
| Text-to-speech | `openbmb/VoxCPM2` | 2B | OpenBMB |

**Tiny Titan strategy:** Use `MiniCPM-V-2_0` (2.8B) as primary. If scene description quality is unacceptable after testing, fall back to `MiniCPM-V-4_5` (8B) — note this in README as "fallback used, Tiny Titan badge forfeited." Do NOT silently swap models.

Total param budget: 6.8B (primary) or 12B (fallback) — both well under 32B cap.

---

## Tech stack
- **Gradio 5.x** with `gr.Server` for custom UI (Off-Brand badge)
- **Modal** for serverless GPU (A10G) running vision + TTS
- **Cohere Transcribe** via `transformers` pipeline (STT runs on Modal too)
- **Python 3.11**
- **No other cloud APIs** — fully off-grid eligible if Modal is not counting

---

## Directory structure
```
third-eye/
├── app.py                  # Gradio entry point
├── modal_backend.py        # Modal: vision inference + TTS
├── cohere_stt.py           # Cohere Transcribe wrapper
├── utils.py                # image/audio byte helpers
├── requirements.txt
├── .env.example            # MODAL_ENDPOINT_URL, HF_TOKEN
├── README.md               # HF Space frontmatter + project description
└── assets/
    ├── custom.css          # High-contrast, large-text, WCAG AA
    ├── sample_menu.jpg     # Example: restaurant menu
    ├── sample_label.jpg    # Example: medicine label
    └── sample_sign.jpg     # Example: street sign
```

---

## README.md — EXACT frontmatter (copy verbatim)
```yaml
---
title: Third Eye
emoji: 👁️
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: "5.0"
app_file: app.py
pinned: false
tags:
  - hackathon
  - build-small
  - backyard-ai
  - accessibility
  - blind
  - openbmb/MiniCPM-V-2_0
  - openbmb/VoxCPM2
  - cohere/cohere-transcribe-03-2026
  - tiny-titan
  - off-brand
---
```

---

## Implementation steps (build in this exact order)

### Step 1 — Modal backend: vision inference
```python
# modal_backend.py
import modal, io
from PIL import Image

app = modal.App("third-eye-backend")

vision_image = modal.Image.debian_slim().pip_install(
    "transformers>=4.40", "torch", "pillow", "accelerate",
    "sentencepiece", "timm"
)

@app.function(gpu="A10G", image=vision_image, timeout=120)
def describe_scene(image_bytes: bytes, question: str, lang: str = "en") -> str:
    from transformers import AutoTokenizer, AutoModel
    import torch
    from PIL import Image
    import io

    model_id = "openbmb/MiniCPM-V-2_0"  # PRIMARY: 2.8B
    # FALLBACK: model_id = "openbmb/MiniCPM-V-4_5"  # 8B, better quality

    model = AutoModel.from_pretrained(model_id, trust_remote_code=True,
                                       torch_dtype=torch.float16).cuda().eval()
    tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

    image = Image.open(io.BytesIO(image_bytes)).convert("RGB")
    prompt = question if question.strip() else "Describe everything you see in detail."

    msgs = [{"role": "user", "content": [image, prompt]}]
    answer = model.chat(image=None, msgs=msgs, tokenizer=tokenizer)
    return answer
```

### Step 2 — Modal backend: TTS
```python
@app.function(gpu="A10G", image=vision_image, timeout=60)
def speak(text: str) -> bytes:
    # Load VoxCPM2 and synthesize speech
    # Return raw WAV bytes
    from transformers import AutoModel
    import torch, io, soundfile as sf

    model = AutoModel.from_pretrained("openbmb/VoxCPM2", trust_remote_code=True)
    # Follow VoxCPM2 inference API from its model card
    audio_array, sr = model.synthesize(text)
    buf = io.BytesIO()
    sf.write(buf, audio_array, sr, format="WAV")
    return buf.getvalue()
```

### Step 3 — Cohere STT wrapper (cohere_stt.py)
```python
def transcribe(audio_path: str) -> str:
    from transformers import pipeline
    pipe = pipeline(
        "automatic-speech-recognition",
        model="CohereLabs/cohere-transcribe-03-2026"
    )
    result = pipe(audio_path)
    return result["text"].strip()
```

### Step 4 — Gradio app (app.py)
Build three modes as gr.Tabs:
- **Tab 1 "Describe"**: webcam image + "What do you see?" auto-prompt → spoken description
- **Tab 2 "Ask"**: webcam image + mic recording → transcribe → ask → spoken answer
- **Tab 3 "Read Text"**: webcam image + "Read all text in this image aloud" fixed prompt → spoken OCR

Main pipeline function:
```python
def run_pipeline(image, audio_path, mode):
    import tempfile, os

    if image is None:
        return None, "No image captured.", None

    img_bytes = image_to_bytes(image)  # utils.py

    if mode == "Ask" and audio_path:
        question = transcribe(audio_path)  # cohere_stt.py
    elif mode == "Read Text":
        question = "Read all text visible in this image, word by word."
    else:
        question = "Describe everything in this image in detail."

    answer = describe_scene.remote(img_bytes, question)
    audio_bytes = speak.remote(answer)
    audio_path_out = bytes_to_wav(audio_bytes)  # utils.py

    return audio_path_out, answer, question
```

### Step 5 — Custom UI (assets/custom.css + gr.Server)
```css
/* High contrast, large font — WCAG AA */
body { background: #0a0a0a; color: #f5f5f5; font-size: 20px; }
button.primary { background: #4f6ef7; font-size: 22px; padding: 16px 32px; }
.output-text { font-size: 24px; line-height: 1.8; }
audio { width: 100%; margin-top: 12px; }
```

Load in app.py:
```python
with open("assets/custom.css") as f:
    css = f.read()
demo = gr.Blocks(css=css)
```

---

## TODO 1 — Bounding-box "zoom and read" mode
After the core pipeline works, add a fourth tab using `gr.ImageEditor`. The user draws a rectangle over a specific region of the image (a single line of text, a price, a warning label). The cropped region is passed to MiniCPM-V with the prompt: "Read the text in this image exactly as written." This lets the user isolate dense text without the model guessing which part to focus on. Implement after TODOs 1-4 above are done.

## TODO 2 — Language selector for non-English users
Add a `gr.Dropdown` for target language (English, Hindi, German, Tamil, Telugu, Kannada). After getting the English answer from MiniCPM-V, pass it to VoxCPM2 with the selected language. VoxCPM2 supports multilingual synthesis natively. This turns Third Eye from an English-only tool into a mother-tongue accessibility tool — directly targeting Cohere's multilingual use-case.

---

## Sponsor + badge alignment

| Award | Why you qualify |
|---|---|
| Backyard AI podium | Accessibility gap — zero similar entries; real user = real impact |
| OpenBMB award | MiniCPM-V-2_0 (primary) + VoxCPM2 (TTS) — two OpenBMB models |
| Cohere award | Transcribe (STT pipeline) |
| Tiny Titan ($1,500) | MiniCPM-V-2_0 = 2.8B ≤ 4B — qualifies |
| Best Demo ($1,000) | Blind user hearing a menu read aloud = emotionally undeniable 30-sec video |
| Off-Brand ($1,500) | Custom CSS + high-contrast design |
| Field Notes (badge) | Write blog post on HF — what worked, what VLM quality was like at 2.8B |

---

## Non-negotiables
- Cold-start: show `gr.Progress` with "Loading AI models (first run: ~30s)..."
- Mic failure: fallback `gr.Textbox` for typed questions — never block the user
- VoxCPM2 failure: fallback to large-font text output
- Every exception: `gr.Warning("...")` — never a bare Python traceback shown to user
- Test the FULL pipeline (image → STT → VLM → TTS → audio playback) before submitting
- Include at least 3 example images in the Space so judges can test without a webcam