# Guarden — Technical Documentation

This document describes the architecture, data model, and machine-learning
components behind **Guarden**, a Gradio application that helps
users identify plants, track a virtual garden, and receive weather-aware
watering, care recommendations, health checks...

---

## 1. High-level architecture

Guarden is a **single-process Gradio app** (`app.py`, ~950 lines) backed by a
small set of pure-Python modules. There is no database server: each user gets
a private, file-based "garden" stored on disk, and three external AI/ML
models are called on demand via Hugging Face.

```
┌─────────────────────────────────────────────────────────────────────┐
│                          Browser (Gradio UI)                         │
│  Location gate → Garden board (drag & drop) → Sidebar (watering /    │
│  forecast / assistant) → Add-plant drawer                            │
└───────────────┬────────────────────────────────────────────────────-┘
                 │ Gradio Blocks events (click / change / .then chains)
┌───────────────▼────────────────────────────────────────────────────-┐
│                              app.py                                   │
│  • Per-user routing (BrowserState user_id → user_data/<uuid>/)       │
│  • Garden CRUD (load/save garden.json, photos, background, links)    │
│  • Board rendering (HTML + SVG overlay + JS drag/drop bridge)         │
│  • Orchestrates calls into modules/* and external APIs                │
└───┬─────────────┬───────────────┬───────────────┬───────────────────┘
    │             │               │               │
    ▼             ▼               ▼               ▼
modules/      modules/        modules/         modules/
classifier.py recommender.py  watering.py       advisor.py
(SigLIP ML     plant.py        weather_utils.py (HF Inference:
 image          (CSV lookup)   (Open-Meteo)       chat LLM +
 classifier)                                      vision LLM)
                                   │
                                   ▼
                          utils/geo.py (Open-Meteo
                                geocoding)
```

### Per-user data layout

```
user_data/
└── <uuid>/                      # one folder per browser, via gr.BrowserState
    ├── garden.json              # list of plant dicts (see §3)
    ├── background.jpg           # optional custom board background
    └── plant_photos/
        └── <plant_id>.jpg       # uploaded photo for each plant
```

A `user_id` (UUID4) is generated on first visit and persisted in the
browser's local storage via `gr.BrowserState`, so the same browser always
maps back to the same `user_data/<uuid>/` folder — no login/auth required.

---

## 2. Tech stack

| Layer | Technology |
|---|---|
| UI / app framework | **Gradio 6.18** (Blocks API), custom CSS theme (`static/style.css`), small vanilla-JS bridge for the drag-and-drop board (`BOARD_JS` in `app.py`) |
| Plant genus classification | Fine-tuned **SigLIP** vision transformer (`transformers`, local inference, CPU/GPU) |
| Gardening chat advisor | LLM via **Hugging Face Inference Providers** (`huggingface_hub.InferenceClient`) |
| Photo health diagnostic | Vision-language model via **Hugging Face Inference Providers** (multimodal chat) |
| Care metadata | CSV lookup table (`data/growth_csv/growth_ds.csv`), `pandas` |
| Weather & geocoding | **Open-Meteo** REST APIs (forecast, archive, geocoding) — no API key needed |
| Persistence | Flat files: `garden.json` (JSON) + JPEG photos, per user, on local disk |
| Sprites | Procedurally generated pixel-art PNGs (`modules/pixel_art.py`, pure PIL, no ML) |

Runtime dependencies are pinned in `requirements.txt`. Training-only
dependencies (`datasets`, `accelerate`, `torchvision`) live alongside the
inference deps because the classifier's training script ships in the same
repo (see §4.1).

---

## 3. Data model

Each plant in `garden.json` is a dict with the following fields (collected
from how `app.py` reads/writes them):

```jsonc
{
  "id": "20260610_211240_094893",        // timestamp-based unique id
  "nickname": "Living Room Ficus",       // user-given name
  "photo": "user_data/<uid>/plant_photos/<id>.jpg",
  "genus": "Ficus",                      // predicted by the classifier
  "confidence": 92.4,                    // classifier confidence, %
  "added": "2026-06-10",
  "last_watered": "2026-06-12",          // ISO date or null
  "watering_history": ["2026-06-01", "2026-06-12"], // append-only log
  "rained": false,                       // true if last_watered was inferred from rain
  "watering_frequency_days": "Regular watering",   // raw CSV text
  "sunlight": "full sunlight",
  "soil": "sandy",
  "fertilization_type": "Balanced",
  "notes": "Ficus needs full sunlight. It thrives in sandy soil. ...",
  "position": { "x": 30.0, "y": 40.0 }, // % position on the garden board
  "neighbors": ["<other plant id>"],     // hand-drawn "neighbor" links
  "health": "Healthy — leaves look ..."  // last VLM health diagnosis, if any
}
```

This structure is the single source of truth: the board, the detail card,
the watering table, the advisor and the health diagnostic all read/write
this same list of dicts via `load_garden(user_id)` / `save_garden(...)`.

---

## 4. Machine-learning components

Guarden uses **three** distinct AI models, each chosen for a different job:
a small fine-tuned **vision classifier** for genus recognition (fast, local,
deterministic), and two **Hugging Face Inference**-hosted generative models
for natural-language and vision-language reasoning (advisor + health check).

### 4.1 Plant genus classifier (`modules/classifier.py`)

**Task**: given a photo of a plant, predict its **botanical genus** (e.g.
`Ficus`, `Aloe`, `Begonia`) out of **289 genus classes**.

**Model**: a fine-tuned [`google/siglip-base-patch16-224`](https://huggingface.co/google/siglip-base-patch16-224)
(SigLIP — a CLIP-style vision transformer, ViT-B/16, 224×224 input, 768-d
hidden size) with a `SiglipForImageClassification` head (289-way softmax).
The fine-tuned weights are pushed to a private HF Hub repo
(`Crocolil/HackatonSmall-storage`) and exported as a clean
`config.json` / `model.safetensors` / `preprocessor_config.json` bundle
(~372 MB) under `training/clean_export/`.

**Training pipeline** (`training/train_classifier.py`):
- Loads a `datasets.DatasetDict` (train/test split) of labelled plant
  photos, with one `ClassLabel` per genus (`data/hf_plant_dataset/`).
- Builds `id2label` / `label2id` from the dataset's `ClassLabel` feature.
- Data augmentation (train split): `RandomResizedCrop(scale=0.8–1.0)`,
  `RandomHorizontalFlip`, `ColorJitter(brightness/contrast/saturation=0.1)`,
  then `ToTensor` + SigLIP's own image-mean/std normalization.
- Eval split: deterministic `Resize` + `CenterCrop` + normalize.
- Fine-tuned end-to-end with 🤗 `Trainer` / `TrainingArguments`:
  - `num_train_epochs=3`, `per_device_*_batch_size=32`, `lr=5e-5`, `seed=42`
  - `bf16=True` when CUDA is available
  - `eval_strategy="epoch"`, `save_strategy="epoch"`,
    `load_best_model_at_end=True`, `metric_for_best_model="accuracy"`
  - Metrics: **top-1 accuracy** and **top-5 accuracy**
    (`compute_metrics` compares `argmax` / top-5 logits vs. labels).
  - Optional `--push-to-hub` to publish the checkpoint to a private repo.

**Inference** (`modules/classifier.py`):
- `CLASSIFIER_MODEL_ID` env var points to the Hub repo of the fine-tuned
  model (loaded lazily, cached as module-level globals).
- `classify_plant(image)`:
  1. `AutoImageProcessor` resizes/normalizes the uploaded `PIL.Image`.
  2. `AutoModelForImageClassification` runs a forward pass (`torch.no_grad()`).
  3. Softmax over the 289 logits → `(genus_name, confidence)`.
- Called from `app.py`'s `add_plants_to_garden()` for every uploaded photo;
  the predicted genus drives everything downstream (care metadata, sprite
  archetype, advisor context).

### 4.2 Care recommendation engine (`modules/plant.py`, `modules/recommender.py`)

Not a learned model — a **deterministic lookup + template** layer that turns
the classifier's genus output into actionable care info:

- `Plant(genus)` looks up `data/growth_csv/growth_ds.csv` (296 genus →
  care-profile rows, derived from a public plants-growth dataset) for
  `Watering`, `Sunlight`, `Soil`, `Fertilization Type`.
- If the genus isn't in the CSV (e.g. a class the classifier knows but the
  growth table doesn't cover), `get_plant_info()` falls back to generic
  defaults (`"Water when soil is dry"`, `"indirect sunlight"`,
  `"well-drained"`, `"No"` fertilizer).
- `generate_care_notes()` assembles a short natural-language note from these
  fields via string templates (no model call) — shown on the plant detail
  card under *"Notes"*.

### 4.3 Watering scheduler (`modules/watering.py` + `modules/weather_utils.py`)

Also rule-based, but **weather-aware**:

- `_parse_watering_frequency()` maps the CSV's free-text watering
  instructions (e.g. *"Keep soil consistently moist"*, *"Water weekly"*,
  *"every 10 days"*) to an integer interval in days, via an exact-match
  table plus regex fallbacks (`DEFAULT_INTERVAL = 4` days if nothing
  matches).
- `should_water(plant, last_watered, date, lat, lon)` returns `True` if:
  - `next_watering_date = last_watered + frequency_days` has passed, **and**
  - `did_or_will_rain(date, lat, lon, threshold=50%)` is `False` — i.e. it
    didn't rain in the past (for historical dates) and isn't forecast to
    rain ≥50% (for today/future), so the app doesn't tell you to water a
    plant that nature is about to water for you.
- `load_garden()` also **back-fills `last_watered`** from `last_rained_date()`
  on every load: if it rained more recently than the recorded watering date,
  the plant is considered watered by rain (`rained: true`), avoiding
  over-watering recommendations after a period of inactivity.
- The sidebar's "Watering today" table is produced by
  `get_watering_recommendations()`, which runs `should_water()` for every
  plant against the live 7-day forecast.

### 4.4 AI gardening advisor — chat (`modules/advisor.py::ask_about_plant`)

**Task**: free-form Q&A about a *specific* plant ("Why are the leaves turning
yellow?", "Can I plant this next to my tomatoes?").

- Model: `ADVISOR_MODEL_ID` (default `Qwen/Qwen2.5-Coder-3B-Instruct`) served
  via **Hugging Face Inference Providers** (`provider="nscale"` by default),
  through `huggingface_hub.InferenceClient.chat_completion`.
- **Grounding / prompt construction** (`_build_system_prompt`): the system
  prompt is dynamically built from the plant's care profile (sunlight, soil,
  watering frequency, fertilization) **and** its live watering status
  (computed via `_watering_status()` from `last_watered`), so the model knows
  whether the plant is overdue or recently watered before answering. If the
  user has drawn "neighbor" links on the board, the linked plants'
  name/genus are injected too, so the model can reason about
  companion-planting effects (shared pests, competition for light/water,
  beneficial pairings).
- The model is instructed to answer in **2–4 sentences**, in the same
  language as the question, and to never recommend toxic/dangerous
  substances.
- On any `InferenceClient` error, the function logs
  `[advisor] HF Inference error: ...` and returns a friendly fallback
  message instead of crashing the UI.
- Wired in `app.py` (`ask_plant_advisor`) to the "🤖 Ask the assistant"
  button in the sidebar's *Plant assistant* panel, which only appears once a
  plant is selected on the board.

### 4.5 Photo-based health diagnostic — vision-language (`modules/advisor.py::diagnose_plant_health`)

**Task**: given a *new* photo of the selected plant, assess its health
(leaves, stems, soil) and store the verdict on the plant record.

- Model: the same `ADVISOR_MODEL_ID` / `ADVISOR_PROVIDER` client as the chat
  advisor (§4.4), again via `InferenceClient.chat_completion` — but this time
  with a **multimodal** message: the uploaded `PIL.Image` is re-encoded as JPEG,
  base64-encoded, and sent as an OpenAI-style content array
  (`{"type": "text", ...}` + `{"type": "image_url", "image_url": {"url":
  "data:image/jpeg;base64,..."}}`).
- The prompt asks the model to start its reply with exactly one status word
  — **`Healthy`**, **`Needs attention`**, or **`Sick`** — followed by a 1–3
  sentence explanation and a suggested action.
- `app.py::diagnose_selected_plant_health` persists the raw model response
  into `plant["health"]`, which is then surfaced on the plant detail card
  (**🩺 Health:** ...) every time the garden is reloaded — so the diagnosis
  survives page refreshes and is visible alongside the watering history.
- Same defensive error handling as §4.4 (`[advisor] HF Inference
  health-check error: ...` + fallback message).

### 4.6 Procedural pixel-art sprites (`modules/pixel_art.py`) — not ML, but genus-aware

Worth a short mention because it *feels* like generative output but is fully
deterministic: each genus is mapped to one of 6 hand-authored 16×16 "plant
archetype" sprites (cactus, succulent, fern, flower, palm, trailing) and one
of 4 pot styles, based on the genus's `Growth` / `Soil` / `Sunlight` values
from the same `growth_ds.csv` (e.g. sandy soil + full sun + slow growth →
cactus in a terracotta pot). Genera missing from the CSV get a **stable
hash-based** archetype/pot assignment so the same unknown genus always
renders the same sprite. Sprites are rendered once with PIL nearest-neighbour
upscaling and cached to `static/sprites/<genus>.png`.

---

## 5. External APIs

All weather/geocoding calls go to **Open-Meteo** (free, no API key):

| Function | Endpoint | Used for |
|---|---|---|
| `utils.geo.city_to_coordinates` | `geocoding-api.open-meteo.com/v1/search` | Turn the user's city into `(lat, lon)` at the location gate |
| `modules.weather_utils.weather_values` | `api.open-meteo.com/v1/forecast` (16-day daily) | 7-day forecast table (conditions, temp, rain %, wind) |
| `modules.weather_utils.did_or_will_rain` | forecast (future) or `archive-api.open-meteo.com` (past) | Decide whether a plant should be watered today / was watered by rain |
| `modules.weather_utils.last_rained_date` | `archive-api.open-meteo.com/v1/archive` (15-day lookback) | Back-fill `last_watered` on garden load |

`weather_comment()` maps Open-Meteo's numeric WMO weather codes to short
emoji + text labels (e.g. `80` → "🌦️ Slight rain showers") shown in the
forecast table.

---

## 6. UI / front-end notes

- Single `gr.Blocks` app, themed with `gr.themes.Soft()` plus a large custom
  stylesheet (`static/style.css`) that overrides Gradio's CSS variables for
  a green "Guarden" theme (custom button gradients, card radii, etc.).
- **Garden board**: plants are rendered as absolutely-positioned `<div>`
  sprites inside `get_garden_board_html()`. A small injected `<script>`
  (`BOARD_JS`) uses pointer events to support:
  - **Drag & drop** → updates `position: {x%, y%}` (hidden `gr.Number` +
    sync button bridge the JS → Python boundary).
  - **Click to select** → opens the detail card + action row + assistant
    panel for that plant.
  - **"🔗 Link Neighbours" mode** → click two sprites to toggle a
    `neighbors` link, drawn as a dashed SVG line between them (re-rendered
    on every board update, so links follow plants when dragged).
- **Sidebar**: watering recommendations table, 7-day forecast table (custom
  CSS turns the Gradio dataframe into card-style rows with a styled header
  row), and the *Plant assistant* panel (chat + health diagnostic), which
  only becomes visible once a plant is selected.
- **Per-user custom background**: an uploaded image is saved as
  `user_data/<uid>/background.jpg` and applied as the board's
  `background-image` via inline CSS.

---

## 7. Deployment

The app is shipped as a **Hugging Face Space** (`sdk: gradio`,
`sdk_version: 6.18.0`, entry point `app.py`, see the `README.md` front
matter). Configuration is entirely via environment variables, with sane
defaults baked in so the app runs locally without any secrets:

| Env var | Default | Purpose |
|---|---|---|
| `WEATHER_CITY` | `"Marseille"` | Initial forecast location before the user sets one |
| `CLASSIFIER_MODEL_ID` | `"your-username/plant-genus-classifier"` | HF Hub repo of the fine-tuned SigLIP genus classifier |
| `ADVISOR_MODEL_ID` / `ADVISOR_PROVIDER` | `Qwen/Qwen2.5-Coder-3B-Instruct` / `nscale` | Chat advisor + health-diagnostic model and HF Inference provider (shared) |
| `HF_TOKEN` | — | Hugging Face token for Inference Providers (advisor + health check) |

`app.launch(allowed_paths=[...])` whitelists `user_data/`, `static/` and
`plant_photos/` so per-user photos, sprites and backgrounds can be served
back to the browser via Gradio's `/gradio_api/file=` route.

---