# Predictive Irrigation Models

This repository contains end-to-end pipelines for predictive irrigation, combining:

- field sensor measurements (tensiometers and related sensors),
- irrigation logs,
- weather observations and generated forecasts,
- optional crop/soil metadata,
- optional satellite-derived vegetation indices,
- and two modeling tracks: XGBoost-based forecasting and AquaCrop-based simulation. 

The implementation is orchestrated with Prefect flows in the `pipelines/` folder and configured through YAML files in `config/`.

## What Is In This Repository

- `main.py`: runs the full workflow in sequence.
- `pipelines/`: data preparation and model flows.
- `tools/`: weather, geospatial, authentication, and Copernicus helpers.
- `aquacrop/`: local AquaCrop engine implementation used by the AquaCrop pipeline.
- `config/`: runtime configuration (plus some private config files expected at runtime, see below).
- `notebooks/notebook_demo.ipynb`: interactive demo notebook.

## Pipeline Overview

The default execution path in `main.py` is:

1. `model_preparation_pipeline()`
2. `aquacrop_preparation_pipeline()`
3. `xgcast_preparation_pipeline()`
4. `aquacrop_pipeline()`
5. `xgcast_model_pipeline()`

Main data flow (paths are created/used by the pipelines):

- `data/03_primary/`: primary prepared inputs (sensor, weather, irrigation, locations, crop, soil, satellite).
- `data/04_model_input/`: merged model-ready tables (`full_table_<consortium>.parquet`).
- `data/05_xgcast_input/`: normalized training/validation/test artifacts for XGBoost.
- `data/05_aquacrop_input/<consortium>/<sensor>/`: weather/irrigation/settings for AquaCrop runs.
- `data/06_xgcast_output/`: trained XGBoost model files.
- `data/06_aquacrop_output/<consortium>/<sensor>/`: AquaCrop simulation outputs.

## Requirements

- Python 3.11+
- `uv` (recommended for environment/dependency management)

Project dependencies are declared in `pyproject.toml`.

## Setup

From the repository root:

```bash
uv sync
```

If you prefer running with the virtual environment directly:

```bash
uv run python -m main
```

## Configuration

Base configuration files committed in this repo:

- `config/params.yml`
- `config/aquacrop_params.yml`
- `config/xgcast_params.yml`
- `config/fieldsensor_irrigator_mapping_anonym.yaml`
- `config/request_scripts/*.js`

### Important: private/non-versioned configs

Some optional pipeline branches (especially satellite download/auth flows) expect private files that are intentionally gitignored, for example:

- `config/copernicus_oauth_config.json`
- `config/keycloak_config.json`
- `config/pre_anonym_params.yml`
- potentially consortium-specific field mapping files (for field-level satellite aggregation)

If those files are missing, full satellite acquisition workflows will not run.

## Running The Pipelines

### Full run (same sequence as `main.py`)

```bash
uv run python -m main
```

### Run individual flows

```bash
uv run python -m pipelines.model_preparation_pipeline
uv run python -m pipelines.aquacrop_preparation_pipeline
uv run python -m pipelines.aquacrop_pipeline
uv run python -m pipelines.xgcast_pipeline
```

## Expected Input Artifacts

At minimum, model preparation expects consortium-scoped files such as:

- `data/03_primary/field_sensor_data_<consortium>.parquet`
- `data/03_primary/irrigation_data_<consortium>.parquet`
- `data/03_primary/locations_ids_<consortium>.parquet`
- `data/03_primary/historical_weather_data_<consortium>.parquet`
- `data/03_primary/forecasted_weather_data_<consortium>.parquet`

Optional inputs are controlled through `data_availability` in `config/params.yml`:

- weather sensor data,
- crop type data,
- soil type data,
- remote sensing data.

## Models

### XGCast (XGBoost)

- training configuration in `config/xgcast_params.yml`.
- outputs model file `data/06_xgcast_output/xgcast_<consortium>.json`.

### AquaCrop

- uses local `aquacrop/` package implementation.
- simulation outputs include water flux, water storage, crop growth, and summary tables under `data/06_aquacrop_output/`.

## Notebook Demo

Open `notebooks/notebook_demo.ipynb` for an interactive walkthrough using consortium/sensor examples and model outputs.

## Known Caveats

- `data/` is gitignored; expected datasets are not included in this repository.
- several remote-sensing paths depend on private configuration and credentials.

## Contributing

Contributions are welcome. Open an issue or submit a pull request with:

- the problem statement,
- reproducible steps,
- and any data/config assumptions required to test the change.