File size: 4,733 Bytes
64ab846 8546967 64ab846 8546967 909f428 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 64ab846 8546967 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 | # Predictive Irrigation Models
This repository contains end-to-end pipelines for predictive irrigation, combining:
- field sensor measurements (tensiometers and related sensors),
- irrigation logs,
- weather observations and generated forecasts,
- optional crop/soil metadata,
- optional satellite-derived vegetation indices,
- and two modeling tracks: XGBoost-based forecasting and AquaCrop-based simulation.
The implementation is orchestrated with Prefect flows in the `pipelines/` folder and configured through YAML files in `config/`.
## What Is In This Repository
- `main.py`: runs the full workflow in sequence.
- `pipelines/`: data preparation and model flows.
- `tools/`: weather, geospatial, authentication, and Copernicus helpers.
- `aquacrop/`: local AquaCrop engine implementation used by the AquaCrop pipeline.
- `config/`: runtime configuration (plus some private config files expected at runtime, see below).
- `notebooks/notebook_demo.ipynb`: interactive demo notebook.
## Pipeline Overview
The default execution path in `main.py` is:
1. `model_preparation_pipeline()`
2. `aquacrop_preparation_pipeline()`
3. `xgcast_preparation_pipeline()`
4. `aquacrop_pipeline()`
5. `xgcast_model_pipeline()`
Main data flow (paths are created/used by the pipelines):
- `data/03_primary/`: primary prepared inputs (sensor, weather, irrigation, locations, crop, soil, satellite).
- `data/04_model_input/`: merged model-ready tables (`full_table_<consortium>.parquet`).
- `data/05_xgcast_input/`: normalized training/validation/test artifacts for XGBoost.
- `data/05_aquacrop_input/<consortium>/<sensor>/`: weather/irrigation/settings for AquaCrop runs.
- `data/06_xgcast_output/`: trained XGBoost model files.
- `data/06_aquacrop_output/<consortium>/<sensor>/`: AquaCrop simulation outputs.
## Requirements
- Python 3.11+
- `uv` (recommended for environment/dependency management)
Project dependencies are declared in `pyproject.toml`.
## Setup
From the repository root:
```bash
uv sync
```
If you prefer running with the virtual environment directly:
```bash
uv run python -m main
```
## Configuration
Base configuration files committed in this repo:
- `config/params.yml`
- `config/aquacrop_params.yml`
- `config/xgcast_params.yml`
- `config/fieldsensor_irrigator_mapping_anonym.yaml`
- `config/request_scripts/*.js`
### Important: private/non-versioned configs
Some optional pipeline branches (especially satellite download/auth flows) expect private files that are intentionally gitignored, for example:
- `config/copernicus_oauth_config.json`
- `config/keycloak_config.json`
- `config/pre_anonym_params.yml`
- potentially consortium-specific field mapping files (for field-level satellite aggregation)
If those files are missing, full satellite acquisition workflows will not run.
## Running The Pipelines
### Full run (same sequence as `main.py`)
```bash
uv run python -m main
```
### Run individual flows
```bash
uv run python -m pipelines.model_preparation_pipeline
uv run python -m pipelines.aquacrop_preparation_pipeline
uv run python -m pipelines.aquacrop_pipeline
uv run python -m pipelines.xgcast_pipeline
```
## Expected Input Artifacts
At minimum, model preparation expects consortium-scoped files such as:
- `data/03_primary/field_sensor_data_<consortium>.parquet`
- `data/03_primary/irrigation_data_<consortium>.parquet`
- `data/03_primary/locations_ids_<consortium>.parquet`
- `data/03_primary/historical_weather_data_<consortium>.parquet`
- `data/03_primary/forecasted_weather_data_<consortium>.parquet`
Optional inputs are controlled through `data_availability` in `config/params.yml`:
- weather sensor data,
- crop type data,
- soil type data,
- remote sensing data.
## Models
### XGCast (XGBoost)
- training configuration in `config/xgcast_params.yml`.
- outputs model file `data/06_xgcast_output/xgcast_<consortium>.json`.
### AquaCrop
- uses local `aquacrop/` package implementation.
- simulation outputs include water flux, water storage, crop growth, and summary tables under `data/06_aquacrop_output/`.
## Notebook Demo
Open `notebooks/notebook_demo.ipynb` for an interactive walkthrough using consortium/sensor examples and model outputs.
## Known Caveats
- `data/` is gitignored; expected datasets are not included in this repository.
- several remote-sensing paths depend on private configuration and credentials.
## Contributing
Contributions are welcome. Open an issue or submit a pull request with:
- the problem statement,
- reproducible steps,
- and any data/config assumptions required to test the change. |