Upload README.md

README.md

@@ -1,77 +1,145 @@
 # Predictive Irrigation Models
 
-This repository contains data preprocessing and analytics pipelines for the distribution of Predictive Irrigation Models, integrating data from field sensors, weather, soil, crop, and remote sensing sources.
+This repository contains end-to-end pipelines for predictive irrigation, combining:
 
-## Project Structure
+- field sensor measurements (tensiometers and related sensors),
+- irrigation logs,
+- weather observations and generated forecasts,
+- optional crop/soil metadata,
+- optional satellite-derived vegetation indices,
+- and two modeling tracks: XGBoost-based forecasting and AquaCrop-based simulation.
+
+The implementation is orchestrated with Prefect flows in the `pipelines/` folder and configured through YAML files in `config/`.
+
+## What Is In This Repository
+
+- `main.py`: runs the full workflow in sequence.
+- `pipelines/`: data preparation and model flows.
+- `tools/`: weather, geospatial, authentication, and Copernicus helpers.
+- `aquacrop/`: local AquaCrop engine implementation used by the AquaCrop pipeline.
+- `config/`: runtime configuration (plus some private config files expected at runtime, see below).
+- `notebooks/notebook_demo.ipynb`: interactive demo notebook.
+
+## Pipeline Overview
+
+The default execution path in `main.py` is:
+
+1. `model_preparation_pipeline()`
+2. `aquacrop_preparation_pipeline()`
+3. `xgcast_preparation_pipeline()`
+4. `aquacrop_pipeline()`
+5. `xgcast_model_pipeline()`
+
+Main data flow (paths are created/used by the pipelines):
+
+- `data/03_primary/`: primary prepared inputs (sensor, weather, irrigation, locations, crop, soil, satellite).
+- `data/04_model_input/`: merged model-ready tables (`full_table_<consortium>.parquet`).
+- `data/05_xgcast_input/`: normalized training/validation/test artifacts for XGBoost.
+- `data/05_aquacrop_input/<consortium>/<sensor>/`: weather/irrigation/settings for AquaCrop runs.
+- `data/06_xgcast_output/`: trained XGBoost model files.
+- `data/06_aquacrop_output/<consortium>/<sensor>/`: AquaCrop simulation outputs.
+
+## Requirements
+
+- Python 3.11+
+- `uv` (recommended for environment/dependency management)
+
+Project dependencies are declared in `pyproject.toml`.
+
+## Setup
+
+From the repository root:
+
+```bash
+uv sync
+```
+
+If you prefer running with the virtual environment directly:
+
+```bash
+uv run python -m main
+```
+
+## Configuration
+
+Base configuration files committed in this repo:
+
+- `config/params.yml`
+- `config/aquacrop_params.yml`
+- `config/xgcast_params.yml`
+- `config/fieldsensor_irrigator_mapping_anonym.yaml`
+- `config/request_scripts/*.js`
+
+### Important: private/non-versioned configs
+
+Some optional pipeline branches (especially satellite download/auth flows) expect private files that are intentionally gitignored, for example:
+
+- `config/copernicus_oauth_config.json`
+- `config/keycloak_config.json`
+- `config/pre_anonym_params.yml`
+- potentially consortium-specific field mapping files (for field-level satellite aggregation)
+
+If those files are missing, full satellite acquisition workflows will not run.
+
+## Running The Pipelines
+
+### Full run (same sequence as `main.py`)
+
+```bash
+uv run python -m main
 ```
-predictive_irrigation_models/
-├── config/
-│ ├── params.yml
-│ ├── xgcast_params.yml
-│ ├── aquacrop_params.yml
-│ └── fieldsensor_irrigator_mapping_anonym.yaml.yml
-├── pipelines/
-│ ├── __init__.py
-│ ├── aquacrop_preparation_pipeline.py
-│ ├── data_collection_pipeline.py
-│ ├── aquacrop_preparation_pipeline.py
-│ ├── demo_run.py
-│ ├── model_preparation_pipeline.py
-│ ├── preprocessing_pipeline.py
-│ ├── resample_impute_pipeline.py
-│ ├── soilcast_pipeline.py
-│ ├── xgcast_pipeline.py
-│ └── xgcast_run.py
-├── data/
-│ ├── 03_primary/
-│ ├── 04_model_input/
-│ ├── 05_aquacrop_input/
-│ ├── 05_xgcast_input/
-│ ├── 06_aquacrop_output/
-│ └── 06_xgcast_output/
-├── tools/
-│ └── ...
-└── README.md
+
+### Run individual flows
+
+```bash
+uv run python -m pipelines.model_preparation_pipeline
+uv run python -m pipelines.aquacrop_preparation_pipeline
+uv run python -m pipelines.aquacrop_pipeline
+uv run python -m pipelines.xgcast_pipeline
 ```
 
-## Features
+## Expected Input Artifacts
+
+At minimum, model preparation expects consortium-scoped files such as:
+
+- `data/03_primary/field_sensor_data_<consortium>.parquet`
+- `data/03_primary/irrigation_data_<consortium>.parquet`
+- `data/03_primary/locations_ids_<consortium>.parquet`
+- `data/03_primary/historical_weather_data_<consortium>.parquet`
+- `data/03_primary/forecasted_weather_data_<consortium>.parquet`
 
-- **Prefect-based Pipelines:** Modular tasks and flows for data collection, transformation, and saving.
-- **Sensor & Weather Data Integration:** Reads and merges raw sensor and weather data for multiple consortia.
-- **Soil, Crop, and Remote Sensing Data:** Integrates geospatial and tabular data sources.
-- **Automated Testing:** Prefect tasks and flows for validating preprocessing results.
-- **Artifact Logging:** Data summary artifacts for monitoring pipeline outputs.
+Optional inputs are controlled through `data_availability` in `config/params.yml`:
 
-## Getting Started
+- weather sensor data,
+- crop type data,
+- soil type data,
+- remote sensing data.
 
-### Prerequisites
+## Models
 
-- Python 3.8+
-- [Prefect](https://www.prefect.io/)
-- uv
+### XGCast (XGBoost)
 
-Install dependencies:
-```uv sync```
+- training configuration in `config/xgcast_params.yml`.
+- outputs model file `data/06_xgcast_output/xgcast_<consortium>.json`.
 
-Configuration
-Edit `config/params.yml` to specify consortia names, data folders, and other parameters.
+### AquaCrop
 
-### Running the Pipeline
+- uses local `aquacrop/` package implementation.
+- simulation outputs include water flux, water storage, crop growth, and summary tables under `data/06_aquacrop_output/`.
 
-From the project root, run:
+## Notebook Demo
 
-```python -m main```
+Open `notebooks/notebook_demo.ipynb` for an interactive walkthrough using consortium/sensor examples and model outputs.
 
-### Running Tests
-From the project root, run:
+## Known Caveats
 
-```python -m test.test_results```
+- `data/` is gitignored; expected datasets are not included in this repository.
+- several remote-sensing paths depend on private configuration and credentials.
 
-### Contributing
-Feel free to open issues or submit pull requests for improvements or bug fixes.
+## Contributing
 
-### License
-MIT License
+Contributions are welcome. Open an issue or submit a pull request with:
 
-### Contact
-For questions or collaboration, please contact the repository owner. ``````
+- the problem statement,
+- reproducible steps,
+- and any data/config assumptions required to test the change.