Tech4Aqua: Multiparametric Physicochemical and Meteorological Dataset with Domain-Driven Feature Engineering for Microbiological Water Quality Early Warning in Northern Portuguese Reservoirs
Description
This dataset contains multiparametric physicochemical, hydrometeorological, and microbiological monitoring records collected within the TECH4AQUA project. It establishes a machine learning framework for proactive early warning of microbiological non-compliance in municipal drinking water storage reservoirs. The repository integrates 21 years (2004–2025) of regulatory grab-sampling across 118 Water Supply Zones (ZAPEs) in northern Portugal (Boticas, Chaves, Montalegre), merged with daily regional meteorological observations (daily maximum ambient temperature, precipitation, and relative humidity) retrieved via historical weather API. Contents included: 1. 01_Data/Base_Tech4Aqua_Features_Avancadas.csv (and .xlsx): Final modeled dataset containing 522 monitoring episodes and 103 engineered features, incorporating thermal-chemical disinfectant degradation ratios, particulate demand ratios, antecedent precipitation EWMA, hydraulic retention proxies, and strict backward-shifted facility memory. 2. 01_Data/Base_Revisada_Tech4Aqua.xlsx: Preprocessed raw transactional monitoring records merged with daily weather data. 3. 01_Data/Data_Dictionary.csv: Detailed variable definitions, units, and statutory regulatory limits. 4. 02_Scripts/: Four standalone Python scripts implementing feature engineering, Bayesian model optimization (Optuna), TreeSHAP interpretability, and a 5-vector forensic data leakage audit. For exhaustive variable definitions, kinetic mathematical formulations, and step-by-step reproduction instructions, please refer to the included README.md.
Files
Steps to reproduce
Prerequisites: Python 3.10 or 3.11 installed with standard scientific libraries. Step 1: Environment Setup Extract the repository package and install the required dependencies using: pip install -r requirements.txt Dependencies include: pandas (>=2.0.0), numpy (>=1.24.0), scikit-learn (>=1.3.0), lightgbm (>=4.0.0), xgboost (>=2.0.0), optuna (>=3.3.0), shap (>=0.43.0), matplotlib (>=3.7.0), and openpyxl (>=3.1.0). Step 2: Feature Engineering & Preprocessing Execute the feature engineering pipeline: python 02_Scripts/01_feature_engineering.py This script cleans raw transactional grab-sample records from '01_Data/Base_Revisada_Tech4Aqua.xlsx', pivots them into an episode-level wide matrix, derives kinetic degradation ratios (Phi, Psi), computes hydrometeorological cumulative windows (3-day and 7-day rain), calculates seasonal cyclical harmonics (sin/cos of month), and applies strict backward shift(1) memory per facility. Output is saved to '01_Data/Base_Tech4Aqua_Features_Avancadas.csv'. Step 3: Model Training & Bayesian Hyperparameter Optimization Execute model training and validation: python 02_Scripts/02_model_training_optuna.py This script evaluates Random Forest, XGBoost, and LightGBM across: - 5-Fold Stratified Cross-Validation (surveillance regime) - 5-Fold Blind Spatial Group Cross-Validation (transferability to unseen ZAPEs) Hyperparameters are tuned using Optuna (Tree-structured Parzen Estimator, 35 iterations per model) optimizing F1-score and Recall. Step 4: TreeSHAP Interpretability Generate global summary beeswarm plots and local non-linear partial dependence curves: python 02_Scripts/03_explainability_shap.py Visualizations will be output to the 'figures/' directory, highlighting non-linear tipping points (e.g., residual chlorine depletion under thermal stress). Step 5: Forensic Leakage Audit Verify mathematical and temporal integrity: python 02_Scripts/04_forensic_audit.py Runs the 5-vector forensic validation looking for target leakage, confirmation of backward shift(1) initialization at t=0, fold-isolated imputation, and chronological holdout testing (Train 2008-2022 vs Test 2022-2025).
Institutions
- Polytechnic Institute of BragançaBragança, Bragança
- Universidade Tecnológica Federal do ParanáParaná, Curitiba