Correction workflow and spatial database model of Aquopts - A Hydrological Optical Data Processing System
Description
In order to improve the capacity of storage, exploration and processing of sensor data, a spatial DBMS was used and the Aquopts system was implemented. In field surveys using different sensors on the aquatic environment, the existence of spatial attributes in the dataset is common, motivating the adoption of PostgreSQL and its spatial extension PostGIS. To enable the insertion of new data sets as well as new devices and sensing equipment, the database was modeled to support updates and provide structures for storing all the data collected in the field campaigns in conjunction with other possible future data sources. The database model provides resources to manage spatial and temporal data and allows flexibility to select and filter the dataset. The data model ensures the storage integrity of the information related to the samplings performed during the field survey in an architecture that benefits the organization and management of the data. However, in addition to the storage specified on the data model, there are several procedures that need to be applied to the data to prepare it for analysis. Some validations are important to identify spurious data that may represent important sources of information about data quality. Other corrections are essential to tweak the data and eliminate undesirable effects. Some equations can be used to produce other factors that can be obtained from the combination of attributes. In general, the processing steps comprise a cycle of important operations that are directly related to the characteristics of the data set. Considering the data of the sensors stored in the database, an interactive prototype system, named Aquopts, was developed to perform the necessary standardization and basic corrections and produce useful data for analysis, according to the correction methods known in the literature. The system provides resources for the analyst to automate the process of reading, inserting, integrating, interpolating, correcting, and other calculations that are always repeated after exporting field campaign data and producing new data sets. All operations and processing required for data integration and correction have been implemented from the PHP and Python language and are available from a Web interface, which can be accessed from any computer connected to the internet. The data access cab be access online (http://sertie.fct.unesp.br/aquopts), but the resources are restricted by registration and permissions for each user. After their identification, the system evaluates the access permissions and makes available the options of insertion of new datasets. The source-code of the entire Aquopts system are available at: https://github.com/carmoafc/aquopts The system and additional results were described on the official paper (under review)