socib/glider_toolbox

MATLAB/Octave scripts to manage data collected by a glider fleet, including data download, data processing and product and figure generation, both in real time and delayed time.

glider_toolbox

The glider toolbox is a set of MATLAB/Octave scripts and functions developed at SOCIB to manage the data collected by a glider fleet. They cover the main stages of the data management process both in real time and delayed time mode: metadata aggregation, data download, data processing, and generation of data products and figures.

Features

The following features are already implemented in the glider toolbox:

• Two main scripts to perform real time and delayed time data processing:
  • main_glider_data_processing_rt
  • main_glider_data_processing_dt
• Two processing functions to perform both modes of data processing:
  • gliderDataProcessing
  • deploymentDataProcessing
• Support for different glider models:
  • Slocum G1 and G2
  • Seaglider
  • SeaExplorer
• Deployment metadata gathering from virtually any database.
• File retrieval from multiple dockservers/basestations for real time processing.
• Improved Slocum raw data loading from ascii files (.dba).
• Improved Seaglider raw data loading from ascii files (.log and .eng).
• SeaExplorer raw data loading from ascii files (.gli and .pld).
• Data processing, including:
  • unit conversions
  • factory calibrations
  • corrections, with optional parameter estimation
  • derivations
• Data interpolation over instantaneous regular vertical profiles.
• Generation of figure products.
• Generation of NetCDF data products.
• Configuration of every processing stage using configuration files.
  • raw data retrieval, storage and load options
  • parameters for conversions, calibrations, derivations, corrections, interpolations and filters
  • parameters for data gridding
  • customizable standard NetCDF product and figure outputs

The following features are planned or in development:

• automatic quality control of processed data.

Documentation

The toolbox is exhaustively self-documented using the standard documentation comment system. Hence the help pages are available using the documentation browser or the help command.

An automatically generated copy of the documentation is available online. These pages are generated by m2html, and may be built from source by running make doc from the toolbox top directory.

In addition, the toolbox contains a quick start document to guide users through the main parts of the installation. It provides an overview of the parts of the toolbox that are configurable by the user.

Outline

This diagram outlines the flow in the delayed time processing script main_glider_data_processing_dt. The real time processing script main_glider_data_processing_rt is similar but includes steps to retrieve the glider data files from the basestations/dockservers. The data flow diagram illustrates the different steps of the data processing of the gliderDataProcessing. It focuses on the real time processing mode in the sense that it includes the retrieval from the dockserver. However, except for the initial steps, it also represents the delayed time processing.

For further details about the processing and its configuration, please refer to the documentation of those scripts (in other words: read the docs! ;-).

Hidden gems

Some common utilities included in the toolbox might be useful even when the burden of an operational set of scripts and functions developed to automatically process the data from a glider fleet is not needed:

• Slocum data file API: dba2mat, dbacat, and dbamerge
• Seaglider data file API: sglog2mat, sglogcat, sgeng2mat, sgengcat, and sglogenmerge
• SeaExplorer data file API: sx2mat, sxcat, and sxmerge
• More convenient NetCDF interface: loadnc and savenc
• RFC-compliant JSON interface: loadjson and savejson
• C-style character array conversion: strc
• String formatting of scalar structs: strfstruct
• SFTP interface compatible with native FTP: sftp

Bugs, issues and contributions

Contributions and criticism are welcome.

If you have any doubt or problem, please fill an issue!

If you fix something or want to add some contribution, many thanks in advance!

A note on style: the MATLAB/Octave code in the toolbox follows these coding style guidelines. If you have your own style don't worry. But if you don't, please consider following them. You may also save some typing using the function and script template helpers there!

Legacy

This toolbox is based on the previous code developed at IMEDEA and SOCIB by Tomeu Garau. He is the true glider man.

Copyright

Copyright (C) 2013-2018 ICTS SOCIB - Servei d'observació i predicció costaner de les Illes Balears http://www.socib.es

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.