sensor-web-harvester

New Custom Network/Offerings read more at the bottom

sensor-web-harvester is a Scala project that harvests sensor data from web sources. The data is then pushed to an SOS using the sos-injection module project. SosInjector is a project that wraps an Sensor Observation Service (SOS). The sos-injection module provides Java classes to enter stations, sensors, and observations into an SOS.

sensor-web-harvester is used to fill an SOS with observations from many well-known sensor sources (such as NOAA and NERRS). This project pulls sensor observation values from the source’s stations. It then formats the data to be placed into the user’s SOS by using the sos-injector. The source stations used are filtered by a chosen bounding box area.

The current sources that observations are pulled from are:

This project uses an H2 metadata database to store station information from the sources. The metadata database information is used to retrieve observations from the stations' sources. The database is file based and autogenerated if it doesn't already exist.

This project can be executed by running the pre-built jar with the command line (see "Running the SOS Injector") or by writing custom Java code (see "Writing Custom Java Code").

Installation

This project can be used on either a Windows or Linux computer. An Apple computer is expected to work however it has not been tested.

The following are the requirements to run this project:

Java 1.6 or newer
An already running instance of an IOOS Customized 52 North SOS

Configuring sensor-web-harvester

The pre-built sensor-web-harvester.jar and example_sos.properties can be downloaded from the GitHub releases page.

The command line takes in a properties file which contains all of the needed variables to perform an SOS update. The properties file requires the following variables:

# The URL where the H2 metadata database should be stored
# Note: if running in Docker, use jdbc:h2:/srv/swhdb/db
database_url = jdbc:h2:/usr/local/sensor_web_harvester

# The URL to the SOS being used.
sos_url = http://localhost:8080/i52n-sos

# The publisher's country
publisher_country = USA

# The publisher's email address
publisher_email = publisher@example.com

# The web address of the publisher
publisher_web_address = http://example.org

# The name of the publishing organization
publisher_name = RA

# The northernmost latitude of the bounding box
north_lat = 50.0

# The southernmost latitude of the bounding box
south_lat = 40.0

# The westernmost longitude of the bounding box
west_lon = -93.0

# The easternmost longitude of the bounding box
east_lon = -75.0

# The network root for the default network in the SOS that contains all the stations. This network is different for each SOS. For example for AOOS the defaut network is urn:ioos:network:aoos:all.  The "network_root_id" is "all" and the "network_root_source_id" is "aoos".
network_root_id = all
network_root_source_id = aoos

# semi-colon seperated value list of sources that are to be updated (optional: this will default to 'all' if it is not the properties file)
# sources = all - this will operate on all known sources
# sources = nerrs;storet;glos - this will operate on the nerrs, storet and glos sources
# Accepted values: all, glos, hads, ndbc, nerrs, noaa_nos_coops, noaaweather, raws, snotel, storet, usgswater
sources = all

An example of a properties file named example_sos.properties is also provided on the Github releases page.

Running with Docker

See below for explanations of the metadata, updatesos, and writeiso modes.

The following is an example of a Docker execution (Docker 1.9.0+ required). You must create your sos.properties config file first. You do not have to install any dependencies other than Docker.

docker run --rm -v swhdb:/srv/swhdb -v $(pwd)/sos.properties:/tmp/sos.properties \
  ioos/sensor-web-harvester -updatesos /tmp/sos.properties

Running sensor-web-harvester

Note: Running these processes can take a long time (hours) as information is downloaded and extracted from many sources.

The sensor-web-harvester has three modes:

metadata

This mode harvests from the source(s) defined in the properties file and updates the metadata database. This command should be run conservatively (approx. 3 times a week) since the sources’ stations do not change often and this command is taxing on the sources’ servers.

java -jar sensor-web-harvester.jar -metadata [path to properties file]

writeiso

This mode writes ISO 19115-2 metadata files based on data in the metadata database

java -jar sensor-web-harvester.jar -writeiso [path to properties file]

updatesos

This mode downloads data from the sources and injects the data into the 52N instance specified in the properties file. Do not call this command more than once hourly (for reasons previously stated).

Example:

java -jar sensor-web-harvester.jar -updatesos [path to properties file]

Writing Custom Java Code

This is example code demonstrating how to update the metadata database and the SOS from within custom Java code.

// Southern California Bounding Box
Location southWestCorner = new Location(32.0, -123.0);
Location northEastCorner = new Location(35.0, -113.0);
BoundingBox boundingBox = new BoundingBox(southWestCorner, northEastCorner);

String databaseUrl = "jdbc:postgresql://localhost:5432/sensor";
String databaseUsername = "sensoruser";
String databasePassword = "sensor";
String sosUrl = "http://localhost:8080/sos/sos";

MetadataDatabaseManager metadataManager = new MetadataDatabaseManager(databaseUrl, 
  databaseUsername, databasePassword, boundingBox)

// Updates the local metadata database with station information
// This call should be made conservatively (approx. 3 times a week) since the 
// sources’ stations do not change often and this call is taxing on the sources’ servers.
metadataManager.update();

// Information about the group publishing this data on the SOS. 
PublisherInfoImp publisherInfo = new PublisherInfoImp();
publisherInfo.setCountry("USA");
publisherInfo.setEmail("publisher@domain.com");
publisherInfo.setName("IOOS");
publisherInfo.setWebAddress("http://www.ioos.gov/");

SosNetworkImp rootNetwork = new SosNetworkImp()
rootNetwork.setId("all")
rootNetwork.setSourceId("aoos")

SosSourcesManager sosManager = new SosSourceManager(databaseUrl, 
  databaseUsername, databasePassword, sosUrl, publisherInfo, rootNetwork);
  
// Updates the SOS with data pulled from the source sites. 
// This uses the metadata database
// Most of the data is hourly. The data should be pulled conservatively (approx. hourly) 
// since the observations do not change often and this action is taxing on the sources’ servers.
sosManager.updateSos();

Create Custom Networks for Sources or Stations

To create custom networks/offerings one must adjust three tables (network, network_source, network_station) in the metadata database. All new networks need to be created and associated to source or stations before they are submitted to the SOS. Meaning that if a station is already created on the SOS it cannot later be associated to a network.

First step, each custom network needs be added to the network table. The tag and source_tag columns are the main columns that need filled in for a new network. The tag and source_tag are combined (urn:ioos:network:[source_tag]:[tag]) to create the id of the network in the SOS.

These custom networks can be assoicated to all stations of a source with the use of the network_source table. To associate a network with a source, place the source's database id and the network's database id in a row.

These custom networks can be associated to specific stations from the network_station table. A row needs to be created for each station that a network is assoicated to. In each of these rows add the network id and the station id to be associated.

List of Sources URLs

HADS

Pull stations Information

http://amazon.nws.noaa.gov/hads/goog_earth/ - used to get the list of state URLs
http://amazon.nws.noaa.gov/cgi-bin/hads/interactiveDisplays/displayMetaData.pl?table=dcp&nesdis_id= - used to pull station information. Needs the stations id or foreign id at the end of the URL

Observation Retrieval

http://amazon.nws.noaa.gov/nexhads2/servlet/DecodedData - pull observations for a station. A POST request with the needed values pairs of:

state = nil
hsa = nil
of = 1
nesdis_ids = [station id]
sinceday = [number of days of observations requested]

Pull stations Information

http://weather.noaa.gov/data/nsd_cccc.txt

Observation Retrieval

http://www.nws.noaa.gov/data/obhistory/[station id].html

RAWS

Pull stations Information

http://www.raws.dri.edu/cgi-bin/wea_info.pl?[station id] - get station information
List of URLs for each state used to find stations

Observation Retrieval

http://www.raws.dri.edu/cgi-bin/wea_list2.pl - pull observations for a station. A POST request with the needed values pairs of:

stn = [station id]
smon = [start month - two digit Integer]
sday = [start day of month - two digit Integer]
syea = [start year - two digit Integer]
emon = [end month - two digit Integer]
eday = [end day of month - two digit Integer]
eyea = [end year - two digit Integer]
dfor = 02
srce = W
miss = 03
flag = N
Dfmt = 02
Tfmt = 01
Head = 02
Deli = 01
unit = M
WsMon = 01
WsDay = 01
WeMon = 12
WeDay = 31
WsHou = 00
WeHou = 24

SnoTel

Pull stations Information

http://www.wcc.nrcs.usda.gov/ftpref/data/water/wcs/earth/snotelwithoutlabels.kmz - contains a list of all the stations
http://www.wcc.nrcs.usda.gov/nwcc/sensors - list the sensor of a station. A POST request with the needed values pairs of:

sitenum = [station id]

Observation Retrieval

http://www.wcc.nrcs.usda.gov/nwcc/view - pull observations for a station. A POST request with the needed values pairs of:

time_zone = PST
sitenum = [station id]
timeseries = Hourly
interval = WEEK
format = copy
report = ALL

USGS Water

Pull stations Information

http://waterservices.usgs.gov/nwis/iv?stateCd=[state tag]&period=PT4H
state tags = "al", "ak", "aq", "az", "ar", "ca", "co", "ct", "de", "dc", "fl", "ga", "gu", "hi", "id", "il", "in", "ia", "ks", "ky", "la", "me", "md", "ma", "mi", "mn", "ms", "mo", "mt", "ne", "nv", "nh", "nj", "nm", "ny", "nc", "nd", "mp", "oh", "ok", "or", "pa", "pr", "ri", "sc", "sd", "tn", "tx", "ut", "vt", "vi", "va", "wa", "wv", "wi", "wy"

Observation Retrieval

http://waterservices.usgs.gov/nwis/iv?sites=[station id]&parameterCd=[sensor ids]&startDT=[start date formated as yyyy-MM-dd'T'HH:mm:ss'Z']&endDT=[end date formated as yyyy-MM-dd'T'HH:mm:ss'Z']

NERRS

This source has a webservice end point at http://cdmo.baruch.sc.edu/webservices2/requests.cfc?wsdl that is used to pull the station information and the observations for each station.

A java jar was create to work with this webservice in java. Below shows references needed to use this jar in Maven or it can be downloaded at http://nexus.axiomalaska.com/nexus/content/repositories/public/com/axiomalaska/nerrs_webservice/1.0.0/nerrs_webservice-1.0.0.jar

Maven Dependency

<repository>
  <id>axiom_public_releases</id>
  <name>Axiom Releases</name>
  <url>http://nexus.axiomalaska.com/nexus/content/repositories/public/</url>
</repository>
<dependency>
  <groupId>com.axiomalaska</groupId>
  <artifactId>nerrs_webservice</artifactId>
  <version>1.0.0</version>
</dependency>

argysamo/sensor-web-harvester

sensor-web-harvester

New Custom Network/Offerings read more at the bottom

Installation

Configuring sensor-web-harvester

Running with Docker

Running sensor-web-harvester

Note: Running these processes can take a long time (hours) as information is downloaded and extracted from many sources.

metadata

writeiso

updatesos

Writing Custom Java Code

Create Custom Networks for Sources or Stations

List of Sources URLs

HADS

Pull stations Information

Observation Retrieval

NDBC

NOAA NOS CO-OPS

NOAA Weather

Pull stations Information

Observation Retrieval

RAWS

Pull stations Information

Observation Retrieval

SnoTel

Pull stations Information

Observation Retrieval

USGS Water

Pull stations Information

Observation Retrieval

NERRS

Maven Dependency