GeoAnalytic-code/daas-catseaice

Creates STAC catalogs of ice chart data provided by National Ice Center and Canadian Ice Service

DaaS-CatSeaice

Publish static STAC catalogs for weekly Ice Charts as published by the National Ice Center and Canadian Ice Service.
Explore the docs »

Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Installation
   • Docker
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgements

About The Project

This code may be used to generate STAC catalogs of ice charts available on government websites. Typically, these are provided as ESRI e00 files or Shapefiles in ZIP archives on a weekly schedule.

The catalog entries are stored locally in a SQLite database which can be updated when needed. A separate function will output STAC catalogs from the database.

Built With

• PyStac
• BeautifulSoup
• Fiona
• Shapely

Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites

• Python 3
• Firefox
• Geckodriver

Also recommended:

• Docker

Installation - Local

1. Clone the repo

   git clone https://github.com/GeoAnalytic-code/daas-catseaice.git

2. Install requirements

   pip install requirements.txt

3. Run tests

   pytest

4. Fill the database - this command will put the database in the data/ folder

   python src/catseaice.py fill -d data/icecharts.sqlite

5. Report the contents of the database

   python src/catseaice.py report -d data/icecharts.sqlite

6. Write a static STAC catalog

   python catseaice.py write data/stac -d data/icecharts.sqlite

Docker

If you don't want to mess around with the requirements and system level stuff, you can use Docker.

1. Ensure you have Docker installed, then

   docker build -t daas/catseaice https://github.com/GeoAnalytic-code/daas-catseaice.git

2. Run it!

   docker run --rm daas/catseaice pytest
   docker run --rm daas/catseaice python3 src/catseaice.py -help

3. To use a permanent database, map the local drive to the docker container

   docker run --rm -v $(pwd):/opt/app/data daas/catseaice python3 src/catseaice.py fill -d data/icecharts.sqlite
   docker run --rm -v $(pwd):/opt/app/data daas/catseaice python3 src/catseaice.py report -d data/icecharts.sqlite
   docker run --rm -v $(pwd):/opt/app/data daas/catseaice python3 src/catseaice.py write data/stac -d data/icecharts.sqlite  

   Note that the first time you use the fill command it will take some time as the program queries the websites at NIC and CIS and populates the database.
   To shorten this step, you can limit the time frame searched, like so:

    docker run --rm -v $(pwd):/opt/app/data daas/catseaice python3 src/catseaice.py fill -d data/icecharts.sqlite -S 2019-01-01

   This command will only search for data from Jan 1, 2019 to the present.

Usage

The examples shown below are for a local installation. For a Docker installation, refer to the usage examples shown above.

Check the version and help:

 $ python catseaice.py --version
 Catalog Ice Charts 1.0

 $ python catseaice.py -h
 Create STAC Catalogs of Ice Charts

 Usage:
   catseaice fill [-A | -S YYYY-MM-DD] [-e | -E] [-d DBNAME]
   catseaice report [-d DBNAME]
   catseaice write BASE_HREF [-t CTYPE] [-d DBNAME]
   catseaice (-h | --help)
   catseaice --version
 
 
 Options:
   -h --help     Show this screen.
   --version     Show version.
   -S YYYY-MM-DD Start date for searching for icecharts
   -A            Search for all available icecharts (otherwise just update the database)
   -e            Calculate exact geometry for all newly discovered charts  (not usually required)
   -E            Calculate exact geometry for each chart in the database (not usually required)
   -d DBNAME     name of the database to use [default: icecharts.sqlite]
   BASE_HREF     root folder/url of output STAC catalog, default is the current directory [default: ...]
   -t CTYPE      STAC catalog type [default: SELF_CONTAINED]
                 other valid values include ABSOLUTE_PUBLISHED and RELATIVE_PUBLISHED

Fill up the database for the first time:

 $ python catseaice.py fill -d /path/to/database.sqlite

This will save the locations of the ice charts to a specified database. If the -d flag is omitted, a default SQLite database named icecharts.sqlite will be created in the local folder.

Note that while querying the NIC site is a couple of form submissions, getting the file information from the CIS site is much more intensive and it is likely that there is some throttling going on. You can limit the amount of time required by setting a start date for the query, like so:

 $ python catseaice.py fill -S 2019-01-01

Export a STAC catalog:

 $ python catseaice.py write /path/to/write/to

This will write the catalog to the specified directory using the SELF_CONTAINED style.

Report the contents of the database:

 $ python catseaice.py report

Details

The fill process will query the NIC and CIS websites and save metadata about any weekly ice charts it finds to a SQLite database. The write process with export a static STAC catalog structure from the database, orgainized in terms of Datasource, Region, and Year.
The actual icechart data remains at the locations (at the National Ice Center or Canadian Ice Service) it was originally found. This means that the catalog may become stale if the files are moved, renamed, or removed.

Roadmap

See the open issues for a list of proposed features (and known issues).

Unit Tests

This command will run the unit tests in a docker container, writing a coverage report to htmlcov/index.html

$ docker run --rm -v $(pwd):/opt/app daas/catseaice pytest --cov=src --cov-report=html

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

David Currie - @GeoAnalyticInc - info@geoanalytic.com

Project Link: https://github.com/GeoAnalytic-code/daas-catseaice

Acknowledgements

A portion of this work was generously supported through the GeoConnections programme, part of Canada's Spatial Data Infrastructure.

• GeoConnections