wis2-grep is a Reference Implementation of a WIS2 Global Replay Service.
- connects to a WIS2 Global Broker, subscribed to the following topics:
origin/a/wis2/#
andcache/a/wis2/#
- on notification messages
- publish to a WIS2 Global Replay Service (OGC API - Features) using one of the supported transaction backends:
- OGC API - Features - Part 4: Create, Replace, Update and Delete
- Elasticsearch direct (default)
- publish to a WIS2 Global Replay Service (OGC API - Features) using one of the supported transaction backends:
- user-defined subscriptions
- users can execute a process to subscribe to notification messages based on topic and/or datetime
- Python 3
- virtualenv
Dependencies are listed in requirements.txt. Dependencies are automatically installed during pywis-pubsub installation.
# setup virtualenv
python3 -m venv --system-site-packages wis2-grep
cd wis2-grep
source bin/activate
# clone codebase and install
git clone https://github.com/wmo-im/wis2-grep.git
cd wis2-grep-management
python3 setup.py install
# setup environment and configuration
cp wis2-grep.env local.env
vim local.env # update accordingly
source local.env
# setup pywis-pubsub - sync WIS2 notification schema
pywis-pubsub schema sync
# setup backend
wis2-grep setup
# setup backend (force reinitialization of backend)
wis2-grep setup --force
# teardown backend
wis2-grep teardown
# connect to Global Broker
# notifications will automatically trigger wis2-grep to publish
# WNM to the API identified in wis2-grep.env (WIS2_GREP_GB)
pywis-pubsub subscribe --config pywis-pubsub.yml
# loading notification messsage manually (single file)
wis2-grep register /path/to/wnm-file.json
# loading notification messages manually (directory of .json files)
wis2-grep load /path/to/dir/of/wnm-files
# clean messages from API index
wis2-grep clean --hours 24
# by topic
curl "http://localhost/collections/wis2-notification-messages/items?q=%22cache/b/wis2%22"
# by bounding box (Canada):
curl "http://localhost/collections/wis2-notification-messages/items?bbox=-142,42,-5,84"
# by publication time (from/to):
curl "http://localhost/collections/wis2-notification-messages/items?datetime=2024-07-24T11:11:11Z/2024-07-25T12:34:21Z"
# by publication time (from):
curl "http://localhost/collections/wis2-notification-messages/items?datetime=2024-07-24T11:11:11Z/.."
# by publication time (to):
curl "http://localhost/collections/wis2-notification-messages/items?datetime=../2024-07-24T11:11:11Z"
# by message identifier
curl "http://localhost/collections/wis2-notification-messages/items/<WNM_ID>"
# sort results by oldest messages (pubtime)
curl "http://localhost/collections/wis2-notification-messages/items?sortby=pubtime"
# sort results by latest messages (pubtime)
curl "http://localhost/collections/wis2-notification-messages/items?sortby=-pubtime"
# return as GeoJSON
curl "http://localhost/collections/wis2-notification-messages/items?f=json"
# return as HTML
curl "http://localhost/collections/wis2-notification-messages/items?f=html"
The Docker setup uses Docker and Docker Compose to manage the following services:
- wis2-grep-api: API powered by pygeoapi
- wis2-grep-broker: MQTT broker
- wis2-grep-management: management service to publish notification messages published from a WIS2 Global Broker instance
- the default Global Broker connection is to NOAA. This can be modified in
wis2-grep.env
to point to a different Global Broker
- the default Global Broker connection is to NOAA. This can be modified in
- wis2-grep-backend: API search engine backend (default Elasticsearch)
See wis2-grep.env
for default environment variable settings.
To adjust service ports, edit docker-compose.override.yml
accordingly.
The Makefile
in the root directory provides options to manage the Docker Compose setup.
# build all images
make build
# build all images (no cache)
make force-build
# start all containers
make up
# API is up at http://localhost
# reinitialize backend
make reinit-backend
# start all containers in dev mode
make dev
# API is up at http://localhost
# view all container logs in realtime
make logs
# login to the wis2-grep-management container
make login
# restart all containers
make restart
# shutdown all containers
make down
# remove all volumes
make rm
# install dev requirements
pip3 install -r requirements-dev.txt
# run tests like this:
python3 tests/run_tests.py
# or this:
python3 setup.py test
All bugs, enhancements and issues are managed on GitHub.