/ihs-deo

Primary LanguagePythonApache License 2.0Apache-2.0

ihs-deo

Project to automate data sourcing and validation from IHS Energy's Energy Web Services.

Features include:

  • Keep your local dataset in sync with IHS

  • Scheduled well & production downloads from IHS data services

  • Highly configurable scheduled tasks:

    • download by county, API list, or by producing entity ID
    • download using your existing queries stored on the IHS website

  • Parses and normalizes IHS data

  • Backed by MongoDB

  • Built in REST API

Available on DockerHub!

Issues

  • Pagination mixin is broken. It doesnt seem to respect the passed query string
  • Indexes are currently managed external to the app. These need to be ported into the model definitions now that they are somewhat stable.

Todo

  • Documentation
  • Improve API error logging
  • Add API endpoints for CLI
  • Unit Testing
  • Documentation
  • Process changes and deletes (nightly)
  • query production in a date range for a specific api?
  • add endpoint: add/remove county definition (name, state_code, county_code)
  • scheduled task: cleanup stale wells
  • mechanism to stop submitting exports if remote capacity is full

Usage

  • Launch application services:

    docker-compose up

Example environment variables

IHS_URL=http://www.ihsenergy.com
IHS_USERNAME=<your_ihs_password>
IHS_PASSWORD=<your_ihs_username>
IHS_APP_KEY=<provided_by_ihs>

DATABASE_URI=mongodb://docker.for.mac.host.internal:27017
CELERY_BROKER_URL=redis://redis:6390/1
IHS_CRON_URL=redis://redis:6390/0
GUNICORN_CMD_ARGS="--bind=0.0.0.0:80 --log-level=info --name=gunicorn-ihs --timeout=120 --graceful-timeout=120 --worker-class=gevent --workers=3"