A simple & opinionated Python command line tool to access Australian National Energy Market (NEM) data provided by the Australian Energy Market Operator (AEMO). It features:
- access to the NEMDE dataset as well as the predispatch, unit-scada, trading-price, demand and interconnectors tables from MMSDM,
- a cache to not re-download data that already exists in
~/nem-data/data
, - adds
interval-start
andinterval-end
columns.
It is designed for use by researchers & data scientists - this tool supports my personal research work. It is not designed for production use - see NEMOSIS for a production grade package.
See A Hackers Guide to AEMO & NEM Data for more context on the data provided by AEMO.
$ pip install nemdata
$ nemdata --help
Usage: nemdata [OPTIONS]
Downloads NEM data from AEMO.
Options:
-t, --table TEXT Available tables: nemde, dispatch-price,
predispatch, unit-scada, trading-price, demand,
interconnectors, p5min, predispatch-sensitivities,
predispatch-demand.
-s, --start TEXT Start date (YYYY-MM or YYYY-MM-DD for NEMDE).
-e, --end TEXT End date (incusive) (YYYY-MM or YYYY-MM-DD for
NEMDE).
--dry-run / --no-dry-run Whether to save downloaded data to disk.
--help Show this message and exit.
Download NEMDE data for the first three days in January 2018:
$ nemdata -t nemde --start 2018-01-01 --end 2018-01-03
Download trading price data from MMSDM for January to March 2018:
$ nemdata -t trading-price -s 2018-01 -e 2018-03
Download trading price data from MMSDM for January to Feburary 2020:
import nemdata
data = nemdata.download(start="2020-01", end="2020-02", table="trading-price")
Load this data back into a pandas DataFrame:
data = nemdata.load()['trading-price']
At this point, data
will have the trading price for all regions.
Downloaded into into $HOME/nem-data/data/
:
$ nemdata -t trading-price -s 2020-01 -e 2020-02
$ tree ~/nem-data
/Users/adam/nem-data
└── data
└── trading-price
├── 2020-01
│ ├── PUBLIC_DVD_TRADINGPRICE_202001010000.CSV
│ ├── clean.csv
│ ├── clean.parquet
│ └── raw.zip
└── 2020-02
├── PUBLIC_DVD_TRADINGPRICE_202002010000.CSV
├── clean.csv
├── clean.parquet
└── raw.zip
A few things happen during data processing:
- rows of the raw CSV are removed to create a rectangular, single table CSV,
interval-start
andinterval-end
timezone aware datetime columns are added,- when using
nemdata.loader.loader
fortrading-price
, all data is resampled to a 5 minute frequency (both before and after the 30 to 5 minute settlement interval change).