EVE Online market data trawler and EMDR uploader.
We recommend using virtualenv
to help manage Python dependencies.
Then from your Git checkout:
virtualenv venv
source venv/bin/activate
python setup.py install
Dependencies will be installed to the virtualenv environment automatically.
You may need the libpq-dev
or libpq-devel
package installed in order for
Python to build some of the dependencies.
Simple case:
venv/bin/CRESTMarketTrawler
Or from within the virtualenv environment, just run CRESTMarketTrawler
.
The trawler will automatically ensure that the CREST API's cache timers are obeyed.
You can further restrict the trawl speed by setting the REQUESTS_PER_SECOND
environment
variable. This will limit the maximum number of requests to CREST per second; setting it
to smaller fractional values will introduce larger pauses between requests.
CREST (and ESI) do not include solarSystemID
in the data feed, but many EMDR
clients rely on its presence. CRESTMarketTrawler includes a location service
that uses the Static Data Export to provide orders with solarSystemID
s. If
the ESI_CLIENT_ID
, ESI_SECRET
and ESI_REFRESH_TOKEN
environment variables
are set, it will also call ESI to obtain solarSystemID
s for player structures
(e.g. citadels).
To obtain a client ID and secret, register an application at
https://developers.eveonline.com. Authenticate a character granting the
esi-universe.read_structures.v1
scope to obtain a refresh token.
To enable writing orders to a Postgres database, you should set the POSTGRES_USERNAME
,
POSTGRES_PASSWORD
, POSTGRES_HOST
and POSTGRES_DB
environment variables. At the
moment, the destination must be a table called live_orders
and conform to the following
schema:
CREATE TABLE live_orders (
orderid bigint NOT NULL,
typeid bigint,
regionid bigint,
price numeric,
volremaining integer,
range smallint,
volentered integer,
minvolume integer,
isbid boolean,
issuedate timestamp without time zone,
duration smallint,
stationid bigint,
solarsystemid bigint,
expiry timestamp without time zone
);
To disable uploading to EMDR, set the DISABLE_EMDR
environment variable to 1
.
Dockerfiles are provided for both the trawler and two ancillary services: a database and web API frontend.
Settings for the database are stored in the docker.env
file.
Running docker-compose up
in the project root directory might not work since
the database takes some time to spool up; so do docker-compose up -d db
and
wait a moment before starting web
and trawler
.