Pulling current charging station data

[sjdonado]

Create a service which pulls current charging station data from OpenChargeMap.

The following fields will be imported:

1. operatorInfo
2. statusType
3. addressInfo
4. Connections

Constrains
maxResults is defined by default to 100. In average it takes less than 1000ms to fetch the first 100 items:

But it has no pagination:

To retrieve all the data, we can either:
a. to increase maxResults, which increases the response time proportionally.
b. to make multiple requests.

Option (a) doesn't scale well, it can lead to timeouts.
Option (b) can be performed using queuing mechanisms, executing multiple requests in parallel, but this may result in rate limits.

Expanding option (b), we can use some query parameters like: countryid, dataproviderid or operatorid to segment the data. The more parameters used in the requests, the smaller the chunks.

Good news is that those ids can be fetched beforehand, with one request to https://api.openchargemap.io/v3/referencedata.

Approach: Option (b) creating jobs (rabbitMQ queue) per each operatorid, this thanks to the relationship: one operator belongs to a country. Requests can be filtered by allowed countries, sending then only the operatorid that belongs to them.

To research

1. OpenChargeMap Rate limits.
2. Pulling frequency. To avoid oversaturate Mongo with write operations.

An example

Acceptance Criteria

1. Pull the data in small chunks per countries + data providers.
2. Store the data in the DB: POI list.
3. Local setup with docker-compose: MongoDB + RabbitMQ.
4. Unit + integration tests.