/docker-piaware-to-influx

Pull ADS-B data from dump1090, convert to InfluxDB line protocol, send to InfluxDB

Primary LanguagePython

docker-piaware-to-influx

Pull ADS-B data from dump1090, convert to InfluxDB line protocol, send to InfluxDB

For more information on what PiAware is, see here: https://flightaware.com/adsb/piaware/

Multi Architecture Support

Currently, this image should pull and run on the following architectures:

  • amd64: Linux x86-64
  • arm32v7, armv7l: ARMv7 32-bit (Odroid HC1/HC2/XU4, RPi 2/3/4)
  • aarch64, arm64v8: ARMv8 64-bit (RPi 4)

Supported tags and respective Dockerfiles

  • latest, 2019-09-12
    • latest-amd64, 2019-09-12-amd64 (2019-09-12 branch, Dockerfile)
    • latest-arm32v7, 2019-09-12-arm32v7 (2019-09-12 branch, Dockerfile.armv7l)
    • latest-aarch64, 2019-09-12-aarch64 (2019-09-12 branch, Dockerfile.aarch64)
  • 2018-07-06
    • 2018-07-06-amd64 (2018-07-06 branch, Dockerfile)
    • 2018-07-06-arm32v7 (2018-07-06 branch, Dockerfile)

Change Log

2019-09-12

  • Implement S3 overlay
  • Logging improvements
  • Fixes for issue #1
  • Add support for arm64v8 / aarch64 architecture

2018-07-06

  • Original release supporting amd64 and arm32v7 architectures

Up-and-Running

Firstly, make sure all your hosts (influxdb, piaware/dump1090 and the docker host that will run this container) have their clocks set correctly and are synchronised with NTP.

Next, you can start the container:

docker run \
 -d \
 --name piaware2influxdb \
 --restart=unless-stopped \
 -e INFLUXDB_URL="http://<influxdb_host>:<influxdb_port>" \
 -e DUMP1090_HOST="<dump1090_host>" \
 -e TZ="<your_timezone>" \
 mikenye/piaware-to-influx

For example:

docker run \
  -d \
  --name=piaware2influx \
  --restart=unless-stopped \
  -e INFLUXDB_URL="http://192.168.3.84:8086" \
  -e DUMP1090_HOST="192.168.3.85" \
  -e TZ="Australia/Perth" \
  mikenye/piaware-to-influx

The container will attempt to connect to the dump1090 instance on port 30003 to receive ADS-B data.

It will then convert the data to line protocol, and send to InfluxDB, using database "piaware" (which will be created if it doesn't exist).

Runtime Configuration Options

There are a series of available variables you are required to set:

  • INFLUXDB_URL - The URL of your InfluxDB instance, eg: http://192.168.1.10:8086
  • DUMP1090_HOST - The IP/hostname of your dump1090 instance, eg: 192.168.1.11. No port required, it will use 30003 by default.
  • TZ - Your local timezone, eg Australia/Perth

Ports

Although this container exposes ports (inherited from the telegraf container), none need to be mapped.

It will need to be able to access:

  • Port 30003 TCP on the dump1090 host
  • The InfluxDB server (however you specify in the INFLUXDB_URL environment variable)

State Tracking

As not all messages received contain sufficient data to send to InfluxDB. This program keeps a small state database in memory so it is able to construct a message to send to InfluxDB if insufficient data is received. Also, every message contains the vessel's "ident" information, but not all messages contain the callsign. Once a callsign is received, it is kept in the state tracking database.

To keep the state tracking memory footprint small, and to ensure information is up-to-date, if no messages have been received from a vessel for a period of 15 minutes or more, the vessel is ejected from the state tracking database. For this reason, it is important to have your hosts' clocks synchronised with NTP, and to specify your timezone as shown above.

Telegraf

Telegraf (https://www.influxdata.com/time-series-platform/telegraf/) runs in this container as well. It handles taking the data generated by piaware2influx.py and writing it to InfluxDB. Teleg raf is used because the clever folks at InfluxData are better at writing software that talks to InfluxDB than I am. It handles buffering, it handles InfluxDB temporarily being unavailable, and lots of other nifty features.

It also means that if you'd like a "piaware2kafka" for example, you could simply fork this project and update the telegraf.conf (which is generated via etc/cont-init.d/01-piaware2influx on container start), as telegraf supports several different output plugins. This container just uses outputs.influxdb.

Logging

The container logs quite a lot of information.

  • It will log each message received
  • It has an automatic "back-off" feature, where it will only log a message for a vessel once per minute

Regardless of the back-off feature, it still logs quite a bit of information, so it is strongly advised to set up container log rotation, if you haven't already (see: https://success.docker.com/article/how-to-setup-log-rotation-post-installation).

Log entries look something like this:

2019-09-12 19:26:50.842863500  [RX: 1629, TX: 930] [Ident: 7C146A] Now receiving from this vessel
2019-09-12 19:26:50.842864500  [RX: 1632, TX: 930] [Ident: 8A017C Callsign: AWQ536] Alt: 100
2019-09-12 19:26:50.842865500  [RX: 1635, TX: 931] [Ident: 8A017C Callsign: AWQ536] Alt: 100
2019-09-12 19:26:50.842865500  [RX: 1636, TX: 932] [Ident: 8A017C Callsign: AWQ536] GroundSpeed: 137, Track: 194, VerticalRate: -768
2019-09-12 19:26:50.842866500  [RX: 1638, TX: 933] [Ident: 8A017C Callsign: AWQ536] Alt: 100, Lat: -31.91347, Long: 115.97291
2019-09-12 19:26:50.842867500  [RX: 1639, TX: 934] [Ident: 8A017C Callsign: AWQ536] GroundSpeed: 138, Track: 194, VerticalRate: -768
2019-09-12 19:26:50.842867500  [RX: 1644, TX: 935] [Ident: 8A017C Callsign: AWQ536] Alt: 50
2019-09-12 19:26:50.842868500  [RX: 1644, TX: 936] [Ident: C82762]: Expiring inactive vessel from state database
2019-09-12 19:26:50.842869500  [RX: 1645, TX: 936] [Ident: 8A017C Callsign: AWQ536] Alt: 25
2019-09-12 19:26:50.842869500  [RX: 1646, TX: 937] [Ident: 8A017C Callsign: AWQ536] GroundSpeed: 138, Track: 194, VerticalRate: -640
2019-09-12 19:26:50.842870500  [RX: 1648, TX: 938] [Ident: 8A017C Callsign: AWQ536] Alt: 25, Lat: -31.91666, Long: 115.97201
2019-09-12 19:26:50.842871500  [RX: 1650, TX: 939] [Ident: 8A017C Callsign: AWQ536] GroundSpeed: 138, Track: 194, VerticalRate: -640
2019-09-12 19:26:50.842871500  [RX: 1655, TX: 940] [Ident: 8A017C Callsign: AWQ536] Alt: 0, Lat: -31.91850, Long: 115.97146
2019-09-12 19:26:50.842872500  [RX: 1657, TX: 941] [Ident: 8A017C Callsign: AWQ536] GroundSpeed: 138, Track: 194, VerticalRate: -704
2019-09-12 19:26:50.842873500  [RX: 1662, TX: 942] [Ident: 8A017C Callsign: AWQ536] GroundSpeed: 138, Track: 194, VerticalRate: -704
2019-09-12 19:26:50.842879500  [RX: 1664, TX: 943] [Ident: 8A017C Callsign: AWQ536] Alt: -100, Lat: -31.92352, Long: 115.97000
2019-09-12 19:26:50.842880500  [RX: 1671, TX: 944] [Ident: 7C1ABB] Now receiving from this vessel
2019-09-12 19:26:50.842881500  [RX: 1671, TX: 944] [Ident: 7C8022]: Expiring inactive vessel from state database
2019-09-12 19:26:50.842881500  [RX: 1673, TX: 944] [Ident: 7C146A] GroundSpeed: 134, Track: 239, VerticalRate: -768
2019-09-12 19:26:50.842882500  [RX: 1677, TX: 947] [Ident: 7C146A Callsign: QFA777] GroundSpeed: 134, Track: 240, VerticalRate: -768
2019-09-12 19:26:50.842883500  [RX: 1686, TX: 948] [Ident: 7C146A Callsign: QFA777] Alt: 725, Lat: -31.90593, Long: 116.02734
2019-09-12 19:26:50.842883500  [RX: 1687, TX: 949] [Ident: 7C146A Callsign: QFA777] Alt: 700, Lat: -31.90622, Long: 116.02667
2019-09-12 19:26:50.842884500  [RX: 1690, TX: 950] [Ident: 7C146A Callsign: QFA777] Alt: 675, Lat: -31.90658, Long: 116.02605
2019-09-12 19:26:50.842885500  [RX: 1696, TX: 951] [Ident: 7C146A Callsign: QFA777] GroundSpeed: 134, Track: 240, VerticalRate: -704
2019-09-12 19:26:50.842885500  [RX: 1698, TX: 952] [Ident: 7C146A Callsign: QFA777] GroundSpeed: 135, Track: 239, VerticalRate: -704
2019-09-12 19:26:50.842886500  [RX: 1699, TX: 953] [Ident: 7C146A Callsign: QFA777] GroundSpeed: 135, Track: 239, VerticalRate: -704
2019-09-12 19:26:50.842887500  [RX: 1700, TX: 954] [Ident: 7C146A Callsign: QFA777] GroundSpeed: 135, Track: 239, VerticalRate: -640

As you can see from the output above, logging is broken up into four sections:

[Messages RX/TX] [Vessel Info] Information
  • RX: is the number of messages received from dump1090
  • TX: is the number of messages sent to InfluxDB. Note, this figure will always be smaller than RX:, as not all messages received contain sufficient data to send to InfluxDB (see State Tracking above). Also some messages don't contain usable (for us) data.
  • Vessel Info contains known information about the vessel. Every message contains the vessel's "ident" information, but a callsign will not be displayed until a message is received containing the vessel's call sign (see State Tracking above).
  • Information: This contains information about the vessel - either data received from the vessel or information from this program regarding the vessel.

Telegraf also logs to the container logs, although it is set to "quiet" so you won't see much unless there's a problem. You will likely notice it when the container starts:

2019-09-12 19:08:46.427865500  2019-09-12T11:08:46Z I! Starting Telegraf 1.12.1

Yes, I know there are two timestamps for the telegraf logs, it can't be helped just deal with it. :-)

Visualising the data

Data can be visualised however you like. Personally, I use Grafana.

As an example, adding a table with the following query:

SELECT last("current_altitude") AS "Altitude", last("current_latitude") AS "Lat", last("current_longitude") AS "Long", last("current_groundspeed") AS "Speed", last("current_track") AS "Heading", last("current_verticalrate") AS "VerticalRate" FROM "piaware" WHERE $timeFilter GROUP BY time(5m), "callsign", "squawk", "hexident" fill(none)

Will give a result such as this:

example Grafana table showing PiAware data