Dragonfly
A data collector primarily geared towards raspberry pis.
Concepts
There are two main aspects to Dragonfly, the server and its satellites.
The server consists of an Express server and api connected to a postgres database. The server serves a react front end. The main purpose of this server is to collect data from various sources and display it for a user to browse on a 'dashboard', 'station', or 'sensor' basis.
A Satellite polls data from one or more 'sensors' and reports this data to the 'server'. Generally, a Satellite is considered a 'station', and each sensor or data input is called a 'sensor'. 'Sensors' can be hardwired to a station, for example, a temperature sensor connected on a raspberry pis GPIO pins. However, a 'sensor' could also be from a virtual source. For example, the GUI offers manual input of readings - this would also be considered a 'sensor'. You can also add plugins to poll apis, for example there is a crypto poller that finds the latest price of bitcoin.
The GUI has three main views - 'dashboard', 'station', 'sensor'. The 'dashboard' view is a customizable list of graphs and widgets that pops up when the user goes to the main route of Dragonfly. The 'station' view is like the 'dashboard' view in that it's customizable however it will only show data from the particular station that is selected. The 'sensor' view is a single graph detailing the particular sensor that is selected. More in-depth information about a sensor can be found and edited here. Also, 'Actions' can be applied to the particular sensor.
'Actions' are scripts that run when a certain criteria is met. They can be created from the 'sensor' view. Common actions could include:
- Send a slack message when the price of a stock has dropped below a given number
- Turn on a light when the time is 6:00 a.m.
- Turn off a heater when the temperature has gone above 80 degrees.
Development
Quickstart
Run each command in a separate terminal tab
docker-compose up
npm run start
npm run start:satellite
npm run generate:fixtures
Setup
git clone https://github.com/cyrillegin/dragonfly.git
cd dragonfly/
npm i
cp .env-template .env
and replace values with your own postgres values.
To setup postgres, ensure that docker is installed and run docker-compose up
. This will download the postgres image and run with the logs outputting to your console.
Raspberry Pi basic setup
Get the latest pi image from the raspberry pi website and flash it to an ssd. In terminal connected to the pi, run the following commands:
sudo ./utility/setupPi.sh python node
Development
npm run start
will build the front and back end and run them in watch mode. The site will be running at localhost:3000
. From here you can add stations and sensors from the ui. If you would like dummy data added to work with, you can run npm run generate:fixtures
while the server is running. This will create two stations with two sensors each with readings in all of the graphs. A few npm scripts have also been added:
npm run watch:client
- runs only the front end in watch modenpm run watch:server
- runs only the back end in watch modenpm run db:shell
- This will drop you into an interactive postgres shell. (TODO: hardcoded database and user)
Handy postgres commands
These assume you used the npm run db:shell
script and are in the postgres docker container.
\l
- List databases\dt
- list tables\c db
- switch to database 'db'
Server deployment
For deployment instructions, I will be assuming that this is being deployed on a raspberry pi on a local network. I am also assuming that node
, npm
, and postgress
have been installed. For more details on setting up a raspberry pi, see: (TODO: how to set up a raspberry pi)
ssh username@ip
- where username is the name of your user on the pi and ip is its ip address, should look something like:shh pi@192.168.0.2
- Enter password
git clone https://github.com/cyrillegin/dragonfly.git
cd dragonfly
cp .env-template .env
and fill out with pis credentialsnpm i
npm run build:prod
//TODOsudo npm run start:prod
//TODO This will start the the server in production mode on the pi. To access it, navigate to the pis ip from any browser on the same network:http://192.168.0.2
. If you run these commands as is, this will display all of the logs directly to the console that has sshd in. Generally speaking though, you don't care about the console. You can set up dragonfly as a system service (TODO: setup instructions for running as a service.) or as a semi-perminant solution, I like to run the server inscreen
(TODO: explain screen)
Server updates
This is for when dragonfly is already up and running.
ssh username@ip
cd path/to/dragonfly/folder
git pull
npm i
// Only needed if new deps have been added.npm run build:server:prod
// Only needed if there are server changesnpm run build:client:prod
// Only needed if there are client changes
If you run the build:server:prod, you'll need to restart the server.
Database migrations
If there was a change to the database schema, there's a few handy commands included in the package.json file:
npm run db:up
- runs all of the database migration scriptsnpm run db:down
- undos the migration scriptsnpm run db:status
- checks to see if you need to run any migrations
Satellite deployment
Satellites can be deployed anywhere, including where the server is deployed, aws (TODO: prove), personal computers, or other raspberry pis. For these instructions, I will assume that this is a different raspberry pi being accessed via ssh from a personal computer.
ssh username@ip
- where username is the name of your user on the pi and ip is its ip address, should look something like:shh pi@192.168.0.2
git clone https://github.com/cyrillegin/dragonfly.git
cd dragonfly/satellite
sudo ./setup.sh
(TODO)python __main__.py
This will start the python server and print out logs to the console. Like the server above, I suggest running this as a service or in screen. There are some sensors that require additional setup, this will be detailed in the Sensors section.
Satellite updates
ssh username@ip
cd path/to/dragonfly/folder
git pull
That should be it. The cherrypy server is setup up to auto restart on code changes. That said, make sure you check your log outputs because some changes could either not be picked up or stall out the server. To be extra sure, you can always restart.
Sensors
GPIO
Once the poller has been wired to the pi, you will need to activate it and get its device id. Run sudo dtoverlay w1-gpio gpiopin=17 pullup=0
, changing the gpiopin to whatever you have yours plugged into (run this command multiple times if you have multiple sensors plugged in).
To list the active devices, run ls /sys/bus/w1/devices/
You should see listed devices that look like 28-0416a47a0aff
. These will be added to the metadata field when adding the sensor in the UI.
Method 2:
Ensure dtoverlay=w1-gpio
is in your /boot/config.txt
. Restart the pi with sudo reboot now
.
Enter: sudo modprobe w1-gpio
and sudo modprobe w1-therm
. Now when you ls /sys/bus/w1/devices/
, your device id should appear.
BMP 180
For this sensor, we'll need to update the boot/config.txt.
sudo vim /boot/config.txt
- find the line with
dtparam=i2c_arm
and make sure that it isn't commented out. (This may require a reboot)
DHT 11
No setup required other than wiring