Finishing MQTT integration

[janecker]

I wanted to start a small discussion how the MQTT integration could completed. For receiving inputs through docker, the mqtt_input script has to run in the background, which means an additional process is required. I am quite new to docker but as far as I understand it best practice would be to have a single container for each service/process. I could think of using multiple approaches for integrating the MQTT service:

1. A new container could be created that is added to the docker-compose. This would require an additional repository with its own Dockerfile. Additionally the emoncms files have to be cloned again or the files could be moved to there own volume that is re-used in the new mqtt service container.
2. The mqtt process could somehow be started in its own process on the startup of the emoncms web container in the background. This is a simple but very dirty way of doing it.
3. Maybe supervisord could be used to start multiple services (apache and the mqtt service). I am not 100% sure yet how this is done when the php-apache base image is used. This would not require an additional repository and the emoncms files could stay where they are.

I think it would be good if it would be possible to start the mqtt service only if mqtt is also enabled in the settings. I am not sure if this is possible with all three proposed variants. Any opinions on my proposed variants? Are there other, maybe better way to solve this issue?

[glynhudson]

as I understand it best practice would be to have a single container for each service/process

Yes, this is also what I understand to be best practice.

new container could be created that is added to the docker-compose. This would require an additional repository with its own Dockerfile

No, additional repository would not be required. Just a sub-folder in this repo. We already use docker-compose to bring up multiple containers e.g. web, db, and redis. Its just that for db and redis we just the off-the-shelf image without any customisation therefore dockerfile is not needed. See: https://nickjanetakis.com/blog/docker-tip-10-project-structure-with-multiple-dockerfiles-and-docker-compose

I think creating another php (not php-apache) container is the best way to run the MQTT service. The user could then choose to start this container or not depending if MQTT input is required. Although it would't use much resources.

[janecker]

Good point, did not think about that. That could be good solution and I played around with an new php 7 based container running the mqtt script. That seems to be working. I will try to polish this version in the upcoming days.

[mattiarossi]

Hi, I am picking up this enhancement request, as I am redeploying my monolithic emoncms install and would like to move it to docker composer.
I am ok with building an additional mqtt container if needed, but I am not sure how to tackle the phpmqtt_input task.
In theory it should go into another container, using php-cli as base image, but it would need to share the data volumes with the main web container for access to the timeseries data, and the locking could break when the underlying volumes are mounted on an nfs share (guess, my case)
Another possibility would be to spawn it in the existing web container, but it would break the 'one purpose per container' pattern
I think I will go the separate container, shared volumes way, but comments are welcome
Regards,
Mattia

[glynhudson]

Hi Mattia,

Thanks for your comment. After considering this issue we have created an all-in-on Emoncms docker based on the ubuntu image. This simplifies everything, even though maybe this is not docker best practice I think it's the way forward for deployment. Here is a development repo, it's not yet on docker hub. Please let us know how you get on with it: https://github.com/TrystanLea/emoncms-docker

[mattiarossi]

Thanks Glyn,
I agree the monolithic solution simplifies things, that's where I am coming from, a single container with emoncms, mosquitto and redis, I don't think I want to keep going that way, for me emoncms is part of a broader home automation setup that includes openhab,node-red,influxdb, grafana, you name it ... and I am reworking all of it to use docker/docker-compose. Do I have gone the separate container way with shared volumes (mindful of all risks involved in flock() potentially misbehaving when accessing time series data).

[haniham]

Hey guys - nearly 1 Year has passed - have you meanwhile figured out a solution, cause i would like to use it (-;

[JumpMaster]

I'm personally running emoncms in 3 (or maybe 4) docker containers.

• emoncms web app
• mqtt server (shared with other system so not part of emoncms)
• MariaDB
• mqtt listener

I also followed the mantra of one process per container. It's a shame that both the mqtt listener and emoncms server both have a full copy of the code from GitHub.

I'll upload my code to GitHub later today.

It's a shame

[TrystanLea]

I've got this working here https://github.com/emoncms/emoncms-docker/pull/27/files
Does that match your approach @JumpMaster ? any tips to improve?

[jamesfidell]

I came to the conclusion that the web app and the script to subscript/publish to MQTT are so tightly coupled in terms of shared code and configuration that there's a fair case to be made for running them in the same container. I have therefore gone the route of using supervisord to start both Apache and the script. It seems to work fine so far, modulo the issues with the script and Redis.

[TrystanLea]

Thanks @jamesfidell I'd be interested to hear more about your solution, would you be able to make a pull request with your implementation?

[jamesfidell]

Sure, I'll try to sort something out. The files are in a bit of a mess at the moment as whilst I am using a Docker container for emoncms, I'm not using the database container (already have a database running elsewhere). I'll clone a fresh copy and tidy it all up for a PR once I have my new templates working.