Mender: Integration

Mender is an open source over-the-air (OTA) software updater for embedded Linux devices. Mender comprises a client running at the embedded device, as well as a server that manages deployments across many devices.

This repository contains a Docker-based environment allowing to run all Mender backend services as a single system. Each service has a dedicated Dockerhub repository, where tagged Docker builds are stored. Images are pulled and started in a coordinated fashion via docker-compose and the associated docker-compose.yml file.

Requirements:

docker-engine 1.10
docker-compose 1.7

Getting started

To start using Mender, we recommend that you begin with the Getting started section in the Mender documentation.

Services

The integration environment brings together the following services:

Mender Device Authentication Service
Mender Deployment Service
Mender Device Inventory Service
Mender User Administration Service
Mender API Gateway
Minio object storage
Storage service proxy based on OpenResty

How to use in production

A provided docker-compose.yml file will provision the following set of services:

       |
  port |        +-----------------------+         +------------------------+
   443 | <----> |  API Gateway          |    +--->|  Device Authentication |
       |        |  (mender-api-gateway) |<---|    |  (mender-device-auth)  | <---+
       |        +-----------------------+    |    +------------------------+     |
       |                                     +--->|  Inventory             |     |
       |                                     |    |  (mender-inventory)    | <---+
       |                                     |    +------------------------+     |     +----------------------------------+
       |                                     +--->|  User Administration   |     +---> |  Workflows Engine                |
       |                                     |    |  (mender-useradm)      | <---+     |  (mender-workflows-server)       |
       |                                     |    +------------------------+     |     |  (mender-workflows-worker)       |
       |                                     +--->|  Deployments           |     |     |  (mender-create-artifact-worker) |
       |                 +----------------------->|  (mender-deployments)  | <---+     +----------------------------------+
       |                 |                        +------------------------+
       |                 |
       |                 v
  port |        +------------------+              +----------+
  9000 | <----> |  Storage Proxy   |<------------>|  Minio   |
       |        |  (storage-proxy) |              |  (minio) |
       |        +------------------+              +----------+
       |

It is customary to provide deployment specific overrides in a separate compose file. This can either be docker-compose.override.yml file (detected and included automatically by docker-compose command) or a separate file. If a separate file is used, it needs to be explicitly included in command line when running docker-compose like this:

docker-compose -f docker-compose.yml -f my-other-file.yml up

Mender artifacts file are served from storage backend provided by Minio object storage in the reference setup.

A demo setup uses docker-compose.demo.yml overlay file to override different aspects of configuration and can be used as an example when deploying to production.

For details on configuration and administration consult Administration guide in Mender documentation.

Integrating a new service

Adding a new service to the setup involves:

creating a dedicated Dockerfile
setting up a Dockerhub repository and a CI build pipeline
adding the service's config under docker-compose.yml

Guidelines and things to consider:

assign the service a name that is unique within docker-compose.yml
add the service to the mender network
setup the correct routing and authentication for the new service in the Mender API Gateway
extend the common service mender-common/common.yml

How to use in development

Running the integration setup brings up a complete system of services; as such it can readily be used for testing, staging or demo environments.

In development scenarios however some additional strategies apply, described in the following sections.

Developing a new service

The default approach to integrating a service, involving the full build pipeline, is not conducive to quick develop/build/test cycles. Therefore, when prototyping a new service against an existing system, it can be useful to:

create a dedicated Dockerfile for your service and build it locally:

cd FOLDER_WITH_DOCKERFILE
docker build -t MY_DOCKER_TAG  .

include the service as usual in docker-compose.yml, paying attention to the image tag you just created:

    #
    # myservice
    #
    myservice:
        image: MY_DOCKER_TAG

add any number of volumes to your service, to mount your local binaries and config files into the Docker container, e.g.:

    myservice:
        ...
        volumes:
             /some/localhost/folder/myconfig.yaml:/usr/bin/myconfig.yaml
             /some/localhost/folder/mybinary:/usr/bin/mybinary
            ...

When you run the setup, your new service will be a part of it; also, it will be running binaries from your local machine, which means you can quickly recompile them and restart integration for changes to take effect.

Note that the correct routing and auth still have to be set up in the Mender API Gateway for the service to be accessible from the outside. To experiment with new configuration:

copy the Gateway's main config file locally
in docker-compose.yml, again mount your local version inside the Gateway container:

    #
    # mender-api-gateway
    #
    mender-api-gateway:
        ...
        /some/localhost/folder/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf

Your changes will take effect when you restart the whole setup.

Troubleshooting/developing an existing service

For troubleshooting and debugging, a similar approach involving Docker volumes can be used. Assuming that a given service's image has been pulled to your local machine, mount your local binaries and config files via docker-compose.yml:

    service:
        ...
        volumes:
             /some/localhost/folder/myconfig.yaml:/usr/bin/config.yaml
             /some/localhost/folder/mybinary:/usr/bin/service-binary
            ...

To obtain the locations of both binaries and config files, refer the service's dedicated Dcokerfile.

Again, recompiling your local binary and restarting integration will make your changes take effect. Note that the correct API Gateway config is probably already set up for an existing service; if not, refer the previous section on how to modify it.

Enabling non-SSL access

For debugging purposes or when using third party SSL reverse proxy, it may be useful to enable non-SSL access.
API Gateway configuration enables plain HTTP on port 80 when setting the SSL environment variable to 'false'.
The nginx configuration will only be changed on container creation. If you previously ran with SSL, delete and re-create the container.
An example compose file can be included like this:

./demo -f docker-compose.no-ssl.yml up

NOTE make sure that plain HTTP port is not published in production deployment. Use a reverse proxy for example.

Demo client

The setup comes with a predefined client service (mender-client) that runs a qemu VM in a container. The client will connect to the backend by accessing docker.mender.io host (an alias assigned to mender-api-gateway service). The client container will not be started by default and needs to be included explicitly when running docker compose by listing multiple compose files as described in compose manual.

To start the backend and a demo client run the following command:

docker-compose -f docker-compose.yml -f docker-compose.client.yml up

Contributing

We welcome and ask for your contribution. If you would like to contribute to Mender, please read our guide on how to best get started contributing code or documentation.

License

Mender is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Security disclosure

We take security very seriously. If you come across any issue regarding security, please disclose the information by sending an email to security@mender.io. Please do not create a new public issue. We thank you in advance for your cooperation.

Connect with us

Join the Mender Hub discussion forum
Follow us on Twitter. Please feel free to tweet us questions.
Fork us on Github
Create an issue in the bugtracker
Email us at contact@mender.io
Connect to the #mender IRC channel on Freenode

Authors

Mender was created by the team at Northern.tech AS, with many contributions from the community. Thanks everyone!

Mender is sponsored by Northern.tech AS.

hunasdf/integration