Export
Mainflux Export service can send message from one Mainflux cloud to another via MQTT, or it can send messages from edge gateway to Mainflux Cloud.
Export service is subscribed to local message bus and connected to MQTT broker in the cloud.
Messages collected on local message bus are redirected to the cloud.
When connection is lost, messages from local bus are stored into Redis
stream. Upon connection reestablishment Export
service consumes messages from Redis
stream and sends it to the Mainflux cloud.
Install
Get the code:
go get github.com/mainflux/export
cd $GOPATH/github.com/mainflux/export
Make:
make
Usage
cd build
./mainflux-export
Configuration
By default Export
service looks for config file at ../configs/config.toml
if no env vars are specified.
[exp]
cache_pass = ""
cache_url = "localhost:6379"
cache_db = "0"
log_level = "debug"
nats = "localhost:4222"
port = "8170"
[mqtt]
username = "<thing_id>"
password = "<thing_password>"
ca = "ca.crt"
cert = "thing.crt"
mtls = "false"
priv_key = "thing.key"
retain = "false"
skip_tls_ver = "false"
url = "tcp://mainflux.com:1883"
[[routes]]
mqtt_topic = "channel/<channel_id>/messages"
subtopic = "subtopic"
nats_topic = "export"
type = "plain"
workers = 10
Http port
port
- HTTP port where status ofExport
service can be fetched.
curl -X GET http://localhost:8170/version
{"service":"export","version":"0.0.1"}%
Redis connection
To configure Redis
connection settings cache_url
, cache_pass
, cache_db
in config.toml
are used.
MQTT connection
To establish connection to MQTT broker following settings are needed:
username
- Mainflux <thing_id>password
- Mainflux <thing_key>url
- url of MQTT broker
Additionally, you will need MQTT client certificates if you enable mTLS. To obtain certificates ca.crt
, thing.crt
and key thing.key
follow instructions here.
Routes
Routes are being used for specifying which subscriber's topic(subject) goes to which publishing topic.
Currently only MQTT is supported for publishing. To match Mainflux requirements mqtt_topic
must contain channel/<channel_id>/messages
, additional subtopics can be appended.
mqtt_topic
-channel/<channel_id>/messages/<custom_subtopic>
nats_topic
-Export
service will be subscribed to NATS subject<nats_topic>.>
subtopic
- messages will be published to MQTT topic<mqtt_topic>/<subtopic>/<nats_subject>
, where dots in nats_subject are replaced with '/'workers
control number of workers that will be used for message forwarding.type
- specifies message transformation, currently onlyplain
is supported, meaning no transformation.
Before running Export
service edit configs/config.toml
and provide username
, password
and url
username
- matchesthing_id
in Mainflux cloud instancepassword
- matchesthing_key
channel
- MQTT part of the topic where to publish MQTT data (channel/<channel_id>/messages
is format of mainflux MQTT topic) and plays a part in authorization.
In order for Export
service to listen on Mainflux NATS deployed on the same machine NATS port must be exposed. Run mainflux using make run
.
Environment variables
Service will look for config.toml
first and if not found it will be configured with env variables and new config file specified with MF_EXPORT_CONFIG_FILE
will be saved with values populated from env vars.
The service is configured using the environment variables presented in the following table. Note that any unset variables will be replaced with their default values.
Variable | Description | Default |
---|---|---|
MF_NATS_URL | Nats url | localhost:4222 |
MF_EXPORT_MQTT_HOST | Mqtt url where to export | tcp://localhost:1883 |
MF_EXPORT_MQTT_USERNAME | MQTT username, thing id in case of mainflux | |
MF_EXPORT_MQTT_PASSWORD | MQTT password, thing key in case of mainflux | |
MF_EXPORT_MQTT_CHANNEL | MQTT channel where to publish | |
MF_EXPORT_MQTT_SKIP_TLS | Skip tls verification | true |
MF_EXPORT_MQTT_MTLS | Use MTLS for authentication | false |
MF_EXPORT_MQTT_CA | CA for tls | ca.crt |
MF_EXPORT_MQTT_CLIENT_CERT | Client cert for authentication in case when MTLS = true | thing.crt |
MF_EXPORT_MQTT_CLIENT_PK | Client key for authentication in case when MTLS = true | thing.key |
MF_EXPORT_MQTT_QOS | MQTT QOS | 0 |
MF_EXPORT_MQTT_RETAIN | MQTT retain | false |
MF_EXPORT_CONFIG_FILE | Configuration file | config.toml |
for values in environment variables to take effect make sure that there is no MF_EXPORT_CONF
file.
If you run with environment variables you can create config file:
MF_EXPORT_PORT=8178 \
MF_EXPORT_LOG_LEVEL=debug \
MF_EXPORT_MQTT_HOST=tcp://localhost:1883 \
MF_EXPORT_MQTT_USERNAME=<thing_id> \
MF_EXPORT_MQTT_PASSWORD=<thing_key> \
MF_EXPORT_MQTT_CHANNEL=<channel_id> \
MF_EXPORT_MQTT_SKIP_TLS=true \
MF_EXPORT_MQTT_MTLS=false \
MF_EXPORT_MQTT_CA=ca.crt \
MF_EXPORT_MQTT_CLIENT_CERT=thing.crt \
MF_EXPORT_MQTT_CLIENT_PK=thing.key \
MF_EXPORT_CONFIG_FILE=export.toml \
../build/mainflux-export&
Service will be subscribed to NATS <nats_topic>.>
subject and send messages to channels/<MF_EXPORT_MQTT_CHANNEL>/messages
+ /
+ <NatsSubject>
.
For example if you are running Mainflux on a gateway if you set nats_topic="channel"
you can make export
service forward messages to other Mainflux instances i.e. into to the Mainflux cloud.
When message gets published to local Mainflux instance it will end on NATS as channels.<local_channel_id>.messages.subtopic
, Export service will pick it up and forward it to <mqtt_topic>
ending on <mqtt_topic>/channels/<local_channel_id>/messages/subtopic
.
Created export.toml
you can edit to add different routes and use in next run.
How to save config via agent
Configuration file for Export
service can be send over MQTT using Agent service.
save, export,
mosquitto_pub -u <thing_id> -P <thing_key> -t channels/<control_ch_id>/messages/req -h localhost -p 18831 -m "[{\"bn\":\"1:\", \"n\":\"config\", \"vs\":\"save, export, <config_file_path>, <file_content_base64>\"}]"
vs="config_file_path, file_content_base64"
- vs determines where to save file and contains file content in base64 encoding payload:
b,_ := toml.Marshal(export.Config)
payload := base64.StdEncoding.EncodeToString(b)