An Ubuntu based Mesos Slave container, packaged with Logstash-Forwarder and managed by Supervisord. All parameters are controlled through environment variables, with some settings auto-configured based on the environment.
- Container Release: 1.2.0
- Mesos: 0.26.0-0.2.145.ubuntu1404
- Docker: 1.9.1-0~trusty
Services Include:
- Mesos Slave - Primary process that offers resources of the host to the Mesos Master(s) for scheduling and running of tasks.
- Consul-Template - An application that can populate configs from a consul service.
- Logrotate - A script and application that aid in pruning log files.
- Logstash-Forwarder - A lightweight log collector and shipper for use with Logstash.
- Redpill - A bash script and healthcheck for supervisord managed services. It is capable of running cleanup scripts that should be executed upon container termination.
- Rsyslog - The system logging daemon.
- Usage
- Example Run Command
- Modification and Anatomy of the Project
- Important Environment Variables
- Service Configuration
- Mesos
- Consul-Template
- Logrotate
- Logstash-Forwarder
- Redpill
- Rsyslog
- Troubleshooting
All mesos commands should be passed via environment variables (please see the example run command below). For Mesos documentation, please see the configuration docs associated with the release here: mesos@d3717e5
In a local proof of concept environment, the only variable that MUST be defined is MESOS_MASTER
.
However, that will leave the slave with a fraction of it's functionality. To run in a useful fashion, the following should be set, ENVIRONMENT
, MESOS_MASTER
, MESOS_WORK_DIR
, and MESOS_DOCKER_SANDBOX_DIRECTORY
.
-
ENVIRONMENT
- when set toproduction
ordevelopment
it will enable all services including:mesos-master
,logstash-forwarder
, andredpill
. -
MESOS_MASTER
- Informs the slave how to connect or discover the Mesos Masters. Please see the Mesos docs for the available options. -
MESOS_WORK_DIR
- Path to the directory in which framework directories are placed. -
MESOS_SANDBOX_DIRECTORY
- Path to directory used to map the sandbox to Docker containers.
In addition to the above, there are several things to note for full compatibility when operating a mesos slave in a container with docker as a supported containerizer.
- The container should be run with
host
networking. - The container requires several volumes to be mounted. For compatibility purposes these should be the same location as it is on the host.
/usr/bin/docker:/usr/bin/docker:ro
/var/run/docker.sock:/var/run/docker.sock:rw
/sys:/sys:ro
- The directory used for
MESOS_WORK_DIR
asrw
- The directory used for
MESOS_SANDBOX_DIRECTORY
asrw
Configuring private registry access is dependant on several factors. For documentation, please visit the Marathon Framework site directly.
In either case, if you intend on baking the credentials into the image. This would be the image to do it.
docker run -d --net=host \
--pid=host \
--name=mesosslave \
--cap-add=SYS_ADMIN \
-e ENVIRONMENT=production \
-e PARENT_HOST=$(hostname) \
-e MESOS_IP=10.10.0.111 \
-e MESOS_MASTER=zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos \
-e MESOS_REGISTRATION_TIMEOUT=5min \
-e MESOS_CONTAINERIZERS=docker,mesos \
-e MESOS_HOSTNAME=10.10.0.111 \
-e MESOS_SANDBOX_DIRECTORY=/data/mesos/sandbox \
-e MESOS_WORKDIR=/data/mesos/workdir \
-v /data/mesos/workdir:/data/mesos/workdir:rw \
-v /data/mesos/sandbox:/data/mesos/sandbox:rw \
-v /usr/bin/docker:/usr/bin/docker:ro \
-v /var/run/docker.sock:/var/run/docker.sock:rw \
-v /sys/:/sys:ro \
mesos-slave
File Structure
The directory skel
in the project root maps to the root of the file system once the container is built. Files and folders placed there will map to their corresponding location within the container.
Init
The init script (./init.sh
) found at the root of the directory is the entry process for the container. It's role is to simply set specific environment variables and modify any subsequently required configuration files.
Supervisord
All supervisord configs can be found in /etc/supervisor/conf.d/
. Services by default will redirect their stdout to /dev/fd/1
and stderr to /dev/fd/2
allowing for service's console output to be displayed. Most applications can log to both stdout and their respectively specified log file.
In some cases (such as with zookeeper), it is possible to specify different logging levels and formats for each location.
Logstash-Forwarder
The Logstash-Forwarder binary and default configuration file can be found in /skel/opt/logstash-forwarder
. It is ideal to bake the Logstash Server certificate into the base container at this location. If the certificate is called logstash-forwarder.crt
, the default supplied Logstash-Forwarder config should not need to be modified, and the server setting may be passed through the SERVICE_LOGSTASH_FORWARDER_ADDRESS
environment variable.
In practice, the supplied Logstash-Forwarder config should be used as an example to produce one tailored to each deployment.
Variable | Default |
---|---|
ENVIRONMENT_INIT |
|
APP_NAME |
mesos-slave |
ENVIRONMENT |
local |
PARENT_HOST |
unknown |
MESOS_LOG_DIR |
/var/log/mesos |
MESOS_WORK_DIR |
|
GLOG_max_log_size |
|
SERVICE_CONSUL_TEMPLATE |
disabled |
SERVICE_LOGROTATE |
|
SERVICE_LOGROTATE_INTERVAL |
3600 (set in script by default) |
SERVICE_LOGROTATE_SCRIPT |
/opt/scripts/purge-mesos-logs.sh |
SERVICE_LOGSTASH_FORWARDER |
|
SERVICE_LOGSTASH_FORWARDER_CONF |
/opt/logstash-forwarder/mesos-slave.conf |
SERVICE_REDPILL |
|
SERVICE_REDPILL_MONITOR |
mesos |
SERVICE_RSYSLOG |
disabled |
-
ENVIRONMENT_INIT
- If set, and the file path is valid. This will be sourced and executed before ANYTHING else. Useful if supplying an environment file or need to query a service such as consul to populate other variables. -
APP_NAME
- A brief description of the container. If Logstash-Forwarder is enabled, this will populate theapp_name
field in the Logstash-Forwarder configuration file. -
ENVIRONMENT
- Sets defaults for several other variables based on the current running environment. Please see the environment section for further information. If logstash-forwarder is enabled, this value will populate theenvironment
field in the logstash-forwarder configuration file. -
PARENT_HOST
- The name of the parent host. If Logstash-Forwarder is enabled, this will populate theparent_host
field in the Logstash-Forwarder configuration file. -
MESOS_LOG_DIR
- The path to the directory in which Mesos stores its logs. -
MESOS_WORK_DIR
- Path to the directory in which framework directories are placed. -
GLOG_max_file_size
- The size in Megabytes that the mesos log file(s) will be allowed to grow to before rotation. -
SERVICE_CONSUL_TEMPLATE - *
SERVICE_CONSUL_TEMPLATE- Enables or disables the consul-template service. If enabled, it will also enable
SERVICE_LOGROTATEand
SERVICE_RSYSLOGto handle logging. (**Options:**
enabledor
disabled`) -
SERVICE_LOGROTATE
- Enables or disabled the Logrotate service. This will be set automatically depending on the environment. (Options:enabled
ordisabled
) -
SERVICE_LOGROTATE_INTERVAL
- The time in seconds between runs of logrotate or the logrotate script. The default (3600 or 1 hour) is set by default in the logrotate script automatically. -
SERVICE_LOGROTATE_SCRIPT
- The path to the script that should be executed instead of logrotate itself to clean up logs. -
SERVICE_LOGSTASH_FORWARDER
- Enables or disables the Logstash-Forwarder service. Set automatically depending on theENVIRONMENT
. See the Environment section below. (Options:enabled
ordisabled
) -
SERVICE_LOGSTASH_FORWARDER_CONF
- The path to the logstash-forwarder configuration. -
SERVICE_REDPILL
- Enables or disables the Redpill service. Set automatically depending on theENVIRONMENT
. See the Environment section below. (Options:enabled
ordisabled
) -
SERVICE_REDPILL_MONITOR
- The name of the supervisord service(s) that the Redpill service check script should monitor. -
SERVICE_RSYSLOG
- Enables of disables the rsyslog service. This is managed bySERVICE_CONSUL_TEMPLATE
, but can be enabled/disabled manually. (Options:enabled
ordisabled
)
Environment
local
(default)
Variable | Default |
---|---|
MESOS_HOSTNAME |
<first ip bound to eth0> |
GLOG_max_log_size |
10 |
SERVICE_LOGROATE |
enabled |
SERVICE_LOGSTASH_FORWARDER |
disabled |
SERVICE_REDPILL |
enabled |
MESOS_WORK_DIR |
/var/lib/mesos |
prod
|production
|dev
|development
Variable | Default |
---|---|
GLOG_max_log_size |
10 |
SERVICE_LOGROATE |
enabled |
SERVICE_LOGSTASH_FORWARDER |
enabled |
SERVICE_REDPILL |
enabled |
debug
Variable | Default |
---|---|
SERVICE_LOGROTATE |
disabled |
SERVICE_LOGSTASH_FORWARDER |
disabled |
SERVICE_REDPILL |
disabled |
CONSUL_TEMPLATE_LOG_LEVEL |
debug * |
* Only set if SERVICE_CONSUL_TEMPLATE
is set to enabled
.
As stated in the Usage section, Mesos-slave configuration information can be found in the github docs releated to the Mesos Release: mesos@d3717e5.
The actual mesos start command is passed to supervisor via the SERVICE_MESOS_CMD
environment variable, and defaults to mesos-slave
.
Variable | Default |
---|---|
MESOS_CONTAINERIZERS |
docker,mesos |
MESOS_LOG_DIR |
/var/log/mesos |
MESOS_MASTER |
|
MESOS_SANDBOX_DIRECTORY |
/mnt/mesos/sandbox |
MESOS_WORK_DIR |
|
SERVICE_MESOS_CMD |
mesos-slave |
-
MESOS_CONTAINERIZES
- Comma separated list of containerizers for use with Mesos. Priority is assigned in the order in which they're passed. -
MESOS_LOG_DIR
- The path to the directory in which Mesos stores its logs. -
MESOS_WORK_DIR
- Path to the directory in which framework directories are placed. -
SERVICE_MESOS_CMD
- The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.
Provides initial configuration of consul-template. Variables prefixed with CONSUL_TEMPLATE_
will automatically be passed to the consul-template service at runtime, e.g. CONSUL_TEMPLATE_SSL_CA_CERT=/etc/consul/certs/ca.crt
becomes -ssl-ca-cert="/etc/consul/certs/ca.crt"
. If managing the application configuration is handled via file configs, no other variables must be passed at runtime.
Variable | Default |
---|---|
CONSUL_TEMPLATE_CONFIG |
/etc/consul/template/conf.d |
CONSUL_TEMPLATE_SYSLOG |
true |
SERVICE_CONSUL_TEMPLATE |
|
SERVICE_CONSUL_TEMPLATE_CMD |
consul-template <CONSUL_TEMPLATE_*> |
The logrotate script is a small simple script that will either call and execute logrotate on a given interval; or execute a supplied script. This is useful for applications that do not perform their own log cleanup.
Variable | Default |
---|---|
SERVICE_LOGROTATE |
|
SERVICE_LOGROTATE_INTERVAL |
3600 |
SERVICE_LOGROTATE_CONFIG |
/etc/logrotate.conf |
SERVICE_LOGROTATE_SCRIPT |
/opt/scripts/purge-mesos-logs.sh |
SERVICE_LOGROTATE_FORCE |
|
SERVICE_LOGROTATE_VERBOSE |
|
SERVICE_LOGROTATE_DEBUG |
|
SERVICE_LOGROTATE_CMD |
/opt/script/logrotate.sh <flags> |
-
SERVICE_LOGROTATE
- Enables or disables the Logrotate service. Set automatically depending on theENVIRONMENT
. See the Environment section. (Options:enabled
ordisabled
) -
SERVICE_LOGROTATE_INTERVAL
- The time in seconds between run of either the logrotate command or the provided logrotate script. Default is set to3600
or 1 hour in the script itself. -
SERVICE_LOGROTATE_CONFIG
- The path to the logrotate config file. If neither config or script is provided, it will default to/etc/logrotate.conf
. -
SERVICE_LOGROTATE_SCRIPT
- A script that should be executed on the provided interval. Useful to do cleanup of logs for applications that already handle rotation, or if additional processing is required. -
SERVICE_LOGROTATE_FORCE
- If present, passes the 'force' command to logrotate. Will be ignored if a script is provided. -
SERVICE_LOGROTATE_VERBOSE
- If present, passes the 'verbose' command to logrotate. Will be ignored if a script is provided. -
SERVICE_LOGROTATE_DEBUG
- If present, passed the 'debug' command to logrotate. Will be ignored if a script is provided. -
SERVICE_LOGROTATE_CMD
- The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.
root@ec58ca7459cb:/opt/scripts# ./logrotate.sh --help
logrotate.sh - Small wrapper script for logrotate.
-i | --interval The interval in seconds that logrotate should run.
-c | --config Path to the logrotate config.
-s | --script A script to be executed in place of logrotate.
-f | --force Forces log rotation.
-v | --verbose Display verbose output.
-d | --debug Enable debugging, and implies verbose output. No state file changes.
-h | --help This usage text.
The below cleanup script will remove all but the latest 5 rotated logs.
#!/bin/bash
mld=${MESOS_LOG_DIR:-/var/log/mesos}
cd "$mld"
(ls -t | grep 'log.INFO.*'|head -n 5;ls)|sort|uniq -u|grep 'log.INFO.*'|xargs --no-run-if-empty rm
(ls -t | grep 'log.ERROR.*'|head -n 5;ls)|sort|uniq -u|grep 'log.ERROR.*'|xargs --no-run-if-empty rm
(ls -t | grep 'log.WARNING.*'|head -n 5;ls)|sort|uniq -u|grep 'log.WARNING.*'|xargs --no-run-if-empty rm
#consul-template uses rsyslog for logging, need to run logrotate to handle that log
if [[ "$SERVICE_CONSUL_TEMPLATE" == "enabled" ]]; then
/usr/sbin/logrotate /etc/logrotate.conf
fi
Logstash-Forwarder is a lightweight application that collects and forwards logs to a logstash server endpoint for further processing. For more information see the Logstash-Forwarder project.
Variable | Default |
---|---|
SERVICE_LOGSTASH_FORWARDER |
|
SERVICE_LOGSTASH_FORWARDER_CONF |
/opt/logstash-forwarder/mesos-slave.conf |
SERVICE_LOGSTASH_FORWARDER_ADDRESS |
|
SERVICE_LOGSTASH_FORWARDER_CERT |
|
SERVICE_LOGSTASH_FORWARDER_CMD |
/opt/logstash-forwarder/logstash-fowarder -config="${SERVICE_LOGSTASH_FOWARDER_CONF}" |
-
SERVICE_LOGSTASH_FORWARDER
- Enables or disables the Logstash-Forwarder service. Set automatically depending on theENVIRONMENT
. See the Environment section. (Options:enabled
ordisabled
) -
SERVICE_LOGSTASH_FORWARDER_CONF
- The path to the logstash-forwarder configuration. -
SERVICE_LOGSTASH_FORWARDER_ADDRESS
- The address of the Logstash server. -
SERVICE_LOGSTASH_FORWARDER_CERT
- The path to the Logstash-Forwarder server certificate. -
SERVICE_LOGSTASH_FORWARDER_CMD
- The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.
Rsyslog is a high performance log processing daemon. For any modifications to the config, it is best to edit the rsyslog configs directly (/etc/rsyslog.conf
and /etc/rsyslog.d/*
).
Variable | Default |
---|---|
SERVICE_RSYSLOG |
disabled |
SERVICE_RSYSLOG_CONF |
/etc/rsyslog.conf |
SERVICE_RSYSLOG_CMD |
/usr/sbin/rsyslogd -n -f $SERVICE_RSYSLOG_CONF |
-
SERVICE_RSYSLOG
- Enables or disables the rsyslog service. This will automatically be set depending on what other services are enabled. (Options:enabled
ordisabled
) -
SERVICE_RSYSLOG_CONF
- The path to the rsyslog configuration file. -
SERVICE_RSYSLOG_CMD
- The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.
Redpill is a small script that performs status checks on services managed through supervisor. In the event of a failed service (FATAL) Redpill optionally runs a cleanup script and then terminates the parent supervisor process.
Variable | Default |
---|---|
SERVICE_REDPILL |
|
SERVICE_REDPILL_MONITOR |
mesos |
SERVICE_REDPILL_INTERVAL |
|
SERVICE_REDPILL_CLEANUP |
|
SERVICE_REDPILL_CMD |
-
SERVICE_REDPILL
- Enables or disables the Redpill service. Set automatically depending on theENVIRONMENT
. See the Environment section. (Options:enabled
ordisabled
) -
SERVICE_REDPILL_MONITOR
- The name of the supervisord service(s) that the Redpill service check script should monitor. -
SERVICE_REDPILL_INTERVAL
- The interval in which Redpill polls supervisor for status checks. (Default for the script is 30 seconds) -
SERVICE_REDPILL_CLEANUP
- The path to the script that will be executed upon container termination. -
SERVICE_REDPILL_CMD
- The command that is passed to supervisor. It is dynamically built from the other redpill variables. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.
root@c90c98ae31e1:/# /opt/scripts/redpill.sh --help
Redpill - Supervisor status monitor. Terminates the supervisor process if any specified service enters a FATAL state.
-c | --cleanup Optional path to cleanup script that should be executed upon exit.
-h | --help This help text.
-i | --interval Optional interval at which the service check is performed in seconds. (Default: 30)
-s | --service A comma delimited list of the supervisor service names that should be monitored.
In the event of an issue, the ENVIRONMENT
variable can be set to debug
. This will stop the container from shipping logs and prevent it from terminating if one of the services enters a failed state.
For mesos itself, the MESOS_LOGGING_LEVEL
variable can be set to INFO
or WARNING
to further diagnose the problem.
If further information is needed; logging may be controlled directly by configuring glog loggig library used by Mesos. For reference; please see the table below:
Note: not all of the glog settings are overridable. Mesos does dictate some of them. Main ones of note are GLOG_v
to increase log verbosity, and GLOG_max_log_size
if log growth and rotation must be tuned.
Environment Variable Name | Type | Default | Description |
---|---|---|---|
GLOG_logtostderr |
bool |
FALSE |
Log messages go to stderr instead of logfiles. |
GLOG_alsologtostderr |
bool |
FALSE |
Log messages go to stderr in addition to logfiles. |
GLOG_colorlogtostderr |
bool |
FALSE |
Color messages logged to stderr (if supported by terminal). |
GLOG_stderrthreshold |
int |
2 |
Log messages at or above this level are copied to stderr in addition to logfiles. This flag obsoletes –alsologtostderr. |
GLOG_alsologtomail |
string |
Log messages go to these email addresses in addition to logfiles. | |
GLOG_logmaillevel |
int |
999 |
Email log messages logged at this level or higher (0 means email all; 3 means email FATAL only ...) |
GLOG_logmailer |
string |
/bin/mail |
Mailer used to send logging email. |
GLOG_drop_log_memory |
bool |
TRUE |
Drop in-memory buffers of log contents. Logs can grow very quickly and they are rarely read before they need to be evicted from memory. Instead, drop them from memory as soon as they are flushed to disk. |
GLOG_log_prefix |
bool |
TRUE |
Prepend the log prefix to the start of each log line. |
GLOG_minloglevel |
int |
0 |
Messages logged at a lower level than this don't actually get logged anywhere. |
GLOG_logbuflevel |
int |
0 |
Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only...). |
GLOG_logbufsecs |
int |
30 |
Buffer log messages for at most this many seconds. |
GLOG_logfile_mode |
int |
0644 |
Log file mode/permissions. |
GLOG_log_dir |
string |
If specified, logfiles are written into this directory instead of the default logging directory. | |
GLOG_log_link |
string |
Put additional links to the log files in this directory. | |
GLOG_max_log_size |
int |
1800 |
Approx. maximum log file size (in MB). A value of 0 will be silently overridden to 1. |
GLOG_stop_logging_if_full_disk |
bool |
FALSE |
Stop attempting to log to disk if the disk is full. |
GLOG_log_backtrace_at |
string |
Emit a backtrace when logging at file:linenum. | |
GLOG_v |
int |
0 |
Show all VLOG(m) messages for m less or equal the value of this flag. |