/mesos-slave

NO LONGER MAINTAINED - An ubuntu based Mesos-Slave container

Primary LanguageShell

- Mesos Slave -

An Ubuntu based Mesos Slave container, packaged with Logstash-Forwarder and managed by Supervisord. All parameters are controlled through environment variables, with some settings auto-configured based on the environment.

Version Information:
  • Container Release: 1.2.0
  • Mesos: 0.26.0-0.2.145.ubuntu1404
  • Docker: 1.9.1-0~trusty

Services Include:

  • Mesos Slave - Primary process that offers resources of the host to the Mesos Master(s) for scheduling and running of tasks.
  • Consul-Template - An application that can populate configs from a consul service.
  • Logrotate - A script and application that aid in pruning log files.
  • Logstash-Forwarder - A lightweight log collector and shipper for use with Logstash.
  • Redpill - A bash script and healthcheck for supervisord managed services. It is capable of running cleanup scripts that should be executed upon container termination.
  • Rsyslog - The system logging daemon.

Index

Usage

All mesos commands should be passed via environment variables (please see the example run command below). For Mesos documentation, please see the configuration docs associated with the release here: mesos@d3717e5

In a local proof of concept environment, the only variable that MUST be defined is MESOS_MASTER.

However, that will leave the slave with a fraction of it's functionality. To run in a useful fashion, the following should be set, ENVIRONMENT, MESOS_MASTER, MESOS_WORK_DIR, and MESOS_DOCKER_SANDBOX_DIRECTORY.

  • ENVIRONMENT - when set to production or development it will enable all services including: mesos-master, logstash-forwarder, and redpill.

  • MESOS_MASTER- Informs the slave how to connect or discover the Mesos Masters. Please see the Mesos docs for the available options.

  • MESOS_WORK_DIR - Path to the directory in which framework directories are placed.

  • MESOS_SANDBOX_DIRECTORY - Path to directory used to map the sandbox to Docker containers.

In addition to the above, there are several things to note for full compatibility when operating a mesos slave in a container with docker as a supported containerizer.

  • The container should be run with host networking.
  • The container requires several volumes to be mounted. For compatibility purposes these should be the same location as it is on the host.
  • /usr/bin/docker:/usr/bin/docker:ro
  • /var/run/docker.sock:/var/run/docker.sock:rw
  • /sys:/sys:ro
  • The directory used for MESOS_WORK_DIR as rw
  • The directory used for MESOS_SANDBOX_DIRECTORY as rw
Marathon Framework and Private Registry Access

Configuring private registry access is dependant on several factors. For documentation, please visit the Marathon Framework site directly.

In either case, if you intend on baking the credentials into the image. This would be the image to do it.

Example Run Command

docker run -d --net=host    \
--pid=host                  \
--name=mesosslave           \
--cap-add=SYS_ADMIN         \
-e ENVIRONMENT=production   \
-e PARENT_HOST=$(hostname)  \
-e MESOS_IP=10.10.0.111     \
-e MESOS_MASTER=zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos  \
-e MESOS_REGISTRATION_TIMEOUT=5min    \
-e MESOS_CONTAINERIZERS=docker,mesos  \
-e MESOS_HOSTNAME=10.10.0.111         \
-e MESOS_SANDBOX_DIRECTORY=/data/mesos/sandbox   \
-e MESOS_WORKDIR=/data/mesos/workdir             \
-v /data/mesos/workdir:/data/mesos/workdir:rw    \
-v /data/mesos/sandbox:/data/mesos/sandbox:rw    \
-v /usr/bin/docker:/usr/bin/docker:ro            \
-v /var/run/docker.sock:/var/run/docker.sock:rw  \
-v /sys/:/sys:ro  \
mesos-slave

Modification and Anatomy of the Project

File Structure The directory skel in the project root maps to the root of the file system once the container is built. Files and folders placed there will map to their corresponding location within the container.

Init The init script (./init.sh) found at the root of the directory is the entry process for the container. It's role is to simply set specific environment variables and modify any subsequently required configuration files.

Supervisord All supervisord configs can be found in /etc/supervisor/conf.d/. Services by default will redirect their stdout to /dev/fd/1 and stderr to /dev/fd/2 allowing for service's console output to be displayed. Most applications can log to both stdout and their respectively specified log file.

In some cases (such as with zookeeper), it is possible to specify different logging levels and formats for each location.

Logstash-Forwarder The Logstash-Forwarder binary and default configuration file can be found in /skel/opt/logstash-forwarder. It is ideal to bake the Logstash Server certificate into the base container at this location. If the certificate is called logstash-forwarder.crt, the default supplied Logstash-Forwarder config should not need to be modified, and the server setting may be passed through the SERVICE_LOGSTASH_FORWARDER_ADDRESS environment variable.

In practice, the supplied Logstash-Forwarder config should be used as an example to produce one tailored to each deployment.

Important Environment Variables

Defaults

Variable Default
ENVIRONMENT_INIT
APP_NAME mesos-slave
ENVIRONMENT local
PARENT_HOST unknown
MESOS_LOG_DIR /var/log/mesos
MESOS_WORK_DIR
GLOG_max_log_size
SERVICE_CONSUL_TEMPLATE disabled
SERVICE_LOGROTATE
SERVICE_LOGROTATE_INTERVAL 3600 (set in script by default)
SERVICE_LOGROTATE_SCRIPT /opt/scripts/purge-mesos-logs.sh
SERVICE_LOGSTASH_FORWARDER
SERVICE_LOGSTASH_FORWARDER_CONF /opt/logstash-forwarder/mesos-slave.conf
SERVICE_REDPILL
SERVICE_REDPILL_MONITOR mesos
SERVICE_RSYSLOG disabled
Description

  • ENVIRONMENT_INIT - If set, and the file path is valid. This will be sourced and executed before ANYTHING else. Useful if supplying an environment file or need to query a service such as consul to populate other variables.

  • APP_NAME - A brief description of the container. If Logstash-Forwarder is enabled, this will populate the app_name field in the Logstash-Forwarder configuration file.

  • ENVIRONMENT - Sets defaults for several other variables based on the current running environment. Please see the environment section for further information. If logstash-forwarder is enabled, this value will populate the environment field in the logstash-forwarder configuration file.

  • PARENT_HOST - The name of the parent host. If Logstash-Forwarder is enabled, this will populate the parent_host field in the Logstash-Forwarder configuration file.

  • MESOS_LOG_DIR - The path to the directory in which Mesos stores its logs.

  • MESOS_WORK_DIR - Path to the directory in which framework directories are placed.

  • GLOG_max_file_size - The size in Megabytes that the mesos log file(s) will be allowed to grow to before rotation.

  • SERVICE_CONSUL_TEMPLATE - * SERVICE_CONSUL_TEMPLATE- Enables or disables the consul-template service. If enabled, it will also enableSERVICE_LOGROTATEandSERVICE_RSYSLOGto handle logging. (**Options:**enabledordisabled`)

  • SERVICE_LOGROTATE - Enables or disabled the Logrotate service. This will be set automatically depending on the environment. (Options: enabled or disabled)

  • SERVICE_LOGROTATE_INTERVAL - The time in seconds between runs of logrotate or the logrotate script. The default (3600 or 1 hour) is set by default in the logrotate script automatically.

  • SERVICE_LOGROTATE_SCRIPT - The path to the script that should be executed instead of logrotate itself to clean up logs.

  • SERVICE_LOGSTASH_FORWARDER - Enables or disables the Logstash-Forwarder service. Set automatically depending on the ENVIRONMENT. See the Environment section below. (Options: enabled or disabled)

  • SERVICE_LOGSTASH_FORWARDER_CONF - The path to the logstash-forwarder configuration.

  • SERVICE_REDPILL - Enables or disables the Redpill service. Set automatically depending on the ENVIRONMENT. See the Environment section below. (Options: enabled or disabled)

  • SERVICE_REDPILL_MONITOR - The name of the supervisord service(s) that the Redpill service check script should monitor.

  • SERVICE_RSYSLOG - Enables of disables the rsyslog service. This is managed by SERVICE_CONSUL_TEMPLATE, but can be enabled/disabled manually. (Options: enabled or disabled)

Environment

  • local (default)
Variable Default
MESOS_HOSTNAME <first ip bound to eth0>
GLOG_max_log_size 10
SERVICE_LOGROATE enabled
SERVICE_LOGSTASH_FORWARDER disabled
SERVICE_REDPILL enabled
MESOS_WORK_DIR /var/lib/mesos
  • prod|production|dev|development
Variable Default
GLOG_max_log_size 10
SERVICE_LOGROATE enabled
SERVICE_LOGSTASH_FORWARDER enabled
SERVICE_REDPILL enabled
  • debug
Variable Default
SERVICE_LOGROTATE disabled
SERVICE_LOGSTASH_FORWARDER disabled
SERVICE_REDPILL disabled
CONSUL_TEMPLATE_LOG_LEVEL debug *

* Only set if SERVICE_CONSUL_TEMPLATE is set to enabled.

Service Configurations

Mesos-Slave

As stated in the Usage section, Mesos-slave configuration information can be found in the github docs releated to the Mesos Release: mesos@d3717e5.

The actual mesos start command is passed to supervisor via the SERVICE_MESOS_CMD environment variable, and defaults to mesos-slave.

Mesos-Slave Environment Variables

Defaults
Variable Default
MESOS_CONTAINERIZERS docker,mesos
MESOS_LOG_DIR /var/log/mesos
MESOS_MASTER
MESOS_SANDBOX_DIRECTORY /mnt/mesos/sandbox
MESOS_WORK_DIR
SERVICE_MESOS_CMD mesos-slave
Description

  • MESOS_CONTAINERIZES - Comma separated list of containerizers for use with Mesos. Priority is assigned in the order in which they're passed.

  • MESOS_LOG_DIR - The path to the directory in which Mesos stores its logs.

  • MESOS_WORK_DIR - Path to the directory in which framework directories are placed.

  • SERVICE_MESOS_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.

Consul-Template

Provides initial configuration of consul-template. Variables prefixed with CONSUL_TEMPLATE_ will automatically be passed to the consul-template service at runtime, e.g. CONSUL_TEMPLATE_SSL_CA_CERT=/etc/consul/certs/ca.crt becomes -ssl-ca-cert="/etc/consul/certs/ca.crt". If managing the application configuration is handled via file configs, no other variables must be passed at runtime.

Consul-Template Environment Variables

Defaults
Variable Default
CONSUL_TEMPLATE_CONFIG /etc/consul/template/conf.d
CONSUL_TEMPLATE_SYSLOG true
SERVICE_CONSUL_TEMPLATE
SERVICE_CONSUL_TEMPLATE_CMD consul-template <CONSUL_TEMPLATE_*>

Logrotate

The logrotate script is a small simple script that will either call and execute logrotate on a given interval; or execute a supplied script. This is useful for applications that do not perform their own log cleanup.

Logrotate Environment Variables

Defaults
Variable Default
SERVICE_LOGROTATE
SERVICE_LOGROTATE_INTERVAL 3600
SERVICE_LOGROTATE_CONFIG /etc/logrotate.conf
SERVICE_LOGROTATE_SCRIPT /opt/scripts/purge-mesos-logs.sh
SERVICE_LOGROTATE_FORCE
SERVICE_LOGROTATE_VERBOSE
SERVICE_LOGROTATE_DEBUG
SERVICE_LOGROTATE_CMD /opt/script/logrotate.sh <flags>
Description

  • SERVICE_LOGROTATE - Enables or disables the Logrotate service. Set automatically depending on the ENVIRONMENT. See the Environment section. (Options: enabled or disabled)

  • SERVICE_LOGROTATE_INTERVAL - The time in seconds between run of either the logrotate command or the provided logrotate script. Default is set to 3600 or 1 hour in the script itself.

  • SERVICE_LOGROTATE_CONFIG - The path to the logrotate config file. If neither config or script is provided, it will default to /etc/logrotate.conf.

  • SERVICE_LOGROTATE_SCRIPT - A script that should be executed on the provided interval. Useful to do cleanup of logs for applications that already handle rotation, or if additional processing is required.

  • SERVICE_LOGROTATE_FORCE - If present, passes the 'force' command to logrotate. Will be ignored if a script is provided.

  • SERVICE_LOGROTATE_VERBOSE - If present, passes the 'verbose' command to logrotate. Will be ignored if a script is provided.

  • SERVICE_LOGROTATE_DEBUG - If present, passed the 'debug' command to logrotate. Will be ignored if a script is provided.

  • SERVICE_LOGROTATE_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.

Logrotate Script Help Text
root@ec58ca7459cb:/opt/scripts# ./logrotate.sh --help
logrotate.sh - Small wrapper script for logrotate.
-i | --interval     The interval in seconds that logrotate should run.
-c | --config       Path to the logrotate config.
-s | --script       A script to be executed in place of logrotate.
-f | --force        Forces log rotation.
-v | --verbose      Display verbose output.
-d | --debug        Enable debugging, and implies verbose output. No state file changes.
-h | --help         This usage text.
Supplied Cleanup Script

The below cleanup script will remove all but the latest 5 rotated logs.

#!/bin/bash

mld=${MESOS_LOG_DIR:-/var/log/mesos}

cd "$mld"

(ls -t | grep 'log.INFO.*'|head -n 5;ls)|sort|uniq -u|grep 'log.INFO.*'|xargs --no-run-if-empty rm
(ls -t | grep 'log.ERROR.*'|head -n 5;ls)|sort|uniq -u|grep 'log.ERROR.*'|xargs --no-run-if-empty rm
(ls -t | grep 'log.WARNING.*'|head -n 5;ls)|sort|uniq -u|grep 'log.WARNING.*'|xargs --no-run-if-empty rm

#consul-template uses rsyslog for logging, need to run logrotate to handle that log
if [[ "$SERVICE_CONSUL_TEMPLATE" == "enabled" ]]; then
  /usr/sbin/logrotate /etc/logrotate.conf
fi

Logstash-Forwarder

Logstash-Forwarder is a lightweight application that collects and forwards logs to a logstash server endpoint for further processing. For more information see the Logstash-Forwarder project.

Logstash-Forwarder Environment Variables

Defaults
Variable Default
SERVICE_LOGSTASH_FORWARDER
SERVICE_LOGSTASH_FORWARDER_CONF /opt/logstash-forwarder/mesos-slave.conf
SERVICE_LOGSTASH_FORWARDER_ADDRESS
SERVICE_LOGSTASH_FORWARDER_CERT
SERVICE_LOGSTASH_FORWARDER_CMD /opt/logstash-forwarder/logstash-fowarder -config="${SERVICE_LOGSTASH_FOWARDER_CONF}"
Description

  • SERVICE_LOGSTASH_FORWARDER - Enables or disables the Logstash-Forwarder service. Set automatically depending on the ENVIRONMENT. See the Environment section. (Options: enabled or disabled)

  • SERVICE_LOGSTASH_FORWARDER_CONF - The path to the logstash-forwarder configuration.

  • SERVICE_LOGSTASH_FORWARDER_ADDRESS - The address of the Logstash server.

  • SERVICE_LOGSTASH_FORWARDER_CERT - The path to the Logstash-Forwarder server certificate.

  • SERVICE_LOGSTASH_FORWARDER_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.

Rsyslog

Rsyslog is a high performance log processing daemon. For any modifications to the config, it is best to edit the rsyslog configs directly (/etc/rsyslog.conf and /etc/rsyslog.d/*).

Defaults
Variable Default
SERVICE_RSYSLOG disabled
SERVICE_RSYSLOG_CONF /etc/rsyslog.conf
SERVICE_RSYSLOG_CMD /usr/sbin/rsyslogd -n -f $SERVICE_RSYSLOG_CONF
Description

  • SERVICE_RSYSLOG - Enables or disables the rsyslog service. This will automatically be set depending on what other services are enabled. (Options: enabled or disabled)

  • SERVICE_RSYSLOG_CONF - The path to the rsyslog configuration file.

  • SERVICE_RSYSLOG_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.

Redpill

Redpill is a small script that performs status checks on services managed through supervisor. In the event of a failed service (FATAL) Redpill optionally runs a cleanup script and then terminates the parent supervisor process.

Redpill Environment Variables

Defaults
Variable Default
SERVICE_REDPILL
SERVICE_REDPILL_MONITOR mesos
SERVICE_REDPILL_INTERVAL
SERVICE_REDPILL_CLEANUP
SERVICE_REDPILL_CMD
Description

  • SERVICE_REDPILL - Enables or disables the Redpill service. Set automatically depending on the ENVIRONMENT. See the Environment section. (Options: enabled or disabled)

  • SERVICE_REDPILL_MONITOR - The name of the supervisord service(s) that the Redpill service check script should monitor.

  • SERVICE_REDPILL_INTERVAL - The interval in which Redpill polls supervisor for status checks. (Default for the script is 30 seconds)

  • SERVICE_REDPILL_CLEANUP - The path to the script that will be executed upon container termination.

  • SERVICE_REDPILL_CMD - The command that is passed to supervisor. It is dynamically built from the other redpill variables. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.

Redpill Script Help Text
root@c90c98ae31e1:/# /opt/scripts/redpill.sh --help
Redpill - Supervisor status monitor. Terminates the supervisor process if any specified service enters a FATAL state.

-c | --cleanup    Optional path to cleanup script that should be executed upon exit.
-h | --help       This help text.
-i | --interval   Optional interval at which the service check is performed in seconds. (Default: 30)
-s | --service    A comma delimited list of the supervisor service names that should be monitored.

Troubleshooting

In the event of an issue, the ENVIRONMENT variable can be set to debug. This will stop the container from shipping logs and prevent it from terminating if one of the services enters a failed state.

For mesos itself, the MESOS_LOGGING_LEVEL variable can be set to INFO or WARNING to further diagnose the problem.

If further information is needed; logging may be controlled directly by configuring glog loggig library used by Mesos. For reference; please see the table below:

Note: not all of the glog settings are overridable. Mesos does dictate some of them. Main ones of note are GLOG_v to increase log verbosity, and GLOG_max_log_size if log growth and rotation must be tuned.

Environment Variable Name Type Default Description
GLOG_logtostderr bool FALSE Log messages go to stderr instead of logfiles.
GLOG_alsologtostderr bool FALSE Log messages go to stderr in addition to logfiles.
GLOG_colorlogtostderr bool FALSE Color messages logged to stderr (if supported by terminal).
GLOG_stderrthreshold int 2 Log messages at or above this level are copied to stderr in addition to logfiles. This flag obsoletes –alsologtostderr.
GLOG_alsologtomail string Log messages go to these email addresses in addition to logfiles.
GLOG_logmaillevel int 999 Email log messages logged at this level or higher (0 means email all; 3 means email FATAL only ...)
GLOG_logmailer string /bin/mail Mailer used to send logging email.
GLOG_drop_log_memory bool TRUE Drop in-memory buffers of log contents. Logs can grow very quickly and they are rarely read before they need to be evicted from memory. Instead, drop them from memory as soon as they are flushed to disk.
GLOG_log_prefix bool TRUE Prepend the log prefix to the start of each log line.
GLOG_minloglevel int 0 Messages logged at a lower level than this don't actually get logged anywhere.
GLOG_logbuflevel int 0 Buffer log messages logged at this level or lower (-1 means don't buffer; 0 means buffer INFO only...).
GLOG_logbufsecs int 30 Buffer log messages for at most this many seconds.
GLOG_logfile_mode int 0644 Log file mode/permissions.
GLOG_log_dir string If specified, logfiles are written into this directory instead of the default logging directory.
GLOG_log_link string Put additional links to the log files in this directory.
GLOG_max_log_size int 1800 Approx. maximum log file size (in MB). A value of 0 will be silently overridden to 1.
GLOG_stop_logging_if_full_disk bool FALSE Stop attempting to log to disk if the disk is full.
GLOG_log_backtrace_at string Emit a backtrace when logging at file:linenum.
GLOG_v int 0 Show all VLOG(m) messages for m less or equal the value of this flag.