NAME

HADaemon::Control - Create init scripts for Perl high-available (HA) daemons

DESCRIPTION

HADaemon::Control provides a library for creating init scripts for HA daemons in perl. It allows you to run one or more main processes accompanied by a set of standby processes. Standby processes constantly check presence of main ones and if later exits or dies promote themselves and replace gone main processes. By doing so, HADaemon::Control achieves high-availability and fault tolerance for a service provided by the deamon. Your perl script just needs to set the accessors for what and how you want something to run and the library takes care of the rest.

The library takes idea and interface from Daemon::Control and combine them with facilities of IPC::ConcurrencyLimit::WithStandby. IPC::ConcurrencyLimit::WithStandby implements a mechanism to limit the number of concurrent processes in a cooperative multiprocessing environment. For more information refer to the documentation of IPC::ConcurrencyLimit and IPC::ConcurrencyLimit::WithStandby

SYNOPSIS

#!/usr/bin/env perl

use strict;
use warnings;
use HADaemon::Control;

my $dc = HADaemon::Control->new({
    name => 'test.pl',
    user => 'nobody',
    pid_dir => '/tmp/test',
    log_file => '/tmp/test.log',
    program => sub { sleep 10; },
});

exit $dc->run();

You can then call the program:

/usr/bin/my_program_launcher.pl start

By default run will use @ARGV for the action, and exit with an LSB compatible exit code. For finer control, you can use run_command, which will return the exit code, and accepts the action as an argument. This enables more programatic control, as well as running multiple instances of HADaemon::Control from one script.

my $dc = HADaemon::Control->new({
    ...
});

my $exit = $daemon->run_command(“start”);

CONSTRUCTOR

The constructor takes the following arguments.

name

The name of the program the daemon is controlling. This will be used in status messages. See also process_name_change.

program

This should be a coderef of actual programm to run.

$daemon->program( sub { ... } );

program_args

This is an array ref of the arguments for the program. Args will be given to the program coderef as @_, the HADaemon::Control instance that called the coderef will be passed as the first arguments. Your arguments start at $_[1].

$daemon->program_args( [ 'foo', 'bar' ] );

pid_dir

This option defines directory where all pidfile will be created

$daemon->pid_dir('/var/run/my_program_launcher');

ipc_cl_options

This option gives ability to tune settings of underlying IPC::ConcurrencyLimit::WithStandby object. By default HADaemon::Control sets following settings:

ipc_cl_options => {
    type              => 'Flock',                             # the only supported type
    max_procs         => 1,                                   # one main process
    standby_max_procs => 1,                                   # one standby process
    interval          => 1,                                   # stanby tries to acquire main lock every second
    retries           => sub { 1 },                           # keep retrying forever
    path              => $daemon->pid_dir . '/lock/',         # path for main locks
    standby_path      => $daemon->pid_dir . '/lock-standby/', # path for standby locks
},

main_stop_file

This option provides an alternative way of stopping main processes apart of sending a signal (ex. TERM). If specified, HADaemon::Control touch this file and wait stop_file_kill_timeout or kill_timeout seconds hoping that main processes will respect the file and exit. If not, normal termination loop is entered (i.e. sending sequence of signals TERM TERM INT KILL). The filename can include %p which is replaced by PID of a process. Default value is undef.

standby_stop_file

The path to stop file for standby process. See do_start, do_stop, do_restart for details. By default is set to:

$daemon->standby_stop_file($daemon->pid_dir . '/standby-stop-file');

stop_signals

An array ref of signals that should be tried (in order) when stopping the daemon. Default signals are TERM, TERM, INT and KILL (yes, TERM is tried twice).

log_file

HADaemon::Control uses log_file for two purposes:

HADaemon::Control redirects STDOUT and STDERR for forked processes to given file
HADaemon::Control prints its own log to given file

If you don't want to mix logs of the application and init script consider using stdout_file and stderr_file. Verbosity of logs of HADaemon::Control can be controled by HADC_TRACE environment variable.

process_name_change

If set, HADaemon::Control will set name of the process to name. Also, it adds process_name_change option into ipc_cl_options. As result, process_name_change makes nice names for both main and standby processes. For example:

my $dc = HADaemon::Control->new({
    name => 'My test daemon',
    pid_dir => '/tmp/test',
    log_file => '/tmp/test.log',
    program => sub { sleep 10; },
    process_name_change => 1,
});

leads to:

My test daemon              # name of main process
My test daemon - standby    # name of standby process

user

When set, the username supplied to this accessor will be used to set the UID attribute. When this is used, uid will be changed from its initial settings if you set it (which you shouldn't, since you're using usernames instead of UIDs). See "uid" for setting numerical user ids.

$daemon->user('www-data');

group

When set, the groupname supplied to this accessor will be used to set the GID attribute. When this is used, gid will be changed from its initial settings if you set it (which you shouldn't, since you're using groupnames instead of GIDs). See "gid" for setting numerical group ids.

$daemon->group('www-data');

uid

If provided, the UID that the program will drop to when forked. This will only work if you are running as root. Accepts numeric UID. For usernames please see "user".

$daemon->uid( 1001 );

gid

If provided, the GID that the program will drop to when forked. This will only work if you are running as root. Accepts numeric GID, for groupnames please see "group".

$daemon->gid( 1001 );

umask

If provided, the umask of the daemon will be set to the umask provided, note that the umask must be in oct. By default the umask will not be changed.

$daemon->umask( 022 );

Or:

$daemon->umask( oct("022") );

stdout_file

If provided stdout of main process will be redirected to the given file.

$daemon->stdout_file( "/tmp/mydaemon.stdout" );

stderr_file

If provided stderr of main process will be redirected to the given file.

$daemon->stderr_file( "/tmp/mydaemon.stderr" );

kill_timeout

This provides an amount of time in seconds between trying different means of terminating the daemon. This value should be increased if your daemon has a longer shutdown period. By default 1 second is used.

$daemon->kill_timeout( 7 );

This value is used both for stop files and signals.

stop_file_kill_timeout

This is a more specific variant of kill_timeout. It provides the amount of seconds we allow the daemon to terminate itself once a stop file has been created.

If provided, this value has priority over kill_timeout.

$daemon->stop_file_kill_timeout( 42 );

signal_kill_timeout

This is a more specific variant of kill_timeout. It provides the amount of seconds between firing different signals to terminate the daemon.

If provided, this value has priority over kill_timeout.

$daemon->signal_kill_timeout( 42 );

quiet

If this boolean flag is set to a true value all output from the init script (NOT your daemon) to STDOUT will be suppressed.

$daemon->quiet( 1 );

close_fds_on_start

By default HADC closes all file descriptors apart of STDIN, STDOUT, STDERR and lock fd (see HADC_lock_fd) when it starts main process. This is done to make sure that main and standby processes are really independent of each other.

If this behaivor is not desirable, one can set close_fds_on_start to 0. But it should be understood that if parent process open any file descriptor (i.e. establish any connection, open file, create pipe, etc) those FDs will be available in *both* main and standby processes. Such case can be dangerous. Consider this: - parent processes (i.e. processes which run HADC) connect to DB - HADC starts main processes which uses that DB connection - HADC starts standby process - after a while main processes crashes and leave connectiong to DB in unpredicted state - standby processes promotes to main one and try to use *same* connection to DB. Since connection is in unpredicted state, it failes to use connection and crashes too.

reset_close_on_exec_main_lock_fd

By default perl sets close-on-exec flag for all opened file descriptors. If this flag is not desirable for lock fd one can set this option to true. This option should be set if client code do exec(). For details about close-on-exec check 'man fcntl'.

INIT FILE CONSTRUCTOR OPTIONS

The constructor also takes the following arguments to generate init file. See "do_get_init_file".

path

The path of the script you are using HADaemon::Control in. This will be used in the LSB file generation to point it to the location of the script. If this is not provided, the absolute path of $0 will be used.

init_config

The name of the init config file to load. When provided your init script will source this file to include the environment variables. This is useful for setting a PERL5LIB and such things.

$daemon->init_config( "/etc/default/my_program" );

If you are using perlbrew, you probably want to set your init_config to
C<$ENV{PERLBREW_ROOT} . '/etc/bashrc'>.

init_code

When given, whatever text is in this field will be dumped directly into the generated init file.

$daemon->init_code( "Arbitrary code goes here." )

lsb_start

The value of this string is used for the 'Required-Start' value of the generated LSB init script. See http://wiki.debian.org/LSBInitScripts for more information.

$daemon->lsb_start( '$remote_fs $syslog' );

lsb_stop

The value of this string is used for the 'Required-Stop' value of the generated LSB init script. See http://wiki.debian.org/LSBInitScripts for more information.

$daemon->lsb_stop( '$remote_fs $syslog' );

lsb_sdesc

The value of this string is used for the 'Short-Description' value of the generated LSB init script. See http://wiki.debian.org/LSBInitScripts for more information.

$daemon->lsb_sdesc( 'My program...' );

lsb_desc

The value of this string is used for the 'Description' value of the generated LSB init script. See http://wiki.debian.org/LSBInitScripts for more information.

$daemon->lsb_desc( 'My program controls a thing that does a thing.' );

METHODS

run_command

This function will process an action on the HADaemon::Control instance. Valid arguments are those which a do_ method exists for, such as start, stop, restart. Returns the LSB exit code for the action processed.

run

This will make your program act as an init file, accepting input from the command line. Run will exit with 0 for success and uses LSB exit codes. As such no code should be used after ->run is called. Any code in your file should be before this. This is a shortcut for

exit HADaemon::Control->new(...)->run_command( @ARGV );

do_start

Is called when start is given as an argument. Starts the forking and exits. The forking includes starting ipc_cl_options-{max_procs}> main and ipc_cl_options-{standby_max_procs}> standby processes. Exit with success only if all processes were spawned. Called by:

/usr/bin/my_program_launcher.pl start

do_stop

Is called when stop is given as an argument. Stops the all running proceses which belongs to the daemon if it can. Stopping is done via:

touching standby_stop_file file to stop standby processes and prevent new proceses to be started via do_fork command.
if main_stop_file specified, touch it to stop main processes. See main_stop_file.
send "TERM TERM INT KILL" sequence of signals to kill main processes

Called by:

/usr/bin/my_program_launcher.pl stop

do_restart

Is called when restart is given as an argument. This command triggers restart cycle which includes several steps:

stop all standby daemons by touching standby_stop_file
start new instances of standby processes
kill main processes one by one. Once a main processes is dead, running standby immediately become main one hence minimize downtime to ipc_cl_options-{interval}> seconds (or miliseconds).
again start standby processes to compensate the lost of standby processes

Called by:

/usr/bin/my_program_launcher.pl restart

do_hard_restart

Is called when hard_restart is given as an argument. Calls do_stop and do_start. Called by:

/usr/bin/my_program_launcher.pl hard_restart

do_fork

Is called when fork is given as an argument. This command is almost equal to do_start, but is design for periodical run in a cronjob. Called by:

/usr/bin/my_program_launcher.pl fork

do_reload

Is called when reload is given as an argument. Sends a HUP signal to the main processes.

/usr/bin/my_program_launcher.pl reload

do_status

Is called when status is given as an argument. Displays the statuses of the program (i.e. all running processes), basic on the PID files. Called by:

/usr/bin/my_program_launcher.pl status

do_foreground

Is called when foreground is given as an argument. Starts the program or code reference and stays in the foreground -- no forking and locking is done, regardless of the compile-time arguments. Additionally, turns quiet on to avoid showing HADaemon::Control output.

/usr/bin/my_program_launcher.pl foreground

do_get_init_file

Is called when get_init_file is given as an argument. Dumps an LSB compatible init file, for use in /etc/init.d/. Called by:

/usr/bin/my_program_launcher.pl get_init_file

pretty_print

This is used to display status to the user. It accepts a message and a color. It will default to green text, if no color is explicitly given. Only supports red and green. If HADC_NO_COLORS environment variable is set no colors are used.

$daemon->pretty_print( "My Status", "red" );

KNOWN ISSUES

HADaemon::Control uses flock based locks. This type of locks have property of getting inherited accross fork system call. This behavior is not desirable and actually destructible for HADaemon::Control. Once the locked is inherited, two processes (parent and child) will own the same lock. Only releasing the lock from both processes allows another one to acuire the lock. To prevent such behivour HADaemon::Control exposes lock's file descriptor via HADC_lock_fd environment variable.

If an application forks, a child process should close lock's file descriptor right after exiting from fork syscal. One of the possible ways is to run:

$ENV{HADC_lock_fd} and POSIX::close($ENV{HADC_lock_fd});

Another source of troubles could be the fact that HADC closes all file descriptors apart of STDIN, STDOUT, STDERR and lock fd upon starting main processes (tunable via close_fds_on_start). This's done for a reason. See close_fds_on_start for details.

AUTHOR

Ivan Kruglov, ivan.kruglov@yahoo.com

CONTRIBUTORS

Alexey Surikov alexey.surikov@booking.com

ACKNOWLEDGMENT

This module was inspired by module Daemon::Control.

This module was originally developed for Booking.com. With approval from Booking.com, this module was generalized and put on CPAN, for which the authors would like to express their gratitude.

COPYRIGHT AND LICENSE

This code is available under the same license as Perl version 5.8.1 or higher.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

AVAILABILITY

The most current version of HADaemon::Control can be found at https://github.com/ikruglov/HADaemon-Control