/perl-and-ssh

Perl and SSH presentation at YAPC::Europe 2013

Perl and SSH

Salvador Fandiño (sfandino@yahoo.com)

YAPC::Europe 2013

How do you automate SSH tasks?

If you use ...

  • system("ssh $host $cmd @args")
  • or backticks
  • Net::SSH
  • Net::SSH::Perl
  • Expect
  • Net::SSH::Expect

...you are living in the past!

Using system or backticks

  • ok for simple things
  • but very inneficient, opens a new SSH connection every time
  • no password authentication
  • you have to properly quote the arguments
  • sometimes twice
  • or be insecure

Net::SSH

  • simple wrapper for the ssh command
  • provides also sshopen2 and sshopen3
  • has same problems as using system

Net::SSH::Perl

  • huge module
  • at its time it represented a great effort
  • I guess most of it is a direct translation of OpenSSH C code
  • but today, nobody maintains it
  • lots of known bugs
  • very difficult to install (Math::Pari)
  • API good for simple things, bad for advanced things
  • supports SFTP via Net::SFTP, doesn't support SCP

Expect

  • Expect is a great module for automating interactive programs
  • It's a good way to automate ssh password authentication
  • After that, it is (usually) the wrong tool
  • It talks to the remote shell, you don't want that!
  • On some systems (AIX, HP-UX) ttys may drop data

BTW...

!perl
my $pty = IO::Pty->new;
my $expect = Expect->init($pty);
my $pid = open2($in, $out, '-');
unless ($pid) {
  defined $pid or die "unable to fork";
  $pty->make_slave_controlling_terminal;
  do { exec @ssh_cmd };
  exit -1;
}

Net::SSH::Expect

  • Builds on top of Expect
  • On the wrong way
  • It talks to the shell
  • It is not reliable

So, what should I use?

Modern SSH modules

  • Net::SSH2
  • Net::OpenSSH
  • Net::SSH::Any

Well, actually...

  • Net::SSH2
  • Net::OpenSSH
  • Net::SSH::Any
  • Net::SSH::Mechanize
  • Net::OpenSSH::Parallel
  • Net::OpenSSH::Compat
  • Net::SFTP::Foreign
  • ...

Net::SSH2

Net::SSH2

  • wrapper for the libssh2 C library
  • quite portable
  • quite easy to install on Unix/Linux, almost easy to install on Windows, don't known about VMS
  • it is a very thin wrapper
  • efficient, can run several commands over one SSH connection
  • C'ish low level API:
    • very simple things are easy to do
    • not so simple things become quite hard
  • supports SCP
  • very primitive and inneficient support for SFTP
  • project started by David B. Robins, currently being actively maintained by Rafael Kitover

Net::SSH2 - usage

!perl
use Net::SSH2;

my $ssh2 = Net::SSH2->new();

$ssh2->connect('example.com') or die $!;

if ($ssh2->auth_keyboard('fizban')) {
    my $chan = $ssh2->channel();
    $chan->exec('program');
}

Net::SSH2 problems

  • very low level C'ish API: not for lazy people
  • libssh2 is not a mature project yet
  • requires a C compiler

Net::OpenSSH

Net::OpenSSH

  • wrapper around OpenSSH ssh
  • uses its connection multiplexing feature
    • several commands can be run over the same connection
    • efficient
  • Perlish API with lots of belts and whistles
  • easy to use, complex things are almost easy
  • can work asynchronously
  • supports SFTP, SCP, rsync and sshfs
  • automatic argument quoting

Net::OpenSSH - usage

!perl
use Net::OpenSSH;
my $ssh = Net::OpenSSH->new($host, user => $user, password => $password);
$ssh->die_on_error("unable to connect");

my @output = $ssh->capture("cat /etc/passwd");

my ($out, $err) = $ssh->capture2("find /");

$ssh->system({stdin_data => "hello\n"},
             "cat >>log");

my $pid = $ssh->spawn({stderr_to_stdout => 1,
                       stdout_file => "tar.log"},
                      'tar', 'cf', '/tmp/my backup.tar', '/home/me');
waitpid($pid,0);

$ssh->scp_get('/tmp/*.tar', '.');

$ssh->rsync_put({verbose => 1, safe_links => 1},
                "etc", "/etc");

my $sftp = $ssh->sftp; # return a Net::SFTP::Foreign object
my $ls = $sftp->ls;

Net::OpenSSH argument quoting

!perl

my $out = $ssh->capture("ls /etc"); # no quoting
my $out = $ssh->capture(ls => '~/My Documents'); # quoting
my $out = $ssh->capture({quote_args => 0},
                        ls => '~/My Documents'); # no quoting

# selectively quoting arguments:
my $out = $ssh->capture(ls => \'~/My Documents/*.pdf');
        # lets file name wildcards be expanded by the remote shell

my $out = $ssh->capture(ls => \\'~/My Documents/*.pdf'); # wrong!

my $out = $ssh->capture(@cmd1, \\'&&', @cmd2, \\'2>/dev/null');

Net::OpenSSH argument quoting

  • On the stable release, argument quoting expects a POSIX compatible shell on the remote side

  • The development release has support for different quoting backends - POSIX (i.e. ksh, bash) and csh already there - maybe Windows/DOS backend in the future

Net::OpenSSH problems

  • Net::OpenSSH does not work on Windows
    • OpenSSH multiplexing feature does not work on Windows, not even under cygwin
  • Requires the OpenSSH ssh command

Net::SSH::Any

Net::SSH::Any

  • API very similar to Net::OpenSSH

  • works on top of

    • Net::SSH2
    • Net::OpenSSH
    • maybe Net::SSH::Perl in the future
    • maybe simple wrapper around native ssh

  • supports SCP and SFTP for file transfers, efficiently

  • a work in progress

  • though, basic functionality is already stable

Net::SSH::Mechanize

Net::SSH::Mechanize

  • It uses the AnyEvent framework
  • It aims to support sudoing smoothly
  • Limited API
  • It talks to the remote shell, unreliable!

Net::OpenSSH::Parallel

Net::OpenSSH::Parallel

  • Run commands in parallel in remote hosts through SSH
  • Build on top of Net::OpenSSH
  • Declarative API:
    • tell the module all the actions you want to perform on the remote hosts
    • let the module take care of everything and do the tasks
    • handle possible errors

Net::OpenSSH::Parallel - usage

!perl
my $pssh = Net::OpenSSH::Parallel->new;

# tell the object what the remote hosts are:
$pssh->add_host('host1', user => foo, password => $pwd);
$pssh->add_host('host2', user => foo, password => $pwd);

# declare the actions you want to run:
$pssh->push('host1', command => "echo hello from host1");
$pssh->push('host2', command => "echo hello from host2");

# for several host in one call:
$pssh->push('host*', command => "echo hello from some host");

# with variable expansion:
$pssh->push('host*', command => "echo hello from host %HOST%");

# and run it:
$pssh->run;

Net::OpenSSH::Parallel - usage

!perl
# other actions:
$pssh->push('*', scp_get => '/var/log/messages.0', 'logs/messages.0.%HOST%');

$pssh->push('*', rsync_put => '/var/www', '/var/www');

# can pass extra arguments to the underlaying Net::OpenSSH methods
$pssh->push('*', rsync_put => { safe_links => 1,
                                stdout_file => ['>>', 'mylog'],
                                stderr_to_stdout => 1 },
			  '/var/www', '/var/www');

# runs a custom sub on a locally forked process
$pssh->push('*', parsub => \&my_sub);

Net::OpenSSH::Parallel - Expect and sudo

!perl
sub sudo_install {
    my ($label, $ssh, @pkgs) = @_;
    my ($pty) = $ssh->open2pty('sudo', 'apt-get', 'install', @pkgs);
    my $expect = Expect->init($pty);
    $expect->raw_pty(1);
    $expect->expect($timeout, ":");
    $expect->send("$passwd\n");
    $expect->expect($timeout, "\n");
    $expect->raw_pty(0);
    while(<$expect>) { print };
    close $expect;
}

$pssh->push('*', parsub => \&sudo_install, 'scummvm');

Net::OpenSSH::Parallel - dependencies

!perl
$pssh->push('dmz*', scp_get => '/etc/passwd', 'passwd.%HOST%');
$pssh->push('safe', join => 'dmz*');
$pssh->push('safe', command => mkdir, "/var/safe/$date");
$pssh->push('safe', scp_put => 'passwd.*', "/var/safe/$date");
$pssh->run;

Net::OpenSSH::Parallel problems

  • Unable to distribute tasks around a set of hosts
  • Can't run more than one command per host simultaneously
  • stdin/stdout/stderr has to go to the file system, no pipes between tasks

Net::OpenSSH::Compat

Net::OpenSSH::Compat

  • Implements most of Net::SSH::Perl and Net::SSH2 APIs on top of Net::OpenSSH
  • Because sometimes people has problems installing them

Net::SFTP::Foreign

On the works...

Other interesting modules

  • SSH::Batch
  • App::MrShell
  • POE::Component::OpenSSH
  • GRID::Machine
  • IPC::PerlSSH
  • (R)?ex
  • ...

Questions

Thank you!

Links