/overthere

Runs something "Over there"

Primary LanguageJavaOtherNOASSERTION

Table of Contents

# Introduction

Overthere is a Java library to manipulate files and execute processes on remote hosts, i.e. do stuff "over there". It was originally developed for and is still used in the XebiaLabs deployment automation product Deployit as a way to perform tasks on remote hosts, e.g. copy configuration files, install EAR files or restart web servers. Another way of looking at it is to say that Overthere gives you java.io.File and java.lang.Process as they should've been: as interfaces, created by a factory and extensible through an SPI mechanism.

Overthere is available under the Apache License, Version 2.0.

P.S.: Check the Overthere Ohloh page for some interesting code analysis statistics. If you use Overthere, don't forget to tell Ohloh! And while you're at it, you might want to vote for Overthere on the Overthere Freecode page too! ;-)

# Getting Overthere

To get Overthere, you have two options:

  1. Add a dependency to Overthere to your project.
  2. Build Overthere yourself.

And, if you want, you can also run the Overthere examples used in the Overthere presentation mentioned above.

Binary releases of Overthere are not provided here, but you can download it straight from the Maven Central repository if you want to.

## Depending on Overthere

  1. If your project is built with Maven, add the following dependency to the pom.xml:

     <dependency>
 	<groupId>com.xebialabs.overthere</groupId>
 	<artifactId>overthere</artifactId>
 	<version>2.0.0-beta-1</version>
 </dependency>

  2. If your project is built using another build tool that uses the Maven Central repository, translate these dependencies into the format used by your build tool.

 ## Building Overthere
  1. Install Gradle 1.0-milestone-7.
  2. Clone the Overthere repository.
  3. Run the command gradle clean build.
## Running the examples
  1. Install Maven 2.2.1 or up.
  2. Clone the Overthere repository.
  3. Go into the examples directory and run the command mvn eclipse:eclipse.
  4. Import the examples project into Eclipse.
  5. Change the login details in the example classes (address, username and password) and run them!
# Programming Overthere

To program Overthere, browse the source code, check the examples and browse the Overthere Javadoc.

For a more thorough introduction to Overthere, check the presentation on Overthere that I gave for J-Fall 2011, a Java conference in the Netherlands (in English).

# Configuring Overthere

The protocols that Overthere uses to connect to remote hosts, such as SSH, CIFS, Telnet and WinRM, are existing protocols for which support is built into many platforms. As such you will not need to install any custom software on the target hosts. Nevertheless in some cases the target platforms have to be configured to correctly work with Overthere. Also, Overthere has a number of configuration features that allow you tweak the way it interfaces with the remote hosts.

 ## Protocols

Overthere supports a number of protocols to connect to remote hosts:

  • local - a connection to the local host. This is a wrapper around java.io.File and java.lang.Process.
  • ssh - a connection using the SSH protocol, to a Unix host or to a Windows host running either OpenSSH on Cygwin (i.e. COPSSH) or WinSSHD.
  • cifs - a connection using the CIFS protocol, also known as SMB, for file manipulation and, depending on the settings, using either Telnet or WinRM for process execution. This protocol is only supported for Windows hosts.
## Connection options

Apart from selecting a protocol to use, you will also need to supply a number of connection options when creating a connection. Common connection options are:

os The operating system of the remote host, either UNIX or WINDOWS. This property is required for all protocols, except for the local protocol.
address The address of the remote host, either an IP address or a DNS name.
port The port to use when connecting to the remote host. The interpretation and the default value for this connection option depend on the protocol that is used.
username The username to use when connecting to the remote host.
password The password to use.
tmp The temporary directory. For each connection, a connection temporary directory with a name like overthere-20111128T132600-7234435.tmp is created. By default that directory is removed when the connection is closed.
tmpFileCreationRetries The number of times Overthere attempts to create a temporary file with a unique name. The default value is 100.
tmpDeleteOnDisconnect If set to false, the connection temporary directory is not removed when the connection. The default value is true.
connectionTimeoutMillis The number of milliseconds Overthere waits for a connection to a remote host to be established. The default value is 120000, i.e. 2 minutes.
jumpstation The connection options used to connect to an SSH jumpstation (See Tunnelling)
Apart from these common connection options, some protocols define additional protocol-specific connection options. These are documented below, with the protocol. ## LOCAL

The local protocol implementation uses the local file manipulation and local process execution capabilities built-in to Java. The os connection property is hardcoded to the operating system of the local host and the tmp property defaults to the system temporary directory as specified by the java.io.tmpdir system property. There are no protocol-specific connection properties.

## SSH

The SSH protocol implementation of Overthere uses the SSH protocol to connect to remote hosts to manipulate files and execute commands. Most Unix systems already have an SSH server installed and configured and a number of different SSH implementations are available for Windows although not all of them are supported by Overthere.

See the section on the host setup for more information on how to setup the remote hosts.

### Connection options

The SSH protocol implementation of Overthere defines a number of additional connection properties, in addition to the common connection options.

connectionType Specifies how the SSH protocol is used. One of the following values must be set:
  • SFTP - uses SFTP to transfer files, to a Unix host. Unless sudo or a similar command is needed to execute commands, this is the best and fastest option to choose for Unix hosts.
  • SFTP_CYGWIN - uses SFTP to transfer files, to a Windows host running OpenSSH on Cygwin.
  • SFTP_WINSSHD - uses SFTP to transfer files, to a Windows host running WinSSHD.
  • SCP - uses SCP to transfer files, to a Unix host. Not needed unless your SSH server has disabled the SFTP subsystem.
  • SUDO - uses SCP to transfer files, to a Unix host. Uses the sudo command, configured with NOPASSWD for all commands, to execute commands. Select this connection type if the username you are connecting with does not have the right permissions to manipulate the files that need to be manipulated and/or to execute the commands that need to be executed.
    If this connection type is selected, the sudoUsername connection property is required and specifies that user that does have the necessary permissions. See below for a more detailed description.
  • INTERACTIVE_SUDO - uses SCP to transfer files, to a Unix host. Uses the sudo command, not been configured with NOPASSWD for all commands, to execute commands. This is similar to the SUDO connection type but also detects the password prompt that is shown by the sudo command when the login user (username) tries to execute a commands as the privileged user (sudoUsername) when that command has not been configured in /etc/sudoers with NOPASSWD.
    N.B.: Because the password of the login user is needed to answer this prompt, this connection type is incompatible with the privateKeyFile option that can be used to authenticate with a private key file.
sudoUsername The username of the user that can manipulate the files that need to be manipulated and that can execute the commands that need to be executed. This connection options is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
privateKeyFile The RSA private key file to use when connecting to the remote host. When this connection option is specified, the password connection option is ignored.
passphrase The passphrase to unlock the RSA private key file specified with the privateKeyFile connection option. If this connection option is not specified, the RSA private key file must have an empty passphrase.
allocateDefaultPty If set to true, the SSH server is requested to allocate a pty (pseudo terminal) for the process, as if the allocatePty option were set to the value dummy:80:24:0:0. The default value is false.
N.B.: This option is needed for some commands when they perform interaction with the user, most notably many implementations of `sudo` (the error message sorry, you must have a tty to run sudo will appear in the output otherwise).
N.B.: Some SSH servers will crash when they are requested to allocate a pty, most notably OpenSSH on AIX. To verify the behaviour of your SSH server, you can manually execute the ssh command with the -T (disable pty allocation) or -t (force pty allocation) flags.
allocatePty If set to a non-null value, the SSH server is requested to allocate a pty (pseudo terminal) for the process with the setting specified by this option. The format is TERM:COLS:ROWS:WIDTH:HEIGHT, e.g. xterm:80:24:0:0. If set, this option overrides the allocateDefaultPty option. The default value is null.
N.B.: This option is needed for some commands when they perform interaction with the user, most notably many implementations of `sudo` (the error message sorry, you must have a tty to run sudo will appear in the output otherwise).
N.B.: Some SSH servers will crash when they are requested to allocate a pty, most notably OpenSSH on AIX. To verify the behaviour of your SSH server, you can manually execute the ssh command with the -T (disable pty allocation) or -t (force pty allocation) flags.
interactiveKeyboardAuthRegex The regular expression to look for in keyboard-interactive prompts before sending the password. The default value is .*Password:[ ]?. When the SSH server is configured to not allow password authentication but is configured to allow keyboard-interactive authentication using passwords, Overthere will compare the interactive-keyboard prompt against this regular expression and send the `password` when they match.
sudoCommandPrefix The command to prefix to the command to be executed to execute it as sudoUsername. The string {0} is replaced with the vaulue of sudoUsername. The default value is sudo -u {0}. This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
sudoOverrideUmask If set to true, Overthere will explicitly change the permissions with chmod -R go+rX after uploading a file or directory with scp. This connection options is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
sudoPasswordPromptRegex The regular expression to be used when looking for sudo password prompts. When the connection type is set to INTERACTIVE_SUDO, Overthere will look for strings that match this regular expression in the first line of the output of a command, and send the password if a match occurs. The default value is .*[Pp]assword.*: This connection option is only applicable for the INTERACTIVE_SUDO connection type.
sudoQuoteCommand If set to true, the original command is added as one argument to the prefix configured with the sudoCommandPrefix connection option. This has the result of quoting the original command, which is needed for commands like su. Compare sudo -u privilegeduser start server1 to su privilegeduser 'start server1'. The default value is false. This connection options is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
### Host setup #### SSH To connect to a remote host using the SSH protocol, you will need to install an SSH server on that remote host. For Unix platforms, we recommend [OpenSSH](http://www.openssh.com/). It is included in all Linux distributions and most other Unix flavours. For Windows platforms two SSH servers are supported:
  • OpenSSH on Cygwin. We recommend COPSSH as a convenient packaging of OpenSSH and Cygwin. It is a free source download but since 22/11/2011 the binary installers are a paid solution.
  • WinSSHD is a commercial SSH server that has a lot of configuration options.

N.B.: The SFTP, SCP, SUDO and INTERACTIVE_SUDO connection types are only available for Unix hosts. To use SSH with Windows hosts, choose either the SFTP_CYGWIN or the SFTP_WINSSHD connection type.

#### SFTP

To use the SFTP connection type, make sure SFTP is enabled in the SSH server. This is enabled by default in most SSH servers.

 #### SFTP_CYGWIN

To use the SFTP_CYGWIN connection type, install COPSSH on your Windows host. In the COPSSH control panel, add the users as which you want to connect and select Linux shell and Sftp in the shell dropdown box. Check Password authentication and/or Public key authentication depending on the authentication method you want to use.
N.B.: Overthere will take care of the translation from Windows style paths, e.g. C:\Program Files\IBM\WebSphere\AppServer, to Cygwin-style paths, e.g. /cygdrive/C/Program Files/IBM/WebSphere/AppServer, so that your code can use Windows style paths.

#### SFTP_WINSSHD

To use the SFTP_WINSSHD connection type, install WinSSHD on your Windows host. In the Easy WinSSHD Settings control panel, add the users as which you want to connect, check the Login allowed checkbox and select Allow full access in the Virtual filesystem layout dropdown box. Alternatively you can check the Allow login to any Windows account to allow access to all Windows accounts.
N.B.: Overthere will take care of the translation from Windows style paths, e.g. C:\Program Files\IBM\WebSphere\AppServer, to WinSSHD-style paths, e.g. /C/Program Files/IBM/WebSphere/AppServer, so that your code can use Windows style paths.

#### SUDO and INTERACTIVE_SUDO

To use the SUDO connection type, the /etc/sudoers configuration will have to be set up in such a way that the user configured with the connection option username can execute the commands below as the user configured with the connection option sudoUsername. The arguments passed to these commands depend on the exact usage of the Overthere connection. Check the INFO messages on the com.xebialabs.overthere.ssh.SshConnection category to see what commands get executed.

  • ls
  • cp
  • mv
  • mkdir
  • rmdir
  • rm
  • chmod
  • Any other command that you want to execute.

The commands mentioned above must be configured with the NOPASSWD setting in the /etc/sudoers file. Otherwise you will have to use the INTERACTIVE_SUDO connection type. When the INTERACTIVE_SUDO connection type is used, the first line of the output will be matched against the regular expression configured with the sudoPasswordPromptRegex connection option. If a match is found, the value of the password connection option is sent.
If the sudoPasswordPromptRegex was set incorrectly, the most common symptom is for the command to appear to hang. If you have trouble determining the proper value for the sudoPasswordPromptRegex connection option, set the log level for the com.xebialabs.overthere.ssh.SshInteractiveSudoPasswordHandlingStream category to TRACE and examine the output.

 ## CIFS

The CIFS protocol implementation of Overthere uses the CIFS protocol, also known as SMB, for file manipulation and, depending on the settings, uses either Telnet or WinRM for process execution. You will most likely not need to install new software although you might need to enable and configure some services:

  • The built-in file sharing capabilities of Windows are based on CIFS and are therefore available and enabled by default.
  • A Telnet Server is available on all Windows Server versions although it might not be enabled.
  • WinRM is available on Windows Server 2008 and up.

See the section on the host setup for more information on how to setup the remote hosts.

### Connection options

The CIFS protocol implementation of Overthere defines a number of additional connection properties, in addition to the common connection options.

connectionType Specifies what protocol is used to execute commands on the remote hsots. One of the following values must be set:
  • TELNET - uses Telnet to execute remote commands. The port connection property specifies the Telnet port to connect to. The default value is 23.
  • WINRM_HTTP - uses WinRM over HTTP to execute remote commands. The port connection property specifies the Telnet port to connect to. The default value is 5985.
  • WINRM_HTTPS - uses WinRM over HTTPS to execute remote commands. The port connection property specifies the Telnet port to connect to. The default value is 5986.
cifsPort The CIFS port to connect to. The default value is 445.
pathShareMappings The path to share mappings to use for CIFS specified as a Map<String, String>, e.g. C:\IBM\WebSphere -> WebSphere. If a path is not explicitly mapped to a share the administrative share will be used. The default value is to use no path/share mappings, i.e. to use only administrative shares.
winrmContext The context used by the WinRM server. The default value is /wsman. This connection options is only applicable for the WINRM_HTTP and WINRM_HTTPS connection types.
winrmEnvelopSize The WinRM envelop size in bytes to use. The default value is 153600. This connection options is only applicable for the WINRM_HTTP and WINRM_HTTPS connection types.
winrmLocale The WinRM locale to use. The default value is en-US. This connection options is only applicable for the WINRM_HTTP and WINRM_HTTPS connection types.
winrmTimeout The WinRM timeout to use in XML schema duration format. The default value is PT60.000S. This connection options is only applicable for the WINRM_HTTP and WINRM_HTTPS connection types.
### Host setup #### CIFS To connect to a remote host using the __CIFS__ protocol, make sure the host is reachable on port 445 and add the __username__ you are using to connect to the __Administrators__ group so that that user can access the [__administrative shares__](http://en.wikipedia.org/wiki/Administrative_share).
__N.B.:__ Overthere will take care of the translation from Windows paths, e.g. `C:\Program Files\IBM\WebSphere\AppServer`, to SMB URLs that use the administrative shares, e.g. smb://username:password@hostname/C$/Program%20Files/IBM/WebSphere/AppServer (which corresponds to the UNC path \\hostname\C$\Program Files\IBM\WebSphere\AppServer), so that your code can use Windows style paths. #### TELNET

To use the TELNET connection type, enable the Telnet Server Service according to these instructions on the Microsoft Technet site. If the remote host is running Windows Server 2003 SP1 or an x64-based version of Windows Server 2003, you will have to install the according to these instructions from the Microsoft Support site. After you have started the Telnet Server, open a command prompt as the Administrator user and enter the command tlntadmn config mode=stream to enable stream mode.

When the Telnet server is enabled any user that is in the Administrators group or that is in the TelnetClients group and that has the "Allow logon locally" privilege can log in using Telnet. See the Microsoft Technet to learn how to grant a user or group the right to logon locally on Windows Server 2008 R2.

#### WINRP_HTTP and WINRM_HTTPS

For a PowerShell script to do what is described below in one go, check Richard Downer's blog

To use the WINRM_HTTP or the WINRM_HTTPS connection type, you'll need to setup WinRM on the remote host by following these instructions:

  1. If the remote host is running Windows Server 2003 R2, you will need to enable WinRM. As the Administrator user, go to the Add/Remove System Components feature in the Control Panel and add WinRm under the section Management and Monitoring Tools.

  2. On the remote host, as the Administrator user, open a Command Prompt and follow the steps below.

  3. Configure WinRM to allow basic authentication:

     winrm set winrm/config/service/Auth @{Basic="true"}

  4. Configure WinRM to allow unencrypted SOAP messages:

     winrm set winrm/config/service @{AllowUnencrypted="true"}

  5. Configure WinRM to provide enough memory to the commands that you are going to run, e.g. 1024 MB:

     winrm set winrm/config/winrs @{MaxMemoryPerShellMB="1024"}

  6. To use the WINRM_HTTP connection type, create an HTTP WinRM listener:

     winrm create winrm/config/listener?Address=*+Transport=HTTP

  7. To use the WINRM_HTTPS connection type, follow the steps below:

    1. (optional) Create a self signed certificate for the remote host by installing selfssl.exe from the IIS 6 resource kit and running the command below or by following the instructions in this blog by Hans Olav:

       C:\Program Files\IIS Resources\SelfSSL>selfssl.exe /T /N:cn=HOSTNAME /V:3650
 Microsoft (R) SelfSSL Version 1.0
 Copyright (C) 2003 Microsoft Corporation. All rights reserved.

 Do you want to replace the SSL settings for site 1 (Y/N)?Y
 The self signed certificate was successfully assigned to site 1.

    2. Open a PowerShell window and enter the command below to find the thumbprint for the certificate for the remote host:

       PS C:\Windows\system32> Get-childItem cert:\LocalMachine\Root\ | Select-String -pattern HOSTNAME

 [Subject]
   CN=HOSTNAME

 [Issuer]
   CN=HOSTNAME

 [Serial Number]
   527E7AF9142D96AD49A10469A264E766

 [Not Before]
   5/23/2011 10:23:33 AM

 [Not After]
   5/20/2021 10:23:33 AM

 [Thumbprint]
   5C36B638BC31F505EF7F693D9A60C01551DD486F

    3. Create an HTTPS WinRM listener for the remote host using the certificate you've just found:

       winrm create winrm/config/Listener?Address=*+Transport=HTTPS @{Hostname="HOSTNAME"; CertificateThumbprint="THUMBPRINT"}

For more information on WinRM, please refer to the online documentation at Microsoft's DevCenter. As a quick reference, have a look at the list of useful commands below:

  • Do a quickconfig for WinRM: winrm qc
  • Do a quickconfig for WinRM with HTTPS: winrm qc -transport:https
  • Dump the complete WinRM configuration: winrm get winrm/config
  • View the listeners that have been configured: winrm enumerate winrm/config/listener
  • Allow all hosts to connect to the WinRM listener: winrm set winrm/config/client @{TrustedHosts="*"}
  • Allow a fixed set of hosts to connect to the WinRM listener: winrm set winrm/config/client @{TrustedHosts="host1,host2..."}
# Tunnelling

Overthere supports the tunnelling of every protocol over SSH. This can be used to reach hosts that live in a DMZ which can only be reached by connecting to a different host first. This in-between host is called the jump station. In order to configure an SSH tunnel, you need to provide a set of nested connection options specifying which host is used as the jump station.

When using a jumpstation to connect to the remote host, Overthere will dynamically allocate an available local port to use for the connection to the end station. Using an additional connection option, you can configure from which port onwards Overthere starts the allocation.

portAllocationRangeStart At which port Overthere starts finding an available local port for connecting to the end station. The default value is: 1025.
# Release History
  • Overthere 2.0.0-beta-2 (23-Mar-2012)
    • Fixed issues #39 and #40
    • Upgraded to latest jCIFS to fix issues with windows domain names and stability using tunnels
    • Set default pty to true in case of interactive sudo and no pty set
  • Overthere 2.0.0-beta-1 (5-Mar-2012)
    • Re-implemented SSH tunnels. Tunnels are now created on demand instead of the user having to specify the localPortForwards explicitly. This makes management of tunnels easier and prevents clashes.
    • Ported Overthere tests to use TestNG instead of JUnit.
  • Overthere 1.0.16 (23-Feb-2012)
    • Reverted changes made to support SSH tunnels in 1.0.14 and 1.0.15 because it did not work as well as we hoped. We are reimplementing it for Overthere 2.0 to be released early March.
    • Fixed command line encoding bugs for SSH/CYGWIN on Windows:
      • Now transforming the first element of the command line to a Cygwin path so that batch files (and executables) in specific directories (instead of on the PATH) can be executed.
      • Encoding the command line as if the target OS is UNIX because OpenSSH on Cygwin uses Windows encoding.
  • Overthere 1.0.15 (21-Feb-2012)
    • Added explicit close() method to the new OverthereConnection interface (it was a class in 1.0.13) that does not throw java.io.IOException.
  • Overthere 1.0.14 (20-Feb-2012)
    • Added support for SSH tunnels to jumpstations.
    • Added support for NTLM authentication.
    • Upgraded to SSH/J 0.7.0.
  • Overthere 1.0.13 (18-Jan-2012)
    • Masked passwords in logging.
    • Made ItestHostFactory also look for itest.properties in ~/.itest (in addition to the classpath and the current working directory).
  • Overthere 1.0.12 (12-Jan-2012)
    • Allowed forward slashes (/) to be used in Windows paths.
    • Made it possible to access non-administrative shares on Windows so that the CIFS connection methods can be used with regular user accounts. See the pathShareMappings connection option.
    • Added the allocatePty connection option to specify an explicit pseudo terminal to use.
  • Overthere 1.0.11 (09-Dec-2011)
    • Fixes to the SSH/WinSSHD implementation and a few other little fixes.
    • Added a lot of documentation.
    • Added examples project.
    • Changed license to ASLv2.
  • Overthere 1.0.10 (23-Nov-2011)
    • Added support for SSH/WinSSHD on Windows.
  • Overthere 1.0.9 (22-Nov-2011)
    • Initial public release with support for SSH on Unix as well as CIFS/TELNET, CIFS/WinRM and SSH/CYGWIN on Windows.