/spur.py

Run commands and manipulate files locally or over SSH using the same interface

Primary LanguagePythonBSD 2-Clause "Simplified" LicenseBSD-2-Clause

spur.py: Run commands and manipulate files locally or over SSH using the same interface

To run echo locally:

import spur

shell = spur.LocalShell()
result = shell.run(["echo", "-n", "hello"])
print(result.output) # prints hello

Executing the same command over SSH uses the same interface -- the only difference is how the shell is created:

import spur

shell = spur.SshShell(hostname="localhost", username="bob", password="password1")
with shell:
    result = shell.run(["echo", "-n", "hello"])
print(result.output) # prints hello

Installation

$ pip install spur

Shell constructors

LocalShell

Takes no arguments:

spur.LocalShell()

SshShell

Requires a hostname. Also requires some combination of a username, password and private key, as necessary to authenticate:

# Use a password
spur.SshShell(
    hostname="localhost",
    username="bob",
    password="password1"
)
# Use a private key
spur.SshShell(
    hostname="localhost",
    username="bob",
    private_key_file="path/to/private.key"
)
# Use a port other than 22
spur.SshShell(
    hostname="localhost",
    port=50022,
    username="bob",
    password="password1"
)

Optional arguments:

  • connect_timeout -- a timeout in seconds for establishing an SSH connection. Defaults to 60 (one minute).

  • missing_host_key -- by default, an error is raised when a host key is missing. One of the following values can be used to change the behaviour when a host key is missing:

    • spur.ssh.MissingHostKey.raise_error -- raise an error
    • spur.ssh.MissingHostKey.warn -- accept the host key and log a warning
    • spur.ssh.MissingHostKey.accept -- accept the host key

  • shell_type -- the type of shell used by the host. Defaults to spur.ssh.ShellTypes.sh, which should be appropriate for most Linux distributions. If the host uses a different shell, such as simpler shells often found on embedded systems, try changing shell_type to a more appropriate value, such as spur.ssh.ShellTypes.minimal. The following shell types are currently supported:

    • spur.ssh.ShellTypes.sh -- the Bourne shell. Supports all features.
    • spur.ssh.ShellTypes.minimal -- a minimal shell. Several features are unsupported:
      • Non-existent commands will not raise spur.NoSuchCommandError.
      • The following arguments to spawn and run are unsupported unless set to their default values: cwd, update_env, and store_pid.

  • look_for_private_keys -- by default, Spur will search for discoverable private key files in ~/.ssh/. Set to False to disable this behaviour.

  • load_system_host_keys -- by default, Spur will attempt to read host keys from the user's known hosts file, as used by OpenSSH, and no exception will be raised if the file can't be read. Set to False to disable this behaviour.

  • sock -- an open socket or socket-like object to use for communication to the target host. For instance:

    sock=paramiko.proxy.ProxyCommand(
    "ssh -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
    "bob@proxy.example.com nc -q0 target.example.com 22"
)

    Examples of socket-like objects include:

Shell interface

run(command, cwd, update_env, store_pid, allow_error, stdout, stderr, encoding)

Run a command and wait for it to complete. The command is expected to be a list of strings. Returns an instance of ExecutionResult.

result = shell.run(["echo", "-n", "hello"])
print(result.output) # prints hello

Note that arguments are passed without any shell expansion. For instance, shell.run(["echo", "$PATH"]) will print the literal string $PATH rather than the value of the environment variable $PATH.

Raises spur.NoSuchCommandError if trying to execute a non-existent command.

Optional arguments:

  • cwd -- change the current directory to this value before executing the command.
  • update_env -- a dict containing environment variables to be set before running the command. If there's an existing environment variable with the same name, it will be overwritten. Otherwise, it is unchanged.
  • store_pid -- if set to True when calling spawn, store the process id of the spawned process as the attribute pid on the returned process object. Has no effect when calling run.
  • allow_error -- False by default. If False, an exception is raised if the return code of the command is anything but 0. If True, a result is returned irrespective of return code.
  • stdout -- if not None, anything the command prints to standard output during its execution will also be written to stdout using stdout.write.
  • stderr -- if not None, anything the command prints to standard error during its execution will also be written to stderr using stderr.write.
  • encoding -- if set, this is used to decode any output. By default, any output is treated as raw bytes. If set, the raw bytes are decoded before writing to the passed stdout and stderr arguments (if set) and before setting the output attributes on the result.

shell.run(*args, **kwargs) should behave similarly to shell.spawn(*args, **kwargs).wait_for_result()

spawn(command, cwd, update_env, store_pid, allow_error, stdout, stderr, encoding)

Behaves the same as run except that spawn immediately returns an object representing the running process.

Raises spur.NoSuchCommandError if trying to execute a non-existent command.

open(path, mode="r")

Open the file at path. Returns a file-like object.

By default, files are opened in text mode. Appending "b" to the mode will open the file in binary mode.

For instance, to copy a binary file over SSH, assuming you already have an instance of SshShell:

with ssh_shell.open("/path/to/remote", "rb") as remote_file:
    with open("/path/to/local", "wb") as local_file:
        shutil.copyfileobj(remote_file, local_file)

Process interface

Returned by calls to shell.spawn. Has the following attributes:

  • pid -- the process ID of the process. Only available if store_pid was set to True when calling spawn.

Has the following methods:

  • is_running() -- return True if the process is still running, False otherwise.
  • stdin_write(value) -- write value to the standard input of the process.
  • wait_for_result() -- wait for the process to exit, and then return an instance of ExecutionResult. Will raise RunProcessError if the return code is not zero and shell.spawn was not called with allow_error=True.
  • send_signal(signal) -- sends the process the signal signal. Only available if store_pid was set to True when calling spawn.

Classes

ExecutionResult

ExecutionResult has the following properties:

  • return_code -- the return code of the command
  • output -- a string containing the result of capturing stdout
  • stderr_output -- a string containing the result of capturing stdout

It also has the following methods:

  • to_error() -- return the corresponding RunProcessError. This is useful if you want to conditionally raise RunProcessError, for instance:
result = shell.run(["some-command"], allow_error=True)
if result.return_code > 4:
    raise result.to_error()

RunProcessError

A subclass of RuntimeError with the same properties as ExecutionResult:

  • return_code -- the return code of the command
  • output -- a string containing the result of capturing stdout
  • stderr_output -- a string containing the result of capturing stdout

NoSuchCommandError

NoSuchCommandError has the following properties:

  • command -- the command that could not be found

API stability

Using the the terminology from Semantic Versioning, if the version of spur is X.Y.Z, then X is the major version, Y is the minor version, and Z is the patch version.

While the major version is 0, incrementing the patch version indicates a backwards compatible change. For instance, if you're using 0.3.1, then it should be safe to upgrade to 0.3.2.

Incrementing the minor version indicates a change in the API. This means that any code using previous minor versions of spur may need updating before it can use the current minor version.

Undocumented features

Some features are undocumented, and should be considered experimental. Use them at your own risk. They may not behave correctly, and their behaviour and interface may change at any time.

Troubleshooting

I get the error "Connection refused" when trying to connect to a virtual machine using a forwarded port on localhost

Try using "127.0.0.1" instead of "localhost" as the hostname.

I get the error "Connection refused" when trying to execute commands over SSH

Try connecting to the machine using SSH on the command line with the same settings. For instance, if you're using the code:

shell = spur.SshShell(
        hostname="remote",
        port=2222,
        username="bob",
        private_key_file="/home/bob/.ssh/id_rsa"
    )
with shell:
    result = shell.run(["echo", "hello"])

Try running:

ssh bob@remote -p 2222 -i /home/bob/.ssh/id_rsa

If the ssh command succeeds, make sure that the arguments to ssh.SshShell and the ssh command are the same. If any of the arguments to ssh.SshShell are dynamically generated, try hard-coding them to make sure they're set to the values you expect.

I can't spawn or run commands over SSH

If you're having trouble spawning or running commands over SSH, try passing shell_type=spur.ssh.ShellTypes.minimal as an argument to spur.SshShell. For instance:

import spur
import spur.ssh

spur.SshShell(
    hostname="localhost",
    username="bob",
    password="password1",
    shell_type=spur.ssh.ShellTypes.minimal,
)

This makes minimal assumptions about the features that the host shell supports, and is especially well-suited to minimal shells found on embedded systems. If the host shell is more fully-featured but only works with spur.ssh.ShellTypes.minimal, feel free to submit an issue.

Why don't shell features such as variables and redirection work?

Commands are run directly rather than through a shell. If you want to use any shell features such as variables and redirection, then you'll need to run those commands within an appropriate shell. For instance:

shell.run(["sh", "-c", "echo $PATH"])
shell.run(["sh", "-c", "ls | grep bananas"])