spurplus: A Python repository from Parquery

Spur+

Spur+ is a library to manage remote machines and perform file operations over SSH.

It builds on top of Spur and Paramiko libraries. While we already find that Spur and Paramiko provide most of the functionality out-of-the-box, we missed certain features:

typing. Since spur supports both Python 2 and 3, it does not provide any type annotations which makes it harder to use with type checkers such as mypy.
pathlib.Path support. We find it easier to manipulate paths using pathlib.Path instead of plain strings. spur+ provides support for both.
a function for creating directories. spur relies on sftp client. While it is fairly straightforward to get an sftp client from spur.SshShell and create a directory, we think that it merits a wrapper function akin to pathlib.Path.mkdir() provided how often this functionality is needed.
reading/writing text and binary data in one go. Similarly to creating directories, spur.SshShell.open() already provides all the functionality you need to read/write files. However, we found the usage code to be more readable when written in one line and no extra variables for file descriptors are introduced.
a function for putting and getting files to/from the remote host, respectively.
a function to sync a local directory to a remote directory (similar to rsync).
a function for computing MD5 checksums.
a function to check if a file exists.
a more elaborate context manager for a temporary directory which allows for specifying prefix, suffix and base directory and gives you a pathlib.Path. In contrast, spur.temporary_directory() gives you only a string with no knobs.
an initializer function to repeatedly re-connect on connection failure. We found this function particularly important when you spin a virtual instance in the cloud and need to wait for it to initialize.
a wrapper around paramiko's SFTP client (spurplus.sftp.ReconnectingSFTP) to automatically reconnect if the SFTP client experienced a connection failure. While original spur.SshShell.open() creates a new SFTP client on every call in order to prevent issues with time-outs, spurplus.SshShell is able to re-use the SFTP client over multiple calls via spurplus.sftp.ReconnectingSFTP.

This can lead up to 10x speed-up (see the benchmark in tests/live_test.py).

Usage

import pathlib

import spurplus

# Re-try on connection failure; sftp client and the underlying spur SshShell
# are automatically closed when the shell is closed.
with spurplus.connect_with_retries(
        hostname='some-machine.example.com', username='devop') as shell:
    p = pathlib.Path('/some/directory')

    # Create a directory
    shell.mkdir(remote_path=p, parents=True, exist_ok=True)

    # Write a file
    shell.write_text(remote_path=p/'some-file', text='hello world!')

    # Read from a file
    text = shell.read_text(remote_path=p/'some-file')

    # Change the permissions
    shell.chmod(remote_path=p/'some-file', mode=0o444)

    # Sync a local directory to a remote.
    # Only differing files are uploaded,
    # files missing locally are deleted before the transfer and
    # the permissions are mirrored from the local.
    sync_to_remote(
        local_path="/some/local/directory",
        remote_path="/some/remote/directory",
        delete=spurplus.Delete.BEFORE,
        preserve_permissions = True)

    # Stat the file
    print("The stat of {}: {}".format(p/'some-file', shell.stat(p/'some-file')))

    # Use a wrapped SFTP client
    sftp = shell.as_sftp()
    # Do something with the SFTP
    for attr in sftp.listdir_attr(path=p.as_posix()):
        do_something(attr.filename, attr.st_size)

Documentation

The documentation is available on readthedocs.

Installation

Create a virtual environment:

python3 -m venv venv3

Activate it:

source venv3/bin/activate

Install spur+ with pip:

pip3 install spurplus

Development

Check out the repository.
In the repository root, create the virtual environment:

python3 -m venv venv3

Activate the virtual environment:

source venv3/bin/activate

Install the development dependencies:

pip3 install -e .[dev]

There are live tests for which you need to have a running SSH server. The parameters of the tests are passed via environment variables:
- TEST_SSH_HOSTNAME (host name of the SSH server, defaults to "127.0.0.1"),
- TEST_SSH_PORT (optional, defaults to 22),
- TEST_SSH_USERNAME (optional, uses paramiko's default),
- TEST_SSH_PASSWORD (optional, uses private key file if not specified) and
- TEST_SSH_PRIVATE_KEY_FILE (optional, looks for private key in expected places if not specified).

We use tox for testing and packaging the distribution. Assuming that the above-mentioned environment variables has been set, the virutal environment has been activated and the development dependencies have been installed, run:

tox

Pre-commit Checks

We provide a set of pre-commit checks that lint and check code for formatting.

Namely, we use:

yapf to check the formatting.
The style of the docstrings is checked with pydocstyle.
Static type analysis is performed with mypy.
Various linter checks are done with pylint.
Doctests are executed using the Python doctest module.

Run the pre-commit checks locally from an activated virtual environment with development dependencies:

./precommit.py

The pre-commit script can also automatically format the code:

./precommit.py  --overwrite

Versioning

We follow Semantic Versioning. The version X.Y.Z indicates:

X is the major version (backward-incompatible),
Y is the minor version (backward-compatible), and
Z is the patch version (backward-compatible bug fix).