/pash

PaSh: Light-touch Data-Parallel Shell Processing

Primary LanguagePythonMIT LicenseMIT

PaSh: Light-touch Data-Parallel Shell Processing

A system for parallelizing POSIX shell scripts.

Quick Jump: Repo Structure | Running PaSh | Installation | Testing | Community & More

Repo Structure

This repo hosts the core pash development. The structure is as follows:

  • annotations: DSL characterizing commands, parallelizability study, and associated annotations.
  • compiler: Shell-Dataflow translations and associated parallelization transformations.
  • docs: Design documents, tutorials, installation instructions, etc.
  • evaluation: Shell pipelines and example scripts used for the evaluation.
  • runtime: Runtime component — e.g., eager, split, and assocaited combiners.
  • scripts: Scripts related to continuous integration, deployment, and testing.

Running PaSh

To parallelize, say, ./evaluation/hello-world.sh with parallelization width of 2, from the top-level directory of the repository run:

./pa.sh ./evaluation/hello-world.sh

Run ./pa.sh --help to get more information about the available commands. Read a longer tutorial, see docs/tutorial.

Installation

Docker: The easiest way to play with pash today is using Docker:

curl img.pash.ndr.md | docker load; docker run --name pash-playground -it pash/18.04

PaSh can be found in the container's /pash directory, so run cd pash; git pull to fetch the latest updates; more information in the pash-on-docker guide.

Linux: Alternatively, if you're on an Ubuntu, run:

curl up.pash.ndr.md | bash

Tests

To execute the current tests, one-liner shell scripts, simply run:

cd compiler
./test_evaluation_scripts.sh

Community & More

Mailing Lists:

  • Discussion: Join this mailing list for discussing all things pash
  • Commits: Join this mailing list for commit notifications

Development/contributions: