Jolt 2.0

This is an experimental reimplementation of Jolt in Go. The main motivation is:

to improve performance over the existing Python implementaiton while maintaining speed of development

to introduce a domain-specific language for build tasks

to improve C++ compilation times by building a complete dependency graph that cross task boundaries

Build task definition

Task recipes are written in a domain-specific language. Example

workspace "ws" {

  project "library" {
    headers = {
      "include/header.h"
    }

    sources = {
      "src/tutorial.proto",
      "src/source.cpp",
    }

    incpaths = {
      "include",
    }

    macros = {
      "DEBUG",
      "FOOBAR"
    }

    binary = "library";

    task "build" {
      // Include build rules for Clang
      inherit "builtin:c++/clang/library";

      // And generate protobufs on the fly
      inherit "builtin:c++/protobuf";

      // Also run cppcheck since it's fast
      inherit "builtin:c++/cppcheck";

      steps {
        transform headers, sources;
      }
    }

    task "clang-tidy" {
      // Include build rules for Clang Tidy
      inherit "builtin:c++/clang/tidy";

      // Skip protobufs
      rule protobuf : skip { ext = {".proto"} }

      steps {
        transform headers, sources;
      }
    }
  }
}

Workspace

The workspace scope defines attributes, rules and projects.

Project

Projects are defined in workspace scope. They may define one or multiple tasks, as well as attributes and rules. When defining a new project the current environment is forked.

Attributes

Attributes are type-safe and must be declared before use.

// Illegal (undefined)
builddir = "build.dir"

// Legal
string filename = "output.txt";

Supported types are string, list, bool. List are lists of strings.

// Declare and initialize a new list
list sources = {
  "src/main.cpp",
  "src/linux.cpp",
}

// Append item
sources += {
  "src/windows.cpp",
}list

// Remove item
sources -= {
  "src/linux.cpp",
}

// Expand wildcard to list of files
sources = glob {
  "src/*.proto",
}

// Illegal (type mismatch)
sources = "";

Tasks

When defining a new task the current environment is forked. Assigning new values to e.g. macros doesn't impact any previously defined task, nor any defined in the future.

The steps block defines sequential steps executed by the task.

transform: Applies rules defined in the current environment to listed inputs. Accepted parameters are file list identifiers.
```
transform sources;
```
script: Runs the encapsulated shell script. The script is rendered as a Go template with data from the current environment before it is executed.
```
script {
  # Run gcc for each source file
  {{range .Attributes.Sources}}gcc -c {{.}} -o {{.}}.o{{end}}
}
```

python: Runs the encapsulated Python script. The script is a Go template rendered with data from the current environment.

python {
  import subprocess

  for source in attributes.sources:
      subprocess.call(["gcc", "-c", source, "-o", source + ".o"])
}

parallel: Runs embedded steps in parallel.

parallel {
  script { echo "Hello world!" }
  python { print("Hello world!") }
}

Rules

Rules can be defined in workspace, project or task scope. Strings assigned to rule parameters are Go templates that will be rendered with data from the environment.

// Rule appending "Hello world" to all transformed .txt files
rule append {
  command = {
    // List of commands
    "cat {{.Input}} > {{.Output}}",
    "echo Hello world! >> {{.Output}}"
  }
  ext = {".txt"}
  // Files generated by the rule
  outputs = {
    "{{.Attributes.Builddir}}/{{.Input.BaseName}}.otxt"
  }
}

Inheritance

The inherit keyword merges another environment into the current environment scope, overwriting or appending attributes as applicable. Several builtin environments can be inherited:

builtin:c++/attributes
builtin:c++/clang
builtin:c++/clang/executable
builtin:c++/clang/library
builtin:c++/clang/tidy
builtin:c++/gcc
builtin:c++/gcc/executable
builtin:c++/gcc/library
builtin:c++/cppcheck
builtin:c++/protobuf

Parameterization

Parameterization is possible using the switch statement. Example:

inherit "builtin:platform";

switch platform {
  case "Linux" {
    inherit "builtin:c++/clang";
    macros = {"LINUX"}
  }
  case "WINDOWS" {
    inherit "builtin:c++/msvc";
    macros = {"WIN32"}
  }
}

Imports and Exports

WIP

Imports define a task's dependencies to other task artifacts.

// Dependencies on a task level
task "build" {
  imports {
    incpaths += {
      "*.cxx.incpaths",
    }
    macros += {
      "*.cxx.macros"
    }
  }

  artifact "api" {
    exports incpaths as "cxx.incpaths";
    exports macros as "cxx.macros";
  }

  artifact "lib" {
    exports {"lib"} as "cxx.libpaths";
    exports {"project"} as "cxx.libraries";
    collects {
      "*.a",
    }
  }
}

// Dependencies on job level
rule compile_c {
  imports {
    incpaths += {
      "cxx.incpaths",
    }
    macros += {
      "cxx.macros",
    }
  }
}

srand/go-jolt