Bluehawk

Bluehawk is a markup processor for extracting and manipulating arbitrary code. With Bluehawk, you can:

Extract code examples for use in documentation
Generate formatted code examples for use in documentation
Replace "finished" code with "todo" code for a branch in a tutorial repo

💡 See our API Documentation or open an issue

Example

Say you're documenting a library. To provide code examples for library functionality, you're forced to copy & paste snippets of code from test cases you've written into your documentation. Every time an API changes, or you want to improve an example, or you want to fix a bug, you have to copy & paste those snippets again. Sooner or later you'll miss a line, or forget to copy and paste a change from your tests to the documentation, or forget to update a line highlight... because you're trying to maintain equivalent code snippets in two places at once.

What if there was a better way? What if you could write your examples in one place, and let a tool take care of removing your assertions and setup and copying the examples into your documentation? Bluehawk does exactly that.

Videos

How do you use Bluehawk in workflows? Here are a couple of short video overviews of how the MongoDB Developer Education team uses Bluehawk to create code examples:

Install

Install the CLI globally:

npm install -g bluehawk

Bluehawk Commands

Bluehawk commands come in two forms: single-line and block. Single-line commands operate upon the current line, while block commands operate upon the span of lines between the start of the command and the end of the command. Since commands aren't valid syntax in most languages, you should place them in comments -- Bluehawk will still process them. To avoid name clashes with various languages and markup frameworks, all Bluehawk commands begin and end with colons (:). The following examples demonstrate the remove command in single-line and block forms:

Single-line commands use :<command>: to markup a single line:

public class Main {
  public static void main(String[] args){
    int a = 2;
    int b = 3;
    int c = a * b;
    assert(c == 6); // :remove:
    System.out.println("Hello world!");
  }
}

Block commands use :<command>-start: and :<command>-end: to mark the beginning and end of a spanned range of lines:

public class Main {
  public static void main(String[] args){
    int a = 2;
    int b = 3;
    int c = a * b;
    // :remove-start:
    assert(c == 6);
    // :remove-end:
    System.out.println("Hello world!");
  }
}

Some commands, like remove in the examples above, don't require any arguments at all. Other commands, such as snippet, require a unique (to that file) identifier. Yet other commands, such as replace, require an attribute list of JSON objects. Pass arguments to commands by listing them after the command itself:

public class Main {
  public static void main(String[] args){
    // :snippet-start: multiply-abc
    int a = 2;
    int b = 3;
    int c = a * b;
    // :remove-start:
    assert(c == 6);
    // :remove-end:
    System.out.println("Hello world!");
    // :snippet-end:
  }
}

💡 For a summary of all of the commands available in your local installation of Bluehawk, run bluehawk list commands.

Attribute Lists

Attribute lists are JSON objects that contain additional information about a command. They must use double quotes for fields, and the opening line of an attribute list must appear on the same line as the command itself.

// :command: {
//    "field": "value"
// }
// :replace-end:

Snippet

The snippet command, also aliased as code-block, marks a range of content in a file as a snippet. You can use the snip CLI command to generate snippet files from these snippets.

Because snippet operates on ranges of content, it is only available as a block command. You must pass snippet an identifier.

Consider the following file:

Main.java:

public class Main {
  public static void main(String[] args){
    // :snippet-start: test-block
    System.out.println("Hello world!");
    // :snippet-end:
  }
}

Running the following command:

bluehawk snip Main.java -d .

Produces the following output:

Main.codeblock.test-block.java:

System.out.println("Hello world!");

State

The state command marks a range of content in a file as part of a particular state. You can use the snip or copy CLI commands with the state flag to generate output files that contain only content from a specific named state. When you use the --state flag to specify a state, all state blocks other than the specified state are removed from the output. All content not in a state block is unaffected and outputs normally. state can be helpful for managing tutorial code with multiple steps, such as a "start" state that only contains // TODO and a "final" state that contains completed implementation code.

Because state operates on ranges of content, it is only available as a block command. You must pass state at least one identifier, which determines the name of the state or states that the block belongs to. You can pass in a list of identifiers either through a space-separated list directly after the command itself, or through the id field of an attribute list.

Consider the following file:

Main.java:

public class Main {
  public static void main(String[] args){
    // :snippet-start: example
    int example = 1;
    // :state-start: hello-world
    System.out.println("Hello world!");
    // :state-end:
    // :state-start: hello-user
    System.out.println("Hello user!");
    // :state-end:
    example++;
    // :snippet-end:
  }
}

Running the following command:

bluehawk snip Main.java -d . --state hello-user

Produces the following output:

Main.codeblock.example.java:

int example = 1;
System.out.println("Hello user!");
example++;

Alternatively, running the following command:

bluehawk snip Main.java -d . --state hello-world

Produces the following output:

Main.codeblock.example.java:

int example = 1;
System.out.println("Hello world!");
example++;

State-Uncomment

The state-uncomment command combines the state and uncomment commands. In terms of syntax, state-uncomment works exactly the same as state, except one layer of commenting is removed from the entire state in produced output. Use state-uncomment to prevent executable code in a state from actually executing in the source code you use to produce output.

Because state-uncomment operates on ranges of content, it is only available as a block command.

Consider the following file:

Main.java:

public class Main {
  public static void main(String[] args){
    // :snippet-start: add-or-subtract
    int example = 1;
    // :state-start: add-one
    example++;
    // :state-end:
    // :state-uncomment-start: subtract-one
    //example--;
    // :state--uncomment-end:
    System.out.println("Example: " + example);
    // :snippet-end:
  }
}

Running the following command:

bluehawk snip Main.java -d . --state subtract-one

Produces the following output:

Main.codeblock.add-or-subtract.java:

    int example = 1;
    example--;
    System.out.println("Example: " + example);

💡 Note that Bluehawk has trimmed one layer of comments from the hello-user state in the produced code block.

With state-uncomment, you can create multiple valid end states but only run one of those states when executing your source code.

Uncomment

The uncomment command removes a single comment from the beginning of each line of the spanned range in all output.

Because uncomment operates on ranges of content, it is only available as a block command.

💡 Comments are only specified in certain language types. For example, plaintext does not have a comment syntax, so this command does nothing in plaintext.

Consider the following file:

Main.java:

public class Main {
  public static void main(String[] args){
    int example = 1;
    // :uncomment-start:
    //example--;
    // :uncomment-end:
    example++;
    System.out.println("Example: " + example);
  }
}

Running the following command:

bluehawk copy Main.java -d .

Produces the following output:

Main.java:

public class Main {
  public static void main(String[] args){
    int example = 1;
    example--;
    example++;
    System.out.println("Example: " + example);
  }
}

Replace

The replace command accepts a JSON dictionary called "terms" as input via an attribute list, and replaces occurrences string keys in the map within the spanned range with their map values in all output. You can use replace to hide implementation details like complicated class names or API endpoint URLs in generated output.

Because replace operates on ranges of content, it is only available as a block command. You must pass an attribute list containing "terms", a dictionary of strings to strings.

Consider the following file:

Main.java:

// :replace-start: {
//    "terms": {
//       "MyMainExample": "Main",
//       "www.example.com/rest/v1": "YOUR_REST_ENDPOINT_HERE"
//    }
// }

/*
 * MyMainExample -- a class that contains only a hello world main method
 * that defines a rest endpoint.
 */
public class MyMainExample {
  String rest_endpoint;

  public static void main(String[] args){
    System.out.println("Hello world!");
    rest_endpoint = "www.example.com/rest/v1"
  }
}
// :replace-end:

Running the following command:

bluehawk copy Main.java -d .

Produces the following output:

Main.java:

/*
 * Main -- a class that contains only a hello world main method
 * that defines a rest endpoint.
 */
public class Main {
  String rest_endpoint;

  public static void main(String[] args){
    System.out.println("Hello world!");
    rest_endpoint = "YOUR_REST_ENDPOINT_HERE"
  }
}

Emphasize

The emphasize command highlights marked lines in formatted output. emphasize makes it easier to keep the correct lines highlighted when you update code samples, because it calculates the highlighted line numbers for you.

You can use emphasize as either a block command or a line command.

💡 The emphasize command only applies to formatted output. Use the --format flag with Bluehawk CLI to get formatted output.

Consider the following file:

Main.java:

public class Main {
  public static void main(String[] args){
    // :code-block-start: modulo
    int dividend = 11;
    int divisor = 3;
    int modulus = dividend % divisor; // :emphasize:
    System.out.println(dividend + " % " + divisor + " = " + modulus);
    // :code-block-end:
  }
}

Running the following command:

bluehawk snip Main.java -d . --format=rst

Produces the following output:

Main.codeblock.modulo.java.code-block.rst:

.. code-block:: java
   :emphasize-lines: 3

   int dividend = 11;
   int divisor = 3;
   int modulus = dividend % divisor;
   System.out.println(dividend + " % " + divisor + " = " + modulus);

Remove

The remove command, also aliased as hide, removes the spanned range from Bluehawk output. remove can be helpful for hiding assertions and state setup from user-facing code samples.

You can use remove as either a block command or a line command.

Consider the following file:

Main.java:

public class Main {

  public static void main(String[] args){
    // :code-block-start: division
    int dividend = 11;
    int divisor = 3;
    int quotient = dividend / divisor;
    assert(quotient == 3) // :remove:
    System.out.println(dividend + " / " + divisor + " = " + quotient);
    // :code-block-end:
  }
}

Running the following command:

bluehawk snip Main.java -d .

Produces the following output:

Main.codeblock.division.java:

int dividend = 11;
int divisor = 3;
int quotient = dividend / divisor;
System.out.println(dividend + " / " + divisor + " = " + quotient);

CLI

Commands

Use commands to generate different kinds of output with Bluehawk, including code blocks, full files of code, and even error checks.

💡 Commands for the Bluehawk CLI are not the same as Bluehawk Commands, the syntax interpreted by Bluehawk to process input files.

Snip

bluehawk snip --destination <output-directory> <input-directory-or-file>

Output "snippet files" that contain only the content of code-block or snippet Bluehawk commands, named in the format <source-file-name>.codeblock.<codeblock-name>.<source-file-extension>. By default, this command generates snippets that omit all state command contents. However, you can use the --state flag to generate snippet files that include content from a single state that you specify.

Copy

bluehawk copy --destination <output-directory> <input-directory-or-file>

Output full bluehawk-processed input files, in their original directory structure, to destination directory. Binary files are copied without Bluehawk processing. You can use the --ignore flag to add gitignore-style ignore patterns that omit matched files from output. By default, this command generates output files that omit all state. However, you can use the --state flag to generate output files that include content from a single state that you specify.

Check

bluehawk check <input-directory-or-file>

Generates non-zero output if processing any input files generates a Bluehawk error, zero output otherwise. Does not generate any files: instead, check outputs directly to command line.

Flags

You can use flags to tweak the output of Bluehawk.

Ignore

Pass a pattern to the --ignore flag to omit any file that matches that pattern from Bluehawk's input files. Bluehawk will not process or generate output for any ignored file. You can use the ignore flag multiple times in a single Bluehawk execution to ignore multiple patterns. .gitignore files in the input directory tree are automatically used as ignore patterns.

State

Pass a state's id to the --state flag to include only the contents of that state, and no other states, in the generated output.

Format

Pass the name of a markup syntax to the --format flag when generating snippets to generate a formatted version of that snippet in the specified markup syntax. This command currently only supports reStructuredText syntax using the identifier rst.

Use Cases

Tested Code Examples

Imagine you want to paste some code from a unit test into your docs. You can mark up the unit test source file like this with Bluehawk commands like :snippet-start:, :snippet-end:, :remove-start:, and :remove-end::

// SomeTest.swift

// ... more tests ...
func someTest() {
    // :snippet-start: some-example
    let person = getPerson()
    // :remove-start: // hide test boilerplate from the code block
    XCTAssert(person.name != "Keith")
    // :remove-end:
    person.doSomething {
        person.doSomethingElse()
    }
    // :snippet-end:
}
// ... more tests ...

Running Bluehawk with the snip command on this file will produce a snippet file called SomeTest.codeblock.some-example.swift that looks something like this:

let person = getPerson()
person.doSomething {
    person.doSomethingElse()
}

You can now import this snippet into your documentation. Now you have the benefit of tested examples that are still easy to read in the docs.

Bluehawk markup can go into any source file, so you don't need to rig every unit test framework you use up to also extract code examples. Just use Bluehawk with the unit test framework that suits your language and your project. Heck, you don't even need a unit test framework. Use Bluehawk in your app or bash script that you run to make sure everything's still more or less working.

Checkpointed Tutorials

Suppose you have a tutorial repo that learners can clone to follow along with your tutorial from a certain starting point, say a "start" branch. You also want learners to be able to check out a "final" branch so they can see the finished project. As the tutorial developer, you would have to maintain these two state branches, which can be tedious and error prone.

To manage this process, you can use Bluehawk to mark up your tutorial source and indicate different states or checkpoints with the :state-start: and :state-end: commands:

// WelcomeViewController.swift

// ... more code ...
// :snippet-start: sign-up
@objc func signUp() {
    // :state-start: final
    setLoading(true);
    app.emailPasswordAuth.registerUser(email: email!, password: password!, completion: { [weak self](error) in
        DispatchQueue.main.async {
            self!.setLoading(false);
            ...
        }
    })
    // :state-end:
    // :state-start: start
    // TODO: Use the app's emailPasswordAuth to registerUser with the email and password.
    // When registered, call signIn().
    // :state-uncomment-end:
}
// :snippet-end:
// ... more code ...

Running bluehawk copy on this file with --state start results in a copy of WelcomeViewController.swift that looks something like this:

// WelcomeViewController.swift

// ... more code ...
@objc func signUp() {
    // TODO: Use the app's emailPasswordAuth to registerUser with the email and password.
    // When registered, call signIn().
}
// ... more code ...

Notice that you still have all of the boilerplate, but no final implementation code. Only the "TODO" is left.

Using the --state final flag produces another version of WelcomeViewController.swift that has the boilerplate and the final implementation code, but no "TODO":

// WelcomeViewController.swift

// ... more code ...
@objc func signUp() {
    setLoading(true);
    app.emailPasswordAuth.registerUser(email: email!, password: password!, completion: { [weak self](error) in
        DispatchQueue.main.async {
            self!.setLoading(false);
            ...
        }
    })
}
// ... more code ...

You can run Bluehawk on an entire directory, and each file in the repo will be copied or transformed to the destination. This makes it easy to copy one state of the entire tutorial source into another repo that learners can clone.

Plugins

You can add commands and listeners by creating a JS file or node project that implements the register() function:

// myPlugin.js
exports.register = (bluehawk) => {
  // Register a new command, :my-command:
  bluehawk.registerCommand("my-command", {
    rules: [],
    process: (request) => {
      // Execute command
    },
  });

  // Register a document listener
  bluehawk.subscribe((finishedDocument) => {
    // Do something with finishedDocument
  });
};

Usage:

bluehawk --plugin ./myPlugin source.txt

You can pass the --plugin flag multiple times to load different plugins or create a plugin that is composed of other plugins.

Usage as a Module

npm install bluehawk

Contributing

For more information about how to run, build, or test Bluehawk yourself, see CONTRIBUTING.md.

nlarew/Bluehawk

Bluehawk

Example

Videos

Install

Bluehawk Commands

Attribute Lists

Snippet

State

State-Uncomment

Uncomment

Replace

Emphasize

Remove

CLI

Commands

Snip

Copy

Check

Flags

Ignore

State

Format

Use Cases

Tested Code Examples

Checkpointed Tutorials

Plugins

Usage as a Module

Contributing