commoner: A JavaScript repository from benjamn

Commoner

Commoner makes it easy to write scripts that flexibly and efficiently transpile any dialect of JavaScript into a directory structure of Node-compatible CommonJS module files.

This task is made possible by

a declarative syntax for defining how module source code should be found and processed,
the use of promises to manage an asynchronous build pipeline, and
never rebuilding modules that have already been built.

The output files can be required seamlessly by Node, or served by any static file server, or bundled together using a tool such as Browserify, WrapUp, or Stitch for delivery to a web browser.

If you pass the --relativize option, Commoner also takes care to rewrite all require calls to use relative module identifiers, so that the output files can be installed into any subdirectory of a larger project, and external tools do not have to give special treatment to top-level modules (or even know which modules are top-level and which are nested).

Commoner was derived from an earlier, more opinionated tool called Brigade that provided additional support for packaging modules together into multiple non-overlapping bundles. Commoner grew out of the realization that many tools already exist for bundling CommonJS modules, but that fewer tools focus on getting to that point.

Installation

From NPM:

npm install commoner

From GitHub:

cd path/to/node_modules
git clone git://github.com/reactjs/commoner.git
cd commoner
npm install .

Usage

Here's the output of bin/commonize --help:

Usage: commonize [options] <source directory> <output directory> [<module ID> [<module ID> ...]]

Options:

  -h, --help                               output usage information
  -V, --version                            output the version number
  -c, --config [file]                      JSON configuration file (no file means STDIN)
  -w, --watch                              Continually rebuild
  -x, --extension <js | coffee | ...>      File extension to assume when resolving module identifiers
  --relativize                             Rewrite all module identifiers to be relative
  --follow-requires                        Scan modules for required dependencies
  --cache-dir <directory>                  Alternate directory to use for disk cache
  --no-cache-dir                           Disable the disk cache
  --source-charset <utf8 | win1252 | ...>  Charset of source (default: utf8)
  --output-charset <utf8 | win1252 | ...>  Charset of output (default: utf8)

In a single sentence: the commonize command finds modules with the given module identifiers in the source directory and places a processed copy of each module into the output directory, along with processed copies of all required modules.

If you do not provide any module identifiers, commonize will process all files that it can find under the source directory that have the preferred file extension (.js by default). If your source files have a file extension other than .js, use the -x or --extension option to specify it. For example, --extension coffee to find .coffee files.

Output

Commoner prints various status messages to STDERR, so that you can see what it's doing, or figure out why it's not doing what you thought it would do.

The only information it prints to STDOUT is a JSON array of module identifiers, which includes the identifiers passed on the command line and all their dependencies. This array contains no duplicates.

Internally, each module that Commoner generates has a hash computed from the module's identifier, source code, and processing steps. Since this hash can be computed before processing takes place, Commoner is able to avoid processing a module if it has ever previously processed the same module in the same way.

If you dig into the code, you'll find that Commoner maintains a cache directory (by default, ~/.commoner/module-cache/) containing files with names like 9ffc5c853aac07bc106da1dc1b2486903ca688bf.js. When Commoner is about to process a module, it checks its hash against the file names in this directory. If no match is found, processing procedes and the resulting file is written to the cache directory with a new hash. If the appropriate hash file is already present in the cache directory, however, Commoner merely creates a hard link between the hash file and a file with a more meaningful name in the output directory.

When you pass the --watch flag to bin/commonize, Commoner avoids exiting after the first build and instead watches for changes to previously read files, printing a new JSON array of module identifiers to STDOUT each time rebuilding finishes. Thanks to the caching of processed modules, the time taken to rebuild is roughly proportional to the number of modified files.

Customization

The bin/commonize script is actually quite simple, and you can write similar scripts yourself. Let's have a look:

#!/usr/bin/env node

require("commoner").resolve(function(id) {
    var context = this;

    return context.getProvidedP().then(function(idToPath) {
        // If a module declares its own identifier using @providesModule
        // then that identifier will be a key in the idToPath object.
        if (idToPath.hasOwnProperty(id))
            return context.readFileP(idToPath[id]);
    });

}, function(id) {
    // Otherwise assume the identifier maps directly to a filesystem path.
    // The readModuleP method simply appends the preferred file extension
    // (usually .js) to the given module identifier and opens that file.
    return this.readModuleP(id);
});

The scriptable interface of the commoner module abstracts away many of the annoyances of writing a command-line script. In particular, you don't have to do any parsing of command-line arguments, and you don't have to worry about installing any dependencies other than commoner in your $NODE_PATH.

What you are responsible for, at a minimum, is telling Commoner how to find the source of a module given a module identifier, and you do this by passing callback functions to require("commoner").resolve. The script above uses two strategies that will be tried in sequence: first, it calls the helper function this.getProvidedP to retrieve an object mapping identifiers to file paths (more about this below); and, if that doesn't work, it falls back to interpreting the identifier as a path relative to the source directory.

Now, you might not care about this.getProvidedP. It's really just a proof of concept that Commoner can support modules that declare their own identifiers using the // @providesModule <identifier> syntax, and I included it by default because it doesn't make a difference unless you decide to use @providesModule. If you don't like it, you could write an even simpler script:

#!/usr/bin/env node

require("commoner").resolve(function(id) {
    return this.readModuleP(id);
});

The point is, it's entirely up to you to define how module identifiers are interpreted. In fact, the source you return doesn't even have to be valid JavaScript. It could be CoffeeScript, or LESS, or whatever language you prefer to write by hand. Commoner doesn't care what your source code looks like, because Commoner allows you to define arbitrary build steps to turn that source code into plain old CommonJS.

Let's consider the example of using LESS to write dynamic CSS modules. First, let's apply what we already know to give special meaning to .less files:

#!/usr/bin/env node

require("commoner").resolve(function(id) {
    if (isLess(id))
        return this.readFileP(id);
}, function(id) {
    return this.readModuleP(id);
});

function isLess(id) {
    return /\.less$/i.test(id);
}

All this really accomplishes is to avoid appending the .js file extension to identifiers that already have the .less extension.

Now we need to make sure the contents of .less files somehow get transformed into plain old CommonJS, and for that we need require("commoner").process:

require("commoner").resolve(function(id) {
    if (isLess(id))
        return this.readFileP(id);
}, function(id) {
    return this.readModuleP(id);
}).process(function(id, source) {
    if (isLess(id))
        return compileLessToJs(source);
    return source;
});

How should compileLessToJs be implemented? At a high level, I propose that we generate a CommonJS module that will append a new <style> tag to the <head> the first time the module is required. This suggests to me that we need to take the CSS generated by LESS and somehow embed it as a string in a CommonJS module with a small amount of boilerplate JS.

Here's a first attempt:

function compileLessToJs(less) {
    var css = require("less").render(less);
    return 'require("css").add(' + JSON.stringify(css) + ");";
}

Implementing a css module with an appropriate add method is an exercise that I will leave to the reader (hint: you may find this StackOverflow answer useful).

This almost works, but there's one problem: require("less").render does not actually return a string! For better or worse, it passes the compiled CSS to a callback function, which would make our task extremely painful if Commoner were not deeply committed to supporting asynchronous processing.

Commoner uses promises for asynchronous control flow, so we need to return a promise if we can't return a string immediately. The easiest way to make a promise is to call this.makePromise in the following style:

#!/usr/bin/env node

require("commoner").resolve(function(id) {
    if (isLess(id))
        return this.readFileP(id);
}, function(id) {
    return this.readModuleP(id);
}).process(function(id, source) {
    if (isLess(id)) {
        return this.makePromise(function(nodeStyleCallback) {
            compileLessToJs(source, nodeStyleCallback);
        });
    }
    return source;
});

function compileLessToJs(less, callback) {
    require("less").render(less, function(err, css) {
        callback(err, 'require("css").add(' + JSON.stringify(css) + ");")
    });
}

And we're done! This example was admittedly pretty involved, but if you followed it to the end you now have all the knowledge you need to write source files like sidebar.less and require them from other modules by invoking require("sidebar.less"). (And, by the way, embedding dynamic CSS modules in your JavaScript turns out to be an excellent idea.)

Generating multiple files from one source module

Commoner is not limited to generating just one output file from each source module. For example, if you want to follow best practices for producing source maps, you probably want to create a .map.json file corresponding to every .js file that you compile.

Recall that normally your .process callback returns a string (or a promise for a string) whose contents will be written as a .js file in the output directory. To write more than one file, just return an object whose keys are the file extensions of the files you want to write, and whose values are either strings or promises for strings representing the desired contents of those files.

Here's an example of generating two different files for every source module, one called <id>.map.json and the other called <id>.js:

require("commoner").resolve(function(id) {
    return this.readModuleP(id);
}).process(function(id, source) {
    var result = compile(source);
    return {
        ".map.json": JSON.stringify(result.sourceMap),
        ".js": [
            result.code,
            "//# sourceMappingURL=" + id + ".map.json"
        ].join("\n")
    };
});

Note that

return {
    ".js": source
};

would be equivalent to

return source;

so you only have to return an object when you want to generate multiple files. However, the .js key is mandatory when returning an object.

For your convenience, if you have a sequence of multiple processing functions, the values of the object returned from each step will be resolved before the object is passed along to the next processing function, so you can be sure all the values are strings (instead of promises) at the beginning of the next processing function.

Configuration

Of course, not all customization requires modifying code. Most of the time, in fact, configuration has more to do with providing different dynamic values to the same code.

For that kind of configuration, you don't need to modify your Commoner script at all, because Commoner scripts accept a flag called --config that can either specify a JSON file or (if --config is given without a file name) read a string of JSON from STDIN.

Examples:

bin/commonize source/ output/ main --config release.json
bin/commonize source/ output/ main --config debug.json
echo '{"debug":false}' | bin/commonize source/ output/ main --config
echo '{"debug":true}' | bin/commonize source/ output/ main --config /dev/stdin

This configuration object is exposed to the .resolve and .process callbacks as this.config. So, for example, if you wanted to implement minification as a processing step, you might do it like this:

require("commoner").resolve(function(id) {
    return this.readModule(id);
}).process(function(id, source) {
    if (this.config.debug)
        return source;
    return minify(source);
});

Perhaps the coolest thing about the configuration object is that Commoner generates a recursive hash of all its properties and their values which is then incorporated into every module hash. This means that two modules with the same identifier and identical source code and processing steps will have distinct hashes if built using different configuration objects.

Custom Options

You can define custom options for your script by using the option function.

require("commoner").resolve(function(id) {
    return this.readModule(id);
}).option(
    '--custom-option',
    'This is a custom option.'
).process(function(id, source) {
    if (this.options.customOption) {
        source = doCustomThing(source);
    }
    return source;
});

For more information of the options object available inside the process function see Commander.

benjamn/commoner