A batteries-included step runner library, suitable for creating migration tooling, codemods, scaffolding CLIs, etc.
Dubstep has utility functions for file system operations, Babel-based codemodding, Git operations and others.
License: MIT
Installation | Usage | API | Recipes | Motivation
yarn add @dubstep/coreimport {
Stepper,
step,
gitClone,
findFiles,
withTextFile,
getRestorePoint,
removeFile,
createRestorePoint,
} from '@dubstep/core';
import inquirer from 'inquirer';
async function run() {
const state = {name: ''};
const stepper = new Stepper([
step('name', async () => {
state.name = await inquirer.prompt({message: 'Name:', type: 'input'});
}),
step('clone', async () => {
gitClone('some-scaffold-template.git', state.name);
}),
step('customize', async () => {
const files = await findFiles('**/*.js', f => /src/.test(f));
for (const file of files) {
withTextFile(file, text => text.replace(/{{name}}/g, state.name));
}
}),
]);
stepper
.run({from: await getRestorePoint(restoreFile)})
.then(() => removeFile(reportFile))
.catch(e => createRestorePoint(reportFile, e));
}
run();All API entities are available as non-default import specifiers, e.g. import {Stepper} from '@dubstep/core';
Utilities can also be imported individually, e.g. import {findFiles} from '@dubstep/core/find-files';
import {Stepper} from '@dubstep/core';
class Stepper {
constructor(preset: Preset)
run(options: StepperOptions): Promise<any> // rejects w/ StepperError
on(type: 'progress', handler: StepperEventHandler)
off(type: 'progress', handler: StepperEventHandler)
}
type Preset = Array<Step>
type StepperOptions = ?{from: ?number, to: ?number}
type StepperEventHandler = ({index: number, total: number, step: string}) => voidA stepper can take a list of steps, run them in series and emit progress events.
import {step} from '@dubstep/core';
type step = (name: string, step: AsyncFunction) => Step;
type Step = {name: string, step: AsyncFunction};
type AsyncFunction = () => Promise<any>;A step consists of a descriptive name and an async function.
import {StepperError} from '@dubstep/core';
class StepperError extends Error {
constructor(error: Error, step: string, index: number),
step: string,
index: number,
message: string,
stack: string,
}A stepper error indicates what step failed. It can be used for resuming execution via restore points.
File system | Babel | Git | Restore points | Misc
import {findFiles} from '@dubstep/core';
type findFiles = (glob?: string, filter?: string => boolean) => Promise<Array<string>>;Resolves to a list of file names that match glob and match the condition from the filter function. Respects .gitignore.
import {moveFile} from '@dubstep/core';
type moveFile = (oldName: string, newName: string) => Promise<any>;Moves an existing file or directory to the location specified by newName. If the file specified by oldName doesn't exist, it no-ops.
import {readFile} from '@dubstep/core';
type readFile = (file: string) => Promise<string>;Reads the specified file into a UTF-8 string. If the file doesn't exist, the function throws a ENOENT error.
import {removeFile} from '@dubstep/core';
type removeFile = (file: string) => Promise<any>;Removes the specified file. If the file doesn't exist, it no-ops.
import {withIgnoreFile} from '@dubstep/core';
type withIgnoreFile = (file: string, fn: IgnoreFileMutation) => Promise<any>;
type IgnoreFileMutation = (data: Array<string>) => Promise<?Array<string>>;Opens a file, parses each line into a string, and calls fn with the array of lines. Then, writes the return value or the array back into the file.
If the file does not exist, fn is called with an empty array, and the file is created (including missing directories).
import {withJsFile} from '@dubstep/core';
type withJsFile = (file: string, fn: JsFileMutation) => Promise<any>;
type JsFileMutation = (program: BabelPath, file: string) => Promise<any>;Opens a file, parses each line into a Babel BabelPath, and calls fn with BabelPath. Then, writes the modified AST back into the file.
If the file does not exist, fn is called with a empty program BabelPath, and the file is created (including missing directories).
See the Babel handbook for more information on BabelPath's API.
import {withJsFiles} from '@dubstep/core';
type withJsFiles = (glob: string, fn: JsFileMutation) => Promise<any>;
type JsFileMutation = (program: BabelPath, file: string) => Promise<any>;Runs withJsFile only on files that match glob.
See the Babel handbook for more information on BabelPath's API.
import {withJsonFile} from '@dubstep/core';
type withJsonFile = (file: string, fn: JsonFileMutation) => Promise<any>;
type JsonFileMutation = (data: any) => Promise<any>;Opens a file, parses each line into a Javascript data structure, and calls fn with it. Then, writes the return value or modified data structure back into the file.
If the file does not exist, fn is called with an empty object, and the file is created (including missing directories).
import {withTextFile} from '@dubstep/core';
type withTextFile = (file: string, fn: TextFileMutation) => Promise<any>;
type TextFileMutation = (data: string) => Promise<?string>;Opens a file, parses each line into a string, and calls fn with it. Then, writes the return value back into the file.
If the file does not exist, fn is called with an empty string, and the file is created.
import {writeFile} from '@dubstep/core';
type writeFile = (file: string, data: string) => Promise<any>;Writes data to file. If the file doesn't exist, it's created (including missing directories)
import {ensureJsImports} from '@dubstep/core';
type ensureJsImports = (path: BabelPath, code: string) => Array<Object<string, string>>;If an import declaration in code is missing in the program, it's added. If it's already present, specifiers are added if not present. Note that the BabelPath should be for a Program node, and that it is mutated in-place.
Returns a list of maps of specifier local names. The default specifier is bound to the key default.
If a specifier is already declared in path, but there's a conflicting specifier in code, the one in path is retained and returned in the output map. For example:
// default specifier is already declared as `a`, but trying to redeclare it as `foo`
ensureJsImports(parseJs(`import a from 'a';`), `import foo from 'a'`);
// > {default: 'a'};A BabelPath can be obtained from withJsFile, withJsFiles or parseJs.
import {visitJsImport} from '@dubstep/core';
type visitJsImport = (
path: BabelPath,
code: string,
handler: (importPath: BabelPath, refPaths: Array<BabelPath>) => void)
: voidThis function is useful when applying codemods to specific modules which requires modifying the ast surrounding specific modules and their usage. This module works robustly across various styles of importing. For example:
visitJsImport(
parseJs(`
import {a} from 'a';
a('test')
console.log(a);
`),
`import {a} from 'a';`,
(importPath, refPaths) => {
// importPath corresponds to the ImportDeclaration from 'a';
// refPaths is a list of BabelPaths corresponding to the usage of the a variable
}
);import {hasImport} from '@dubstep/core';
type hasImport = (path: BabelPath<Program>, code: string) => booleanChecks if a given program node contains an import matching a string.
hasImport(
parseJs(`
import {a} from 'a';
console.log(a);
`),
`import {a} from 'a';`
); // trueimport {collapseImports} from '@dubstep/core';
type collapseImports = (path: BabelPath<Program>) => BabelPath<Program>This function collapses multiple import declarations with the same source into a single import statement by combining the specifiers. For example:
import A, {B} from 'a';
import {C, D} from 'a';
// =>
import A, {B, C, D} from 'a';import {generateJs} from '@dubstep/core';
type generateJs = (path: BabelPath) => string;Converts a Program BabelPath into a Javascript code string.
A BabelPath can be obtained from withJsFile, withJsFiles or parseJs.
import {insertJsAfter} from '@dubstep/core';
type insertJsAfter = (path: BabelPath, target: string, code: string, wildcards: Array<string>) => voidInserts the statements in code after the target statement, transferring expressions contained in the wildcards list. Note that path should be a BabelPath to a Program node..
const path = parseJs(`const a = 1;`);
insertJsAfter(path, `const a = $VALUE`, `const b = 2;`, ['$VALUE']);
// before
const a = 1;
// after
const a = 1;
const b = 2;It also supports spread wildcards:
const path = parseJs(`const a = f(1, 2, 3);`);
insertJsAfter(path, `const a = f(...$ARGS)`, `const b = 2;`, ['$ARGS']);
// before
const a = f(1, 2, 3);
// after
const a = f(1, 2, 3);
const b = 2;import {insertJsBefore} from '@dubstep/core';
type insertJsBefore = (path: BabelPath, target: string, code: string, wildcards: Array<string>) => voidInserts the statements in code before the target statement, transferring expressions contained in the wildcards list. Note that path should be a BabelPath to a Program node..
const path = parseJs(`const a = 1;`);
insertJsBefore(path, `const a = $VALUE`, `const b = 2;`, ['$VALUE']);
// before
const a = 1;
// after
const b = 2;
const a = 1;It also supports spread wildcards:
const path = parseJs(`const a = f(1, 2, 3);`);
insertJsBefore(path, `const a = f(...$ARGS)`, `const b = 2;`, ['$ARGS']);
// before
const a = f(1, 2, 3);
// after
const b = 2;
const a = f(1, 2, 3);import {parseJs} from '@dubstep/core';
type parseJs = (code: string, options: ParserOptions) => BabelPath;
type ParserOptions = ?{mode: ?('typescript' | 'flow')};Parses a Javascript code string into a BabelPath. The default mode is flow. The parser configuration follows Postel's Law, i.e. it accepts all syntax options supported by Babel in order to maximize its versatility.
See the Babel handbook for more information on BabelPath's API.
import {parseStatement} from '@dubstep/core';
type parseStatement = (code: string, options: ParserOptions) => Node;
type ParserOptions = ?{mode: ?('typescript' | 'flow')};Parses a Javascript code statement into a Node. Similar to parseJs but extracts the statement node.
import {removeJsImports} from '@dubstep/core';
type removeJsImports = (path: BabelPath, code: string) => voidRemoves the specifiers declared in code for the relevant source. If the import declaration no longer has specifiers after that, the declaration is also removed. Note that path should be a BabelPath for a Program node.
In addition, it removes all statements that reference the removed specifier local binding name.
A BabelPath can be obtained from withJsFile, withJsFiles or parseJs.
import {replaceJs} from '@dubstep/core';
type replaceJs = (path: BabelPath, source: string, target: string, wildcards: Array<string>) => void;Replaces code matching source with the code in target, transferring expressions contained in the wildcards list. Note that path should be a BabelPath to a Program node.
replaceJs(
parseJs(`complex.pattern('foo', () => 'user code')`),
`complex.pattern('foo', $CALLBACK)`,
`differentPattern($CALLBACK)`,
['$CALLBACK']
);
complex.pattern('foo', () => 'user code'); // before
differentPattern(() => 'user code'); // afterIt also supports spread wildcards:
replaceJs(
parseJs('foo.bar(1, 2, 3);'),
`foo.bar(...$ARGS)`,
`transformed(...$ARGS)`,
['$ARGS']
);
foo.bar(1, 2, 3); // before
transformed(1, 2, 3); // afterimport {t} from '@dubstep/core';
t.arrayExpression([]);
// etc.This is a flow-typed version of the exports from @babel/types. It is useful for creating AST nodes and asserting on existing AST nodes.
import {gitClone} from '@dubstep/core';
type gitClone = (repo: string, target: string) => Promise<any>;Clones a repo into the target directory. If the directory exists, it no-ops.
import {gitCommit} from '@dubstep/core';
type gitCommit = (message: string) => Promise<any>;Creates a local commit containing all modified files with the specified message (but does not push it to origin).
import {createRestorePoint} from '@dubstep/core';
type createRestorePoint = (file: string, e: StepperError) => Promise<any>;Creates a restore file that stores StepperError information.
import {getRestorePoint} from '@dubstep/core';
type getRestorePoint = (file: string) => Promise<number>;Resolves to the index of the failing step recorded in a restore file.
import {exec} from '@dubstep/core';
type exec = (command: string, options: Object) => Promise<string>;Runs a CLI command in the shell and resolves to stdout output. Options provided are passed directly into execa.
const migrateA = [
step('foo', async () => gitClone('some-repo.git', 'my-thing')),
step('bar', async () => moveFile('a', 'b')),
];A task that needs to run the migrateA preset but also need to run a similar task migrateB could be expressed in terms of a new preset:
const migrateAll = [...migrateA, ...migrateB];We retain full programmatic control over the steps, and can compose presets with a high level of granularity:
const migrateAndCommit = [
...migrateA,
async () => gitCommit('migrate a'),
...migrateB,
async () => gitCommit('migrate b'),
];If a step in a preset fails, it may be desirable to resume execution of the preset from the failing step (as opposed to restarting from scratch). Resuming a preset can be useful, for example, if a manual step is needed in the middle of a migration in order to unblock further steps.
const restoreFile = 'migration-report.json';
new Stepper([
/* ... */
])
.run({
from: await getRestorePoint(restoreFile),
})
.then(() => removeFile(restoreFile), e => createRestorePoint(restoreFile, e));// fix-health-path-check.js
export const fixHealthPathCheck = async () => {
await withJsFiles(
'.',
f => f.match(/src\/.\*\.js/),
path => {
return replaceJs(path, `ctx.url === '/health'`, `ctx.path === '/health'`);
}
);
};
// index.js
import {fixHealthPathCheck} from './fix-health-path-check';
new Stepper([
step('fix health path check', () => fixHealthPathCheck()),
// ...
]).run();// fix-health-path-check.js
export const fixHealthPathCheck = async ({path}) => {
const old = '/health';
withJsFiles(
'.',
f => f.match(/src\/.*\.js/),
path => {
replaceJs(path, `ctx.url === '${old}'`, `ctx.path === '${path}'`);
}
);
return old;
};
// index.js
import {fixHealthPathCheck} from './fix-health-path-check';
const state = {path: '', old: ''};
new Stepper([
step('get path', async () => {
state.path = await inquirer.prompt({
message: 'Replace with what',
type: 'input',
});
}),
step('fix health path check', () => {
state.old = await fixIt({path: state.path});
}),
step('show old', async () => {
console.log(state.old);
}),
]).run();import template from '@babel/template';
export const compatPluginRenderPageSkeleton = ({pageSkeletonConfig}) => {
const build = template(`foo($VALUE)`);
withJsFiles(
'.',
f => f.match(/src\/.\*\.js/),
path => {
path.traverse({
FunctionExpression(path) {
if (someCondition(path)) {
path.replaceWith(build({VALUE: 1}));
}
},
});
}
);
};Since the core step runner library is agnostic of state, it's possible to use state management libraries like Redux to make complex state machines more maintainable, and to leverage the ecosystem for things like file persistence.
const rootReducer = (state, action) => ({
who: action.type === 'IDENTIFY' ? action.who : state.who || '',
});
const store = redux.createStore(
rootReducer,
createPersistenceEnhancer('state.json') // save to disk on every action
);
new Stepper([
step('who', async () => {
const who = await inquirer.prompt({message: 'who?', type: 'input'});
store.dispatch({type: 'IDENTIFY', who});
}),
step('resumable', async () => {
const {who} = store.getState(); // restore state from disk if needed
console.log(who);
}),
]).run();Maintaining Javascript codebases at scale can present some unique challenges. In large enough organizations, it's not uncommon to have dozens or even hundreds of projects. Even if projects were diligently maintained, it's common for a major version bump in a dependency to require code migrations. Keeping a large number of projects up-to-date with security updates and minimizing waste of development time on repetitive, generalizable code migrations are just some of the ways Dubstep can help maintain high code quality and productivity in large organizations.
Dubstep aims to provide reusable well-tested utilities for file operations, Javascript codemodding and other tasks related to codebase migrations and upgrades. It can also be used to implement granular scaffolding CLIs.
Dubstep aims to be a one-stop shop for generic codebase transformation tasks. It was designed by taking into consideration experience with the strengths and weaknesses of several existing tools.
Javascript-specific codemodding tools such as jscodeshift or babel-codemods can be limited when it comes to cross-file concerns and these tools don't provide adequate platforms for transformations that fall outside of the scope of Javascript parsing (for example, they can't handle JSON or .gitignore files).
Shell commands are often used for tasks involving heavy file manipulation, but large shell scripts typically suffer from poor portability/readability/testability. The Unix composition paradigm is also inefficient for composing Babel AST transformations.
Picking-and-choosing NPM packages can offer a myriad of functionality, but there's no cohesiveness between packages, and they often expose inconsistent API styles (e.g. callback-based). Dubstep can easily integrate with NPM packages by leveraging ES2017 async/await in its interfaces.
Generallly speaking, there are two schools of thought when it comes to maintaining years-long projects at large organizations.
The "don't fix what ain't broken" approach is a conservative risk management strategy. It has the benefit that it requires little maintenance effort from a project owner, but it has the downside that projects become a source of technology fragmentation over time. With this approach, all technical debt that accumulates over the years eventually needs to be paid in one big lump sum (i.e. an expensive rewrite).
Another downside is that this approach doesn't work well with a push maintenance model. Typically, dependencies in small projects are managed via a pull model, i.e. the project owner updates dependencies at their own convenience. However, in typical large cross-team monorepos, dependencies are managed via a push model, i.e. whoever makes a change to a library is responsible to rolling out version bumps and relevant codemods to all downstreams using that library.
The "always update" approach aims to keep codebases always running on the latest-and-greatest versions of their dependencies, and to reduce duplication of effort (e.g. in bug fixes across duplicated code or fragmented/similar technologies). This approach pays off technical debt incrementally, but consistently across codebases, with the help of tooling to ensure that quality in codebases remains high as improvements are made to upstream libraries (both as patches and breaking changes). The downside of this approach is that it requires a higher investment in terms of maintenance effort, but this is typically offset by offloading the cost of migrations/codemods to a platform/infrastructure team, rather than having every project team waste time on similar/repetitive manual migration tasks.
Regardless of which maintenance model an organization uses, migration tooling can be useful anywhere non-trivial improvements need to be made. Some examples include moving away from proprietary frameworks towards easier-to-hire-for open source ones, or moving away from undesirable technologies towards desirable ones (e.g. if a company decides to migrate from Angular 1.x to React for whatever reason).