jsoverson/shift-refactor

A suite of utilities to query and modify JavaScript source

Shift Refactor

shift-refactor is a suite of utility functions designed to analyze and modify JavaScript source files.

It originated as a tool to reverse engineer obfuscated JavaScript but is general-purpose enough for arbitrary transformations.

Who is this for?

Anyone who works with JavaScript ASTs (Abstract Syntax Trees). If you're not familiar with ASTs, here are a few use cases where they come in useful:

• Automatic refactoring, making sweeping changes to JavaScript source files (Developers, QA).
• Analyzing JavaScript for linting, complexity scoring, etc (Developers, QA).
• Extracting API details to auto-generate documentation or tests (Developers, QA).
• Scraping JavaScript for information or security vulnerabilities (Pen Testers, QA, Security Teams, Hacker types).
• Programmatically transforming malicious or obfuscated JavaScript (Reverse Engineers).

Status

Stable.

Installation

$ npm install shift-refactor

Usage

The script below finds and prints all literal strings in a script.

// Read 'example.js' as text
const fs = require('fs');
const src = fs.readFileSync('example.js', 'utf-8');

const { refactor } = require('.');

// Create a refactor query object
const $script = refactor(src);

// Select all `LiteralStringExpression`s
const $stringNodes = $script('LiteralStringExpression')

// Turn the string AST nodes into real JS strings
const strings = $stringNodes.codegen();

// Output the strings to the console
strings.forEach(string => console.log(string));

Advanced Example

This script takes the obfuscated source and turns it into something much more readable.

const { refactor } = require('.'); // require('shift-refactor');
const Shift = require('shift-ast');

// Obfuscated source
const src = `var a=['\x74\x61\x72\x67\x65\x74','\x73\x65\x74\x54\x61\x72\x67\x65\x74','\x77\x6f\x72\x6c\x64','\x67\x72\x65\x65\x74','\x72\x65\x61\x64\x65\x72'];var b=function(c,d){c=c-0x0;var e=a[c];return e;};(function(){class c{constructor(d){this[b('0x0')]=d;}['\x67\x72\x65\x65\x74'](){console['\x6c\x6f\x67']('\x48\x65\x6c\x6c\x6f\x20'+this[b('0x0')]);}[b('0x1')](e){this['\x74\x61\x72\x67\x65\x74']=e;}}const f=new c(b('0x2'));f[b('0x3')]();f[b('0x1')](b('0x4'));f[b('0x3')]();}());`;

const $script = refactor(src);

const strings = $script(`Script > :first-child ArrayExpression > .elements`);

const destringifyDeclarator = $script(`VariableDeclarator[binding.name="b"][init.params.items.length=2]`);

destringifyDeclarator.rename('destringify');

const destringifyOffset = destringifyDeclarator.$(`BinaryExpression > LiteralNumericExpression`);

const findIndex = (c, d) => c - destringifyOffset.first().value;

$script(`CallExpression[callee.name="destringify"]`).replace(
  node => {
    return new Shift.LiteralStringExpression({
      value: strings.get(findIndex(node.arguments[0].value)).value
    })
  }
)

$script(`[binding.name="a"]`).delete();
$script(`[binding.name="destringify"]`).delete();

$script.convertComputedToStatic();

console.log($script.print());

Query Syntax

The query syntax is from shift-query (which is a port of esquery) and closely resemble CSS selector syntax.

The following selectors are supported:

• AST node type: FunctionDeclaration
• wildcard: *
• attribute existence: [attr]
• attribute value: [attr="foo"] or [attr=123]
• attribute regex: [attr=/foo.*/]
• attribute conditons: [attr!="foo"], [attr>2], [attr<3], [attr>=2], or [attr<=3]
• nested attribute: [attr.level2="foo"]
• field: FunctionDeclaration > IdentifierExpression.name
• First or last child: :first-child or :last-child
• nth-child (no ax+b support): :nth-child(2)
• nth-last-child (no ax+b support): :nth-last-child(1)
• descendant: ancestor descendant
• child: parent > child
• following sibling: node ~ sibling
• adjacent sibling: node + adjacent
• negation: :not(ExpressionStatement)
• matches-any: :matches([attr] > :first-child, :last-child)
• subject indicator: !IfStatement > [name="foo"]
• class of AST node: :statement, :expression, :declaration, :function, or :target

Useful sites & tools

• Shift-query's online sandbox to test queries quickly.
• Shift-query CLI tool to query JavaScript on the command line.
• AST Explorer to explore JavaScript AST's visually (make sure to select "shift" on the top menu bar).
• Shift-AST.org - home of the Shift JavaScript tool suite.

API

refactor(string | Shift AST)

Create a refactor query object.

Note:

This function assumes that it is being passed complete JavaScript source or a root AST node (Script or Module) so that it can create and maintain global state.

Example

const { refactor } = require('shift-refactor');

const $script = refactor(`/* JavaScript Source *\/`);

Refactor Query Object

The API is meant to look and feel like jQuery since – like jQuery – it works with CSS-style queries and regularly accesses nodes on a tree. Each query object is both a function and an instance of the internal RefactorSession class.

Calling the query object as a function will produce a new query object, You can call a refactor query with a query to produce a new query object with the new nodes or you can call methods off the object to act on the nodes already selected. The examples prefix refactor query objects with a $ to indicate they are refactor query objects and not naked Nodes or other objects.

Example

const { refactor } = require('shift-refactor');

const $script = refactor(src);
const $variableDecls = $script('VariableDeclarationStatement')
const $bindingIdentifiers = $variableDecls('BindingIdentifier');
const names = $bindingIdentifiers.map(node => node.name);

Methods

• .$(queryOrNodes)
• .append(replacer)
• .closest(closestSelector)
• .codegen()
• .declarations()
• .delete()
• .filter(iterator)
• .find(iterator)
• .findMatchingExpression(sampleSrc)
• .findMatchingStatement(sampleSrc)
• .findOne(selectorOrNode)
• .first(selector)
• .forEach(iterator)
• .get(index)
• .logOut()
• .lookupVariable()
• .lookupVariableByName(name)
• .map(iterator)
• .nameString()
• .parents()
• .prepend(replacer)
• .print()
• .query(selector)
• .raw()
• .references()
• .rename(newName)
• .replace(replacer)
• .replaceAsync(replacer)
• .replaceChildren(query, replacer)
• .statements()
• .toJSON()
• .type()

.$(queryOrNodes)

Sub-query from selected nodes

Example

const { refactor } = require('shift-refactor');

const src = `
let a = 1;
function myFunction() {
  let b = 2, c = 3;
}
`

$script = refactor(src);

const funcDecl = $script('FunctionDeclaration[name.name="myFunction"]');
const innerIdentifiers = funcDecl.$('BindingIdentifier');
// innerIdentifiers.nodes: myFunction, b, c (note: does not include a)

.append(replacer)

Inserts the result ofreplacerafter the selected statement.

Note:

Only works on Statement nodes.

Example

const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src = `
var message = "Hello";
console.log(message);
`

$script = refactor(src);

$script('LiteralStringExpression[value="Hello"]').closest(':statement').append('debugger');

.closest(closestSelector)

Finds the closest parent node that matches the passed selector.

Example

const { refactor } = require('shift-refactor');

const src = `
function someFunction() {
  interestingFunction();
}
function otherFunction() {
  interestingFunction();
}
`

$script = refactor(src);

// finds all functions that call `interestingFunction`
const fnDecls = $script('CallExpression[callee.name="interestingFunction"]').closest('FunctionDeclaration');

.codegen()

Generates JavaScript source for the first selected node.

Example

const { refactor } = require('shift-refactor');

const src = `
for (var i=1; i < 101; i++){
  if (i % 15 == 0) console.log("FizzBuzz");
  else if (i % 3 == 0) console.log("Fizz");
  else if (i % 5 == 0) console.log("Buzz");
  else console.log(i);
}
`

$script = refactor(src);

const strings = $script("LiteralStringExpression")

console.log(strings.codegen());

.declarations()

Finds the declaration for the selected Identifier nodes.

Note:

Returns a list of Declaration objects for each selected node, not a shift-refactor query object.

Example

const { refactor } = require('shift-refactor');

const src = `
const myVariable = 2, otherVar = 3;
console.log(myVariable, otherVar);
`

$script = refactor(src);

// selects the parameters to console.log() and finds their declarations
const decls = $script('CallExpression[callee.object.name="console"][callee.property="log"] > .arguments').declarations();

.delete()

Delete nodes

Example

const { refactor } = require('shift-refactor');

$script = refactor('foo();bar();');

$script('ExpressionStatement[expression.callee.name="foo"]').delete();

.filter(iterator)

Filter selected nodes via passed iterator

Example

const { refactor } = require('shift-refactor');

const src = `
let doc = window.document;
function addListener(event, fn) {
  doc.addEventListener(event, fn);
}
`

$script = refactor(src);

const values = $script('BindingIdentifier').filter(node => node.name === 'doc');

.find(iterator)

Finds node via the passed iterator iterator

Example

const { refactor } = require('shift-refactor');

const src = `
const myMessage = "He" + "llo" + " " + "World";
`

$script = refactor(src);

$script('LiteralStringExpression')
  .find(node => node.value === 'World')
  .replace('"Reader"');

.findMatchingExpression(sampleSrc)

Finds an expression that closely matches the passed source.

Note:

Used for selecting nodes by source pattern instead of query. The passed source is parsed as a Script and the first statement is expected to be an ExpressionStatement.Matching is done by matching the properties of the parsed statement, ignoring additional properties/nodes in the source tree.

Example

const { refactor } = require('shift-refactor');

const src = `
const a = someFunction(paramOther);
const b = targetFunction(param1, param2);
`

$script = refactor(src);

const targetCallExpression = $script.findMatchingExpression('targetFunction(param1, param2)');

.findMatchingStatement(sampleSrc)

Finds a statement that matches the passed source.

Note:

Used for selecting nodes by source pattern vs query. The passed source is parsed as a Script and the first statement alone is used as the statement to match. Matching is done by matching the properties of the parsed statement, ignoring additional properties/nodes in the source tree.

Example

const { refactor } = require('shift-refactor');

const src = `
function someFunction(a,b) {
  var innerVariable = "Lots of stuff in here";
  foo(a);
  bar(b);
}
`

$script = refactor(src);

const targetDeclaration = $script.findMatchingStatement('function someFunction(a,b){}');

.findOne(selectorOrNode)

Finds and selects a single node, throwing an error if zero or more than one is found.

Note:

This is useful for when you want to target a single node but aren't sure how specific your query needs to be to target that node and only that node.

Example

const { refactor } = require('shift-refactor');

const src = `
let outerVariable = 1;
function someFunction(a,b) {
  let innerVariable = 2;
}
`

$script = refactor(src);

// This would throw, because there are multiple VariableDeclarators
// $script.findOne('VariableDeclarator');

// This won't throw because there is only one within the only FunctionDeclaration.
const innerVariableDecl = $script('FunctionDeclaration').findOne('VariableDeclarator');

.first(selector)

Returns the first selected node. Optionally takes a selector and returns the first node that matches the selector.

Example

const { refactor } = require('shift-refactor');

const src = `
func1();
func2();
func3();
`

$script = refactor(src);

const func1CallExpression = $script('CallExpression').first();

.forEach(iterator)

Iterate over selected nodes

Example

const { refactor } = require('shift-refactor');

const src = `
let a = [1,2,3,4];
`

$script = refactor(src);

$script('LiteralNumericExpression').forEach(node => node.value *= 2);

.get(index)

Get selected node at index.

Example

const { refactor } = require('shift-refactor');

const src = `
someFunction('first string', 'second string', 'third string');
`
$script = refactor(src);

const thirdString = $script('LiteralStringExpression').get(2);

.logOut()

console.log()s the selected nodes. Useful for inserting into a chain to see what nodes you are working with.

Example

const { refactor } = require('shift-refactor');

const src = `
let a = 1, b = 2;
`

$script = refactor(src);

$script("VariableDeclarator").logOut().delete();

.lookupVariable()

Looks up the Variable from the passed identifier node

Note:

ReturnsVariableobjects from shift-scope, that contain all the references and declarations for a program variable.

Example

const { refactor } = require('shift-refactor');

const src = `
const someVariable = 2, other = 3;
someVariable++;
function thisIsAVariabletoo(same, as, these) {}
`

$script = refactor(src);

// Finds all variables declared within a program
const variables = $script('BindingIdentifier').lookupVariable();

.lookupVariableByName(name)

Looks up Variables by name.

Note:

There may be multiple across a program. Variable lookup operates on the global program state. This method ignores selected nodes.

Example

const { refactor } = require('shift-refactor');

const src = `
const someVariable = 2, other = 3;
`

$script = refactor(src);

const variables = $script.lookupVariableByName('someVariable');

.map(iterator)

Transform selected nodes via passed iterator

Example

const { refactor } = require('shift-refactor');

const src = `
let doc = window.document;
function addListener(event, fn) {
  doc.addEventListener(event, fn);
}
`

$script = refactor(src);

const values = $script('BindingIdentifier').map(node => node.name);

.nameString()

Retrieve the names of the first selected node. Returns undefined for nodes without names.

Example

const { refactor } = require('shift-refactor');

const src = `
var first = 1, second = 2;
`

$script = refactor(src);
const firstName = $script('BindingIdentifier[name="first"]').nameString();

.parents()

Retrieve parent node(s)

Example

const { refactor } = require('shift-refactor');

const src = `
var a = 1, b = 2;
`

$script = refactor(src);
const declarators = $script('VariableDeclarator');
const declaration = declarators.parents();

.prepend(replacer)

Inserts the result ofreplacerbefore the selected statement.

Note:

Only works on Statement nodes.

Example

const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src = `
var message = "Hello";
console.log(message);
`

$script = refactor(src);

$script('ExpressionStatement[expression.type="CallExpression"]').prepend(new Shift.DebuggerStatement());

.print()

Generates JavaScript source for the first selected node.

Example

const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src = `
window.addEventListener('load', () => {
  lotsOfWork();
})
`

$script = refactor(src);

$script("CallExpression[callee.property='addEventListener'] > ArrowExpression")
  .replace(new Shift.IdentifierExpression({name: 'myListener'}));

console.log($script.print());

.query(selector)

Sub-query from selected nodes

Note:

synonym for .$()

.raw()

Returns the raw Shift node for the first selected node.

Example

const { refactor } = require('shift-refactor');

const src = `
const a = 2;
`

$script = refactor(src);

const declStatement = $script('VariableDeclarationStatement').raw();

.references()

Finds the references for the selected Identifier nodes.

Note:

Returns a list of Reference objects for each selected node, not a shift-refactor query object.

Example

const { refactor } = require('shift-refactor');

const src = `
let myVar = 1;
function someFunction(a,b) {
  myVar++;
  return myVar;
}
`

$script = refactor(src);

const refs = $script('BindingIdentifier[name="myVar"]').references();

.rename(newName)

Rename all references to the first selected node to the passed name.

Note:

Uses the selected node as the target, but affects the global state.

Example

const { refactor } = require('shift-refactor');

const src = `
const myVariable = 2;
myVariable++;
const other = myVariable;
function unrelated(myVariable) { return myVariable }
`
$script = refactor(src);

$script('VariableDeclarator[binding.name="myVariable"]').rename('newName');

.replace(replacer)

Replace selected node with the result of the replacer parameter

Example

const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src = `
function sum(a,b) { return a + b }
function difference(a,b) {return a - b}
`

$script = refactor(src);

$script('FunctionDeclaration').replace(node => new Shift.VariableDeclarationStatement({
  declaration: new Shift.VariableDeclaration({
    kind: 'const',
    declarators: [
      new Shift.VariableDeclarator({
        binding: node.name,
        init: new Shift.ArrowExpression({
          isAsync: false,
          params: node.params,
          body: node.body
        })
      })
    ]
  })
}))

.replaceAsync(replacer)

Async version of .replace() that supports asynchronous replacer functions

Example

const { refactor } = require('shift-refactor');

$script = refactor('var a = "hello";');

async function work() {
 await $script('LiteralStringExpression').replaceAsync(
   (node) => Promise.resolve(`"goodbye"`)
 )
}

.replaceChildren(query, replacer)

Recursively replaces child nodes until no nodes have been replaced.

Example

const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src = `
1 + 2 + 3
`

$script = refactor(src);

$script.replaceChildren(
 'BinaryExpression[left.type=LiteralNumericExpression][right.type=LiteralNumericExpression]',
 (node) => new Shift.LiteralNumericExpression({value: node.left.value + node.right.value})
);

.statements()

Returns the selects the statements for the selected nodes. Note: it will "uplevel" the inner statements of nodes with a.bodyproperty.Does nothing for nodes that have no statements property.

Example

const { refactor } = require('shift-refactor');

const src = `
console.log(1);
console.log(2);
`

$script = refactor(src);

const rootStatements = $script.statements();

.toJSON()

JSON-ifies the current selected nodes.

Example

const { refactor } = require('shift-refactor');

const src = `
(function(){ console.log("Hey")}())
`

$script = refactor(src);

const json = $script.toJSON();

.type()

Return the type of the first selected node

Example

const { refactor } = require('shift-refactor');
const Shift = require('shift-ast');

const src = `
myFunction();
`

$script = refactor(src);

const type = $script('CallExpression').type();