Steps to run test262 with Babel

Question

Steps to run test262 with Babel

Closed this issue 5 years ago · 4 comments

Not an issue per se, but after looking at https://bocoup.com/weblog/advancing-the-open-web-through-test262 I thought it would be cool if we could run Babel with the tests as well? Not sure we have the capacity to deal with all the bugs (and many that might not be possible to fix, I figured it would be at least something to think about/try).

It seems as if we could use babel-node (http://babeljs.io/docs/usage/cli/#babel-node) instead of a node path and do something like test262-harness './test262/test/**/*.js' --hostType node --hostPath './babel/packages/babel-cli/bin/babel-node.js with the es2015 preset. Will probably need babel-polyfill as well.

Will look into it more but wanted to ask if there was anything I should look out for / discussion

Answer 1 · 2016-08-19T19:54:13.000Z

hostType node should work fine with babel-node assuming it's a binary, but it looks like a JS file? In which case you probably want hostPath to point to vanilla node.exe and use hostArguments to pass the path to babel-node.

I've been thinking about the best way to support transpilers as I've been experimenting with test262 and TypeScript again. I think it has to be a separate argument from the hosts as you will want to select your underlying runtime in addition to any transpilers. My current thinking is that eshost agents can have a transform hook, something like:

function transform(code) {
  return babelTranspile(code);
}
esh.createAgent('node', { hostPath: ' ... ' }).then(agent =>
  agent.transform = transform;
  return agent.evalScript('foo::bar(1)::baz()');
)

The downsides with this approach is you may want a transform pipeline (eg. multi-stage babel or minifiers or something) and you may also want to include runtime polyfills. While you could do any of this with a single transform hook, I may want to support it in a more first-class way. It's also not clear how eshost-cli could expose this behavior on the command line. Anyway, let me know your thoughts.

Answer 2 · 2016-08-19T20:04:56.000Z

Yeah, it's actually just a js file (https://github.com/babel/babel/blob/master/packages/babel-cli/src/_babel-node.js) that you can run in place of node so normally node a.js and now babel-node a.js. I forgot it uses babel-register so it already includes the runtime polyfills as well.

The transform hook would work too and in that case we can run just the babel.transform(code, opts) call. And your right the underlying environment/runtime could be a different version of node.

While you could do any of this with a single transform hook

Yeah is there an issue with just putting everything it that single transform function? Sorry, I don't have much context on eshost or even much with test262 and just jumped in here haha

Answer 3 · 2017-08-05T23:06:23.000Z

I'm trying to use this to test the Flow parser and running into problems handling early errors. I want to ensure that the Flow parser can parse valid tests, and rejects tests that expect early errors.

as far as I can tell, it's not possible for the transform to know the test attributes. agentPool.runTest calls evalScript with test.contents, which eshost writes to a temp file, and the temp file is the only argument passed to babel-node.

since the agent doesn't know the test attributes, it can't tell whether the test was expecting an early error. If the Flow parser successfully parses when it isn't supposed to, I want to fail the test (e.g. replace the content with assert(false, 'expected a parse error');).

also, agent.transform = ... as proposed above would mean that the transform happens in the master process so it happens serially, right? that seems bad, I think the custom agent approach like babel-node is similarly flexible but parallelizes better.

Maybe setting some environment variables, like TEST262_NEGATIVE_PHASE=early TEST262_NEGATIVE_TYPE=SyntaxError, so that babel-node (or my equivalent) can handle those cases would work. But it looks like there's a remote agent which might complicate things.

Answer 4 · 2019-04-29T18:32:09.000Z

The best way to achieve this is to use a preprocessor. This will allow you to transpile the test source code and capture failures to create a Result Object to override the internal eshost agent's normal operation: https://github.com/bterlson/test262-harness#preprocessor

To create and use a preprocessor:

Setup, install:

mkdir preprocessors && cd preprocessors
npm init -y
npm install @babel/core @babel/preset-env @babel/plugin-proposal-class-properties @babel/plugin-proposal-private-methods --save
touch babel-preprocessor.js

Put this in babel-preprocessor.js

const babel = require("@babel/core");
const config = {
  "plugins": [
    "@babel/plugin-proposal-class-properties",
    // "@babel/plugin-proposal-private-methods",
  ]
};

module.exports = function(test) {
  try {
    test.contents = babel.transform(test.contents, config).code;
  } catch (error) {
    test.result = {
      stderr: `${error.name}: ${error.message}\n`,
      stdout: '',
      error
    };
  }

  return test;
};

Then run:

test262-harness --hostType=v8 --hostPath=`which v8` --preprocessor="babel-preprocessor.js" $HOME/path/to/test262/test/language/statements/class/elements/syntax/valid/*.js

Which will have the following result:

$ test262-harness --hostType=v8 --hostPath=`which v8` --preprocessor="babel-preprocessor.js" $HOME/clonez/test262/test/language/statements/class/elements/syntax/valid/*.js
FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-privatemeth-duplicate-get-set.js (default)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (141:7):

  139 |
  140 | class C {
> 141 |   get #m() {}
      |       ^
  142 |   set #m(_) {}
  143 | }
  144 |

FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-privatemeth-duplicate-get-set.js (strict mode)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (142:7):

  140 |
  141 | class C {
> 142 |   get #m() {}
      |       ^
  143 |   set #m(_) {}
  144 | }
  145 |

FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-privatemeth-duplicate-meth-nestedclassmeth.js (default)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (143:7):

  141 |   constructor() {
  142 |     class B {
> 143 |       #m() {}
      |       ^
  144 |     }
  145 |   }
  146 |

FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-privatemeth-duplicate-meth-nestedclassmeth.js (strict mode)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (144:7):

  142 |   constructor() {
  143 |     class B {
> 144 |       #m() {}
      |       ^
  145 |     }
  146 |   }
  147 |

FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-static-private-async-gen-meth-prototype.js (default)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (141:18):

  139 |
  140 | class C {
> 141 |   static async * #prototype() {}
      |                  ^
  142 | }
  143 |

FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-static-private-async-gen-meth-prototype.js (strict mode)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (142:18):

  140 |
  141 | class C {
> 142 |   static async * #prototype() {}
      |                  ^
  143 | }
  144 |

FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-static-private-async-meth-prototype.js (default)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (141:16):

  139 |
  140 | class C {
> 141 |   static async #prototype() {}
      |                ^
  142 | }
  143 |

FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-static-private-async-meth-prototype.js (strict mode)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (142:16):

  140 |
  141 | class C {
> 142 |   static async #prototype() {}
      |                ^
  143 | }
  144 |

FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-static-private-gen-meth-prototype.js (default)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (141:12):

  139 |
  140 | class C {
> 141 |   static * #prototype() {}
      |            ^
  142 | }
  143 |

FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-static-private-gen-meth-prototype.js (strict mode)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (142:12):

  140 |
  141 | class C {
> 142 |   static * #prototype() {}
      |            ^
  143 | }
  144 |

FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-static-private-meth-prototype.js (default)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (141:10):

  139 |
  140 | class C {
> 141 |   static #prototype() {}
      |          ^
  142 | }
  143 |

FAIL ../../../clonez/test262/test/language/statements/class/elements/syntax/valid/grammar-static-private-meth-prototype.js (strict mode)
  Expected no error, got SyntaxError: unknown: Support for the experimental syntax 'classPrivateMethods' isn't currently enabled (142:10):

  140 |
  141 | class C {
> 142 |   static #prototype() {}
      |          ^
  143 | }
  144 |

Ran 44 tests
32 passed
12 failed

Now! Re-open babel-preprocessor.js and uncomment the second plugin:

const babel = require("@babel/core");
const config = {
  "plugins": [
    "@babel/plugin-proposal-class-properties",
    "@babel/plugin-proposal-private-methods",
  ]
};

module.exports = function(test) {
  try {
    test.contents = babel.transform(test.contents, config).code;
  } catch (error) {
    test.result = {
      stderr: `${error.name}: ${error.message}\n`,
      stdout: '',
      error
    };
  }

  return test;
};

And re-run:

test262-harness --hostType=v8 --hostPath=`which v8` --preprocessor="babel-preprocessor.js" $HOME/path/to/test262/test/language/statements/class/elements/syntax/valid/*.js

Which will have this result:

$ test262-harness --hostType=v8 --hostPath=`which v8` --preprocessor="babel-preprocessor.js" $HOME/clonez/test262/test/language/statements/class/elements/syntax/valid/*.js
Ran 44 tests
44 passed
0 failed