Incorporating Test262

[jugglinmike]

I recently extended the Test262 project to include additional metadata for "negative" tests. Specifically: tests that are expected to produce early errors and syntax errors can now be automatically distinguished for tests that are expected to produce runtime errors.

My goal was to make the tests consumable by JavaScript parsers. I've since incorporated the tests into the JSHint project, and it's been very useful for fixing bugs and preventing regressions. I'd like to help other Free and Open Source parsers do the same.

Judging from the response to a previously-reported issue on the topic, I believe there is interest for including Test262 here. That issue was closed with Babel's inclusion of the project, but I think there is still value in testing the parser itself. If any of the maintainers can confirm their interest, I would be happy to follow up with some more specific practical questions (e.g. submodule usage, etc.) and go from there.

[hzoo]

Hey @jugglinmike! We really appreciate you taking the time to report an issue. The collaborators
on this project attempt to help as many people as possible, but we're a limited number of volunteers,
so it's possible this won't be addressed swiftly.

If you need any help, or just have general Babel or JavaScript questions, we have a vibrant Slack
community that typically always has someone willing to help. You can sign-up here
for an invite.

[hzoo]

Hey @jugglinmike thanks for reaching out 😀! I've also talked to @leobalter/@littledan a bit about this so definetely have had some interest for a while (also ref https://twitter.com/littledan/status/880436506996994048)

Also your previous link is referring to this issue currently

[jugglinmike]

Excellent :) Some initial questions:

• How should the Test262 project be included here? Some possibilities include a
  git submodule or a git checkout that is fetched as necessary at test time
  (and excluded from the project via a .gitignore file)
• How should the tests be executed? My initial guess would be to define a new
  target in the project's Makefile, but I don't know if you have portability
  concerns with that approach
• Related to the above, how should test runs be scheduled? I imagine you'll
  want a new "job" in the TravisCI system dedicated to this. They could be run
  as part of the npm test script, but this will likely be a little
  heavy-handed for general-purpose development.
• Would you like these tests to run in anything other than the current LTS
  release of Node.js?

With those answered, I should be able to put together an initial proof of
concept so that we have something more concrete to discuss.

Also your previous link is referring to this issue currently

Looks like I forgot to include the URL--whoops! I've fixed that in the original
comment.

[jugglinmike]

@hzoo I think I may have some time to give this a try over the weekend. I'm
reluctant to start without knowing your goals, though--do you have a few
moments to take a look at those questions above?

[xtuc]

I made a first try here babel/babel#4987.

[hzoo]

Prev issue I found was babel/babel#4987

How should the Test262 project be included here? Some possibilities include a
git submodule or a git checkout that is fetched as necessary at test time
(and excluded from the project via a .gitignore file)

Yeah could possibly a git submodule, I guess an npm package would be too slow to install? Otherwise a separate repo that downloads the babel repo (opposite approach) not sure, we can try it here.

How should the tests be executed? My initial guess would be to define a new
target in the project's Makefile, but I don't know if you have portability
concerns with that approach

Yeah that's fine

Related to the above, how should test runs be scheduled? I imagine you'll
want a new "job" in the TravisCI system dedicated to this. They could be run
as part of the npm test script, but this will likely be a little
heavy-handed for general-purpose development.

yeah new "job" just like we have for linting, etc. Not sure if it needs to run every PR yet since we'll need to figure out how to filter everything at first.

Would you like these tests to run in anything other than the current LTS
release of Node.js?

yeah probably just lts

All your responses are reasonable 😄

[jugglinmike]

@xtuc I saw that--very good. However, as I mentioned above, even when/if Babel runs Test262, I think there is still value in testing the parser itself.

Yeah could possibly a git submodule, I guess an npm package would be too slow
to install?

I raised this issue a while back, but nothing ever came of it. More recently, I've been thinking that Test262 may not be well-suited to typical software publication channels due to its consumption model. Folks that use the test suite generally expect to get the latest version of master on their own schedule. If the project didn't publish a new version for each commit, then consumers would be artificially gated on whatever release schedule it followed. On the other hand, supporting this level of flexibility through a package manager would require publishing a new version with every commit, which seems a bit wasteful.

Though I'm just trying to give some background on my current thinking. I'd be happy to get more input! For the time being, I'll work with a git submodule.

[jugglinmike]

I've submitted a patch for this at #654.

[hzoo]

Closed by #654

Also if you go to https://github.com/babel you can accept an invite to the org (read access so we can assign people to stuff, mostly for https://github.com/babel/proposals) but can give write access if anyone wants too cc @rwaldron who didn't accept yet either 😄

[rwaldron]

I always forget that Github made it so I have to accept "invites", vs the old way of just adding people.