babel/babylon

Numeric Separator: Stage 1

hzoo opened this issue · 12 comments

hzoo commented

Advance Numeric separators proposal to Stage 1 slides (Sam Goto) (@samuelgoto)

Info

Proposed at TC39 Meeting: May 2017
Slides at the meeting: https://docs.google.com/presentation/d/1hcajTemZB2Ruo4EePOyFiva1xpyv5ukKk4aQ0B83dUA/edit#slide=id.p
Proposal Repo: https://github.com/samuelgoto/proposal-numeric-separator

Example

var thousands = 10_000; // Instead of 10000.
var credit_card_number = 1234_5678_9012_3456; // Instead of 123456789012345.
var social_security_number = 999_99_9999; // Instead of 999999999.
var pi = 3.14_15; // Instead of 3.1415
var bytes = 0b11010010_01101001_10010100_10010010; // Instead of 0b11010010011010011001010010010010.
var 0xCAFE_F00D; // Instead of 0XCAFEF00D.

Basic Rules

  • to use the _ character.
  • only one consecutive underscore is allowed.
  • only between digits (not allowed at the beginning or end of literals)

ESTree/Parsing

  • looks like we need to figure out how to change the tokenizer tt.num
  • Start reading a number at getTokenFromCode :

    babylon/src/tokenizer/index.js

    Lines 456 to 464 in 9709c8d

    case 48: // '0'
    const next = this.input.charCodeAt(this.state.pos + 1);
    if (next === 120 || next === 88) return this.readRadixNumber(16); // '0x', '0X' - hex number
    if (next === 111 || next === 79) return this.readRadixNumber(8); // '0o', '0O' - octal number
    if (next === 98 || next === 66) return this.readRadixNumber(2); // '0b', '0B' - binary number
    // Anything else beginning with a digit is an integer, octal
    // number, or float.
    case 49: case 50: case 51: case 52: case 53: case 54: case 55: case 56: case 57: // 1-9
    return this.readNumber(false);
  • calls readNumber, or readRadixNumber or readInt
  • ultimately need the value to include the _ or to just keep the regular value + track where the _ are?

Transform

  • if the AST is simple (just the value/raw) would be trivial to remove (regex replace _ with '')?

ESTree: I guess there's nothing to add to the AST spec since this changes the value of the number itself?

cc @ajklein and @domenic

rwaldron commented

@hzoo I was thinking about writing a plugin for this, as a way to get involved. I will likely need your guidance :)

  • 👍2
rwaldron commented

[Deleted all that non-sense. Everything is good now]

hzoo commented

awesome @rwaldron 😄 , already added some thoughts about the parser/transform. I think the hard part is figuring out how to change the parser itself. The transform should be trivial to remove the _

like in others https://github.com/babel/babel/blob/7.0/packages/babel-plugin-transform-es2015-sticky-regex/src/index.js

Also, can you try with yarn instead of npm install? It's possible I did screw something up yesterday though. https://github.com/babel/babylon/blob/master/CONTRIBUTING.md#contributing

I did do a babel alpha.10 release yesterday and we are using ^ so that's probably the issue

Ok we can discuss in slack if you want to join: http://slack.babeljs.io/

EDIT: the issue is that rollup has a dependency on babel v6 (not a peerDep) and we are on v7 with removed some APIs

  • 👍1
rwaldron commented

already added some thoughts about the parser/transform

I'll follow your notes in the op—thanks!

hzoo commented

Ok I found a bug after a lot of investigation.. I pinned the deps in master so can pull and it should be ok now.

rwaldron commented

I have an implementation and a little test runner that verifies all of the following:

☉ 68 tests run:
✔ 68 tests passed
✘ 0 tests failed
.............................
✔ 1_1 is 11 (module)
✔ 1_1 is 11 (script)
✔ 1_1.1_1 is 11.11 (module)
✔ 1_1.1_1 is 11.11 (script)
✔ .1_1 is 0.11 (module)
✔ .1_1 is 0.11 (script)
✔ 0x1_1 is 17 (module)
✔ 0x1_1 is 17 (script)
✔ 0xa_1 is 161 (module)
✔ 0xa_1 is 161 (script)
✔ 0xA_1 is 161 (module)
✔ 0xA_1 is 161 (script)
✔ 0b01_1 is 3 (module)
✔ 0b01_1 is 3 (script)
✔ 0b0_1_1 is 3 (module)
✔ 0b0_1_1 is 3 (script)
✔ 0o1_1 is 9 (module)
✔ 0o1_1 is 9 (script)
✔ 0o0_11 is 9 (module)
✔ 0o0_11 is 9 (script)
✔ 1_ is Invalid NumericLiteralSeparator (1:1) (module)
✔ 1_ is Invalid NumericLiteralSeparator (1:1) (script)
✔ 1_1_ is Invalid NumericLiteralSeparator (1:3) (module)
✔ 1_1_ is Invalid NumericLiteralSeparator (1:3) (script)
✔ 1_1__ is Invalid NumericLiteralSeparator (1:3) (module)
✔ 1_1__ is Invalid NumericLiteralSeparator (1:3) (script)
✔ 1__1 is Invalid NumericLiteralSeparator (1:1) (module)
✔ 1__1 is Invalid NumericLiteralSeparator (1:1) (script)
✔ 1_1_.1_1 is Invalid NumericLiteralSeparator (1:3) (module)
✔ 1_1_.1_1 is Invalid NumericLiteralSeparator (1:3) (script)
✔ 1_1._1_1 is Invalid NumericLiteralSeparator (1:4) (module)
✔ 1_1._1_1 is Invalid NumericLiteralSeparator (1:4) (script)
✔ 0x1_1_ is Invalid NumericLiteralSeparator (1:5) (module)
✔ 0x1_1_ is Invalid NumericLiteralSeparator (1:5) (script)
✔ 0xa_1_ is Invalid NumericLiteralSeparator (1:5) (module)
✔ 0xa_1_ is Invalid NumericLiteralSeparator (1:5) (script)
✔ 0x_a_1 is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0x_a_1 is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0x__1_1_ is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0x__1_1_ is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0x_1__1 is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0x_1__1 is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0x_1_1_ is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0x_1_1_ is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0o_1_1 is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0o_1_1 is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0o_11 is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0o_11 is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0o_01_1_ is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0o_01_1_ is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0b_0_1_1 is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0b_0_1_1 is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0b_01_1_ is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0b_01_1_ is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0b01_1_ is Invalid NumericLiteralSeparator (1:6) (module)
✔ 0b01_1_ is Invalid NumericLiteralSeparator (1:6) (script)
✔ 0o1_1_ is Invalid NumericLiteralSeparator (1:5) (module)
✔ 0o1_1_ is Invalid NumericLiteralSeparator (1:5) (script)
✔ 0o_1_1_ is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0o_1_1_ is Invalid NumericLiteralSeparator (1:2) (script)
✔ ._1_1 is Unexpected token (1:0) (module)
✔ ._1_1 is Unexpected token (1:0) (script)
✔ 0o01_8 is Unexpected token, expected ; (1:5) (module)
✔ 0o01_8 is Unexpected token, expected ; (1:5) (script)
✔ 0b2_1 is Expected number in radix 2 (1:2) (module)
✔ 0b2_1 is Expected number in radix 2 (1:2) (script)
✔ 0xZ_1 is Expected number in radix 16 (1:2) (module)
✔ 0xZ_1 is Expected number in radix 16 (1:2) (script)

I'm now figuring out a way to migrate my fixtures.

samuelgoto commented
rwaldron commented

I believe we converged to keeping parseInt intact and changing the Number constructor to take the new syntax (which leads to a backwards incompatible change, but seemed like an acceptable one).

That's correct, but I have to admit that I don't understand where that fits for Babylon? Guidance please :)

samuelgoto commented
hzoo commented

Yeah it wouldn't be related to the parser. After the parser change, we need to make the transform itself and then determine whether we need a new polyfill for those kinds of methods or not.

hzoo commented

@babel-bot move to babel/babel

hzoo commented

Hey @hzoo! I've moved your issue to the correct repository. Please make sure to keep an eye on the new issue for the latest information.

  • 😄1