Regression and missing meta when parsing `not in` on 1.19

[lukaszsamson]

Elixir and Erlang/OTP versions

Erlang/OTP 27 [erts-15.2.5] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

Elixir 1.19.0-rc.0 (64c61df) (compiled with Erlang/OTP 27)

Operating system

any

Current behavior

1. First problem: parser crashes when not in is followed by newline

On 1.19

iex(2)> Code.string_to_quoted!("1 not in\n2")
** (FunctionClauseError) no function clause matching in :elixir_parser.meta_from_location/1    
    
    The following arguments were given to :elixir_parser.meta_from_location/1:
    
        # 1
        1
    
    (elixir 1.19.0-rc.0) lib/elixir/src/elixir_parser.yrl:676: :elixir_parser.meta_from_location/1
    (elixir 1.19.0-rc.0) lib/elixir/src/elixir_parser.yrl:756: :elixir_parser.build_op/3
    (elixir 1.19.0-rc.0) lib/elixir/src/elixir_parser.yrl:155: :elixir_parser.yeccpars2_97/7
    (elixir 1.19.0-rc.0) /Users/foo/.asdf/installs/erlang/27.3.2/lib/parsetools-2.6/include/yeccpre.hrl:66: :elixir_parser.yeccpars0/5
    iex:2: (file)

on 1.18 the same code parses

Code.string_to_quoted!("1 not in\n2")
{:not, [line: 1], [{:in, [line: 1], [1, 2]}]}

Most likely introduced in 8ac8230 which changed extra on not in token from newline count to meta from not token.

1.19

:elixir_tokenizer.tokenize(~c"1 not in\n2", 1, 1, [])
{:ok, 2, 2, [],
 [
   {:int, {2, 1, 2}, ~c"2"},
   {:eol, {1, 9, 1}},
   {:in_op, {1, 3, {1, 7, nil}}, :"not in"},
   {:int, {1, 1, 1}, ~c"1"}
 ], []}

vs 1.18

:elixir_tokenizer.tokenize(~c"1 not in\n2", 1, 1, [])
{:ok, 2, 2, [],
 [
   {:int, {2, 1, 2}, ~c"2"},
   {:eol, {1, 9, 1}},
   {:in_op, {1, 3, nil}, :"not in"},
   {:int, {1, 1, 1}, ~c"1"}
 ], []}

2. Second problem
   Previously, the the EoL count was properly passed on both in and not in tokens but newlines meta not produced on not in AST node

1.18

:elixir_tokenizer.tokenize(~c"1\n\n\nnot in 2", 1, 1, [])
{:ok, 4, 9, [],
 [
   {:int, {4, 8, 2}, ~c"2"},
   {:in_op, {4, 1, 3}, :"not in"},
   {:int, {1, 1, 1}, ~c"1"}
 ], []}

Code.string_to_quoted!("1\n\n\nnot in 2", columns: true, token_metadata: true)
{:not, [line: 4, column: 1], [{:in, [line: 4, column: 1], [1, 2]}]}

:elixir_tokenizer.tokenize(~c"1\n\n\nin 2", 1, 1, [])
{:ok, 4, 5, [],
 [{:int, {4, 4, 2}, ~c"2"}, {:in_op, {4, 1, 3}, :in}, {:int, {1, 1, 1}, ~c"1"}],
 []}
Code.string_to_quoted!("1\n\nin 2", columns: true, token_metadata: true)
{:in, [newlines: 2, line: 3, column: 1], [1, 2]}

now on 1.19 newline count is not passed on not in token. Still no newlines meta

iex(1)> :elixir_tokenizer.tokenize(~c"1\n\n\nin 2", 1, 1, [])
{:ok, 4, 5, [],
 [{:int, {4, 4, 2}, ~c"2"}, {:in_op, {4, 1, 3}, :in}, {:int, {1, 1, 1}, ~c"1"}],
 []}
iex(2)> Code.string_to_quoted!("1\n\nin 2", columns: true, token_metadata: true)
{:in, [newlines: 2, line: 3, column: 1], [1, 2]}
iex(3)> :elixir_tokenizer.tokenize(~c"1\n\n\nnot in 2", 1, 1, [])
{:ok, 4, 9, [],
 [
   {:int, {4, 8, 2}, ~c"2"},
   {:in_op, {4, 1, {4, 5, nil}}, :"not in"},
   {:int, {1, 1, 1}, ~c"1"}
 ], []}
iex(4)> Code.string_to_quoted!("1\n\nnot in 2", columns: true, token_metadata: true)
{:not, [line: 3, column: 1], [{:in, [line: 3, column: 5], [1, 2]}]}

Expected behavior

1. No crash when newline after not in
2. newlines meta correctly set on not in AST node when newline before not in