Incorrect column meta on `{` in map token

[lukaszsamson]

Elixir and Erlang/OTP versions

Erlang/OTP 27 [erts-15.2.5] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

Elixir 1.18.4 (compiled with Erlang/OTP 27)

Operating system

any

Current behavior

:elixir_tokenizer.tokenize(~c"%{}", 1, 1, []) |> elem(4) |> Enum.reverse()
[%{}: {1, 1, nil}, "{": {1, 1, nil}, "}": {1, 3, nil}]

Note the column is 1 instead of 2

Similar code for structs works correctly

:elixir_tokenizer.tokenize(~c"%A{}", 1, 1, []) |> elem(4) |> Enum.reverse()
[
  {:%, {1, 1, nil}},
  {:alias, {1, 2, ~c"A"}, :A},
  {:"{", {1, 3, nil}},
  {:"}", {1, 4, nil}}
]

AST is not affected by this bug:

Code.string_to_quoted!("%{}", columns: true, token_metadata: true)
{:%{}, [closing: [line: 1, column: 3], line: 1, column: 1], []}

Expected behavior

Correct column on token

[josevalim]

The tokenizer is private API. We will fix this but I want to reiterate it is private API and it should not be used.

[josevalim]

Fixed in 6a7511f. Thanks.