`tokenize` function does not provide values for tokens as demonstrated in README

[joeattardi]

Hi, great package and great package name!

Tell us about your environment

• Momoa Version:
  3.0.1

• Node Version:
  20.10.0

• npm Version:
  10.2.3

Which function(s) is causing a problem?

[ ] interpret()
[ ] iterator()
[ ] parse()
[ ] print()
[ ] traverse()
[X] tokenize()

Example JSON code that demonstrates the problem:

{ name: 'Joe', age: 42 }

What did you do?

I ran the example code from the README to tokenize this JSON and get the token values:

const { tokenize } = require("@humanwhocodes/momoa");

for (const token of tokenize(some_json_string)) {
    console.log(token.type);
    console.log(token.value);
}

What did you expect to happen?

The value of the tokens would be printed out

What actually happened?

undefined is printed for all the token values

What do you think the solution is?

Include the token values or, alternately, update the docs

[nzakas]

Oops, thanks, this is a documentation issue. An earlier version did return a value in each token, but I removed it for performance reasons and clearly forgot to update the docs.

[joeattardi]

I wasn't sure which was the right fix :) Thanks!

[nzakas]

Fixed in 6c2b9ba