/spacetoken

rust proc-macro like tokenizer with space awareness

Primary LanguageRustApache License 2.0Apache-2.0

SpaceToken

A proc-macro like tokenizer, providing a TokenTree similar from a proc macro context.

  • support for unicode identifier and operators
  • simple access to the literals