How does `tokenize` separate `SourceLine`s?
Opened this issue · 1 comments
Eisverygoodletter commented
The function tokenize
outputs an array of SourceLine
s. Is it guaranteed that if i provide it with Text
with some multiline code, every line's tokens will be mapped to one SourceLine
without any formatting/line changing behaviour?
For example, if I had this single line of oddly written piece of C code:
int do_something() { call_a_function(); int some_number = 3; return 123; }
Would very token (e.g. int
, do_something
, ()
, {
, call_a_function
...) will end up on the same SourceLine
with every token in sequence?
jgm commented
Yes -- why don't you simply try it?