Floby/node-tokenizer

`finish` dispatched before completion

Opened this issue · 2 comments

joshjung commented

I'm trying to get some sort of output when the tokenizer has completed. end is never dispatched for me. In addition, I can listen for finish but because process.nextTick() force the final tokenization to occur on the next NodeJS frame, the finish is dispatched right before the last token is processed.

Is there a reason that _transform and _flush use process.nextTick()?

joshjung commented

For reference on the project that I was trying to use tokenizer for:

https://github.com/joshjung/slim-to-jade

I'm giving up for now because it was becoming too much work (was running into issues with tokens that crossed the chunk size for example). Let me know if you have any time @Floby to discuss! Would really, really appreciate it.

farskipper commented

Hey, I also had and issue with end not being dispatched, so I ended up making tokenizer2 which wraps through2 to ensure all the stream APIs all work as expected.