capnmidnight/Primrose

Retokenize only the lines that have changed.

Closed this issue · 0 comments

Tokenization is the most expensive part of Primrose, and it occurs on the full document every time the document changes. For small documents, tokenization and rendering all fit within a single render frame. But for large documents, tokenization can push rendering off significantly. On the 2030 LoC main primrose.js file, tokenization took almost 19 frames to complete, leaving the UI completely unresponsive in that time.

Tokenization should only occur on a the lines of text that changed. If the whole document is being pasted, that is the whole set of lines. But if a single character is being inserted, the entire document should not have to be reparsed.