ruff
2cbe1733 - Use CommentRanges in backwards lexing (#7360)

Commit
2 years ago
Use CommentRanges in backwards lexing (#7360) ## Summary The tokenizer was split into a forward and a backwards tokenizer. The backwards tokenizer uses the same names as the forwards ones (e.g. `next_token`). The backwards tokenizer gets the comment ranges that we already built to skip comments. --------- Co-authored-by: Micha Reiser <micha@reiser.io>
Author
Parents
Loading