ruff
86ccdcc9 - Add support for multi-character operator tokens to `SimpleTokenizer` (#6563)

Commit
2 years ago
Add support for multi-character operator tokens to `SimpleTokenizer` (#6563) ## Summary Allows for proper lexing of tokens like `->`. The main challenge is to ensure that our forward and backwards representations are the same for cases like `===`. Specifically, we want that to lex as `==` followed by `=` regardless of whether it's a forwards or backwards lex. To do so, we identify the range of the sequential characters (the full span of `===`), lex it forwards, then return the last token. ## Test Plan `cargo test`
Author
Parents
Loading