ruff
3f44d52d - Fix bogus token range in forward tokenizer (#30611)

Commit
6 hours ago
Fix bogus token range in forward tokenizer (#30611) ## Summary Today, when tokenizing part of a source file, `SimpleTokenizer` can report a final `Bogus` token extending past the requested range. This can also cause the backwards tokenizer to panic (#28113). This PR limits the final `Bogus` token to the requested range and corrects the documentation for bogus tokens. Testing: Added integration coverage, and all tokenizer integration tests pass.
Author
Parents
Loading