ruff
6dade5b9
- Tokenizer: Emit only a single bogus token (#7425)
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
2 years ago
Tokenizer: Emit only a single bogus token (#7425) **Summary** Instead of emitting a bogus token per char, we now only emit on single last bogus token. This leads to much more concise output. **Test Plan** Updated fixtures
References
#7425 - Tokenizer: Emit only a single bogus token
Author
konstin
Parents
97510c88
Loading