ruff
Add support for multi-character operator tokens to `SimpleTokenizer`
#6563

Merged

Add support for multi-character operator tokens to `SimpleTokenizer` #6563

charliermarsh merged 1 commit into main from charlie/tokenizer

charliermarsh requested a review from

MichaReiser 2 years ago

charliermarsh commented on 2023-08-14

charliermarsh force pushed from b2c1beb9 to 519e0f7f 2 years ago

charliermarsh force pushed from 519e0f7f to 6f36ed78 2 years ago

charliermarsh removed review request from

MichaReiser 2 years ago

charliermarsh marked this pull request as draft 2 years ago

charliermarsh force pushed from 6f36ed78 to eac7f9d0 2 years ago

charliermarsh force pushed from eac7f9d0 to b07cb08c 2 years ago

charliermarsh marked this pull request as ready for review 2 years ago

charliermarsh requested a review from

MichaReiser 2 years ago

MichaReiser requested changes on 2023-08-15

Add support for multi-character operator tokens to SimpleTokenizer

b073cd11

charliermarsh force pushed from b07cb08c to b073cd11 2 years ago

charliermarsh requested a review from

MichaReiser 2 years ago

MichaReiser approved these changes on 2023-08-16

charliermarsh merged 86ccdcc9 into main 2 years ago

charliermarsh deleted the charlie/tokenizer branch 2 years ago

Reviewers

MichaReiser

Assignees

No one assigned

Labels

None yet

Milestone

No milestone