Speed up identifier and keyword lexing #29463
charliermarsh
merged
9 commits into
main
from
charlie/ruff-ascii-identifiers
charliermarsh
changed the title
charliermarsh
changed the title
Base automatically changed from
charlie/ruff-shared-postfix-atom
to
main
13 days ago
MichaReiser
commented
on 2026-10-23
MichaReiser
approved these changes
on 2026-10-23
Scan ASCII identifiers with byte lookups
d90ac200
Scan identifier bytes in fixed-size batches
755524c0
Use sixteen-byte identifier batches
91f63452
Skip keyword slicing for long ASCII identifiers
341444df
Compare keyword candidates as bytes
69ac5fc5
Recognize Python keywords with a compact hash table
ec5c926c
Simplify keyword table construction
c5db21aa
Clarify identifier scanning and generate keyword tables
bdbce68c
Document the lexer keyword hash tradeoff
6818c903
Reviewers
Assignees
No one assigned
Labels
performance
parser
Milestone
No milestone
Login to write a write a comment.
Login via GitHub