cpython
gh-153569: simplify tokenizer source and formatted-string state
#156654
Open
Open
gh-153569: simplify tokenizer source and formatted-string state #156654
pablogsal
wants to merge
15 commits into
python:main
from
pablogsal:gh-153569-tokenizer-source-ftstring-clean
gh-153569: bound tokenizer input storage
230bc5d0
gh-153569: return tokenizer tokens as source spans
f134e816
gh-153569: remove the unused tokenizer cursor and line index
14e461fe
gh-89760: avoid following junctions during test cleanup
9318772e
gh-153569: store formatted-string text as source spans
ea936bc3
gh-153569: derive token locations from the live buffer
9485f2aa
gh-153569: derive tokenizer failures from done state
1c9b9dd8
gh-153569: keep buffer relocation inside the reader
14c25d1c
gh-153569: use tokenizer source for parser diagnostics
6a2343b9
gh-153569: make the reader own interactive state
86289ab5
gh-153569: initialize file encoding without tokenizing
eaa1cbf7
gh-153569: remove redundant reader counters
5195c03d
gh-153569: let the reader construct tokenizer state
d68ced4d
gh-153569: keep only active formatted-string state
884997b9
gh-153569: assert tokenizer state invariants
9ce1fd9d
Reviewers
No reviews
Assignees
No one assigned
Labels
skip news
Milestone
No milestone
Login to write a write a comment.
Login via GitHub