cpython
gh-153569: Add tokenizer source and cursor primitives
#153587

Merged

gh-153569: Add tokenizer source and cursor primitives #153587

pablogsal merged 3 commits into python:main from pablogsal:gh-153569-tokenizer-offset-api

pablogsal marked this pull request as ready for review 51 days ago

pablogsal requested a review from

lysnikolaou 51 days ago

pablogsal requested a review from

ericsnowcurrently 51 days ago

pablogsal requested a review 51 days ago

pablogsal requested a review from

erlend-aasland 51 days ago

pablogsal requested a review from

AA-Turner 51 days ago

pablogsal requested a review from

emmatyping 51 days ago

pablogsal requested a review from

itamaro 51 days ago

bedevere-app added awaiting core review

pablogsal marked this pull request as draft 51 days ago

bedevere-app removed awaiting core review

pablogsal force pushed from bce3caa6 to 2682fe54 51 days ago

pablogsal changed the title ~~gh-153569: Base the tokenizer API on source offsets~~ gh-153569: Add tokenizer source and cursor primitives 51 days ago

pablogsal added skip news

pablogsal marked this pull request as ready for review 51 days ago

bedevere-app added awaiting core review

pablogsal force pushed from 2682fe54 to 1bf43aaa 51 days ago

maurycy commented on 2026-07-11

pablogsal enabled auto-merge (squash) 45 days ago

disabled auto-merge 45 days ago
Manually disabled by user

pablogsal force pushed from b19b995c to 0dea29d6 45 days ago

gh-153569: Add tokenizer source and cursor primitives

697e7dd9

fixup! gh-153569: Add tokenizer source and cursor primitives

100072cd

fixup! gh-153569: Add tokenizer source and cursor primitives

4b0454f5

pablogsal force pushed from 0dea29d6 to 4b0454f5 6 days ago

pablogsal merged f54fd2ab into main 6 days ago

pablogsal deleted the gh-153569-tokenizer-offset-api branch 6 days ago

bedevere-app removed awaiting core review

Reviewers

maurycy

lysnikolaou

ericsnowcurrently

erlend-aasland

AA-Turner

emmatyping

itamaro

Assignees

No one assigned

Labels

skip news

Milestone

No milestone

cpython gh-153569: Add tokenizer source and cursor primitives #153587 Merged

gh-153569: Add tokenizer source and cursor primitives #153587

cpython
gh-153569: Add tokenizer source and cursor primitives
#153587

Merged