PR #2831 Speedup tokenization

Speedup tokenization #2831

ggerganov merged 2 commits into master from ik/speedup_tokenization

Speedup tokenization

5021d7bc

ikawrakow requested a review from

ggerganov 2 years ago

ggerganov approved these changes on 2023-08-27

ggerganov commented on 2023-08-27

Fixit: it was missing the piece after the last found occurence

86e35115

ggerganov merged 463173a6 into master 2 years ago

ikawrakow deleted the ik/speedup_tokenization branch 2 years ago

Reviewers

ggerganov

Assignees

No one assigned

Labels

None yet

Milestone

No milestone