PR #2876 10X faster BPE tokenizer

10X faster BPE tokenizer #2876

ikawrakow merged 2 commits into master from ik/faster_bpe_tokenizer

10X faster BPE tokenizer

45bdb9af

ikawrakow requested a review from

ggerganov 2 years ago

akawrykow commented on 2023-08-29

Remove comment that no longer applies

849a31f1

ggerganov approved these changes on 2023-08-29

ikawrakow merged e37e69dc into master 2 years ago

ikawrakow deleted the ik/faster_bpe_tokenizer branch 2 years ago

Reviewers

ggerganov

akawrykow

Assignees

No one assigned

Labels

None yet

Milestone

No milestone