PR #713 Optimizes BPE Tokenization

Optimizes BPE Tokenization #713

apaniukov wants to merge 6 commits into openvinotoolkit:master from apaniukov:gigaimprovements

Improve Short String BPE

75fa7ae1

apaniukov requested a review from

pavel-esir 8 days ago

github-actions added tokenizers_cpp

github-actions added tokenizers_test

github-actions added CI

Add new cache for BPE

51e982ae

Separate Tests in CI

9488c56a

Optimize Trie

03596cb9

Shrink Cache Fingerprint

0c88f7f3

Fix Build

cf530a9d

Reviewers

pavel-esir

Assignees

No one assigned

Labels

tokenizers_cpp tokenizers_test CI

Milestone

No milestone