PR #355 Add tokenizer test + revert to C++11

Add tokenizer test + revert to C++11 #355

ggerganov merged 2 commits into master from tokenizer-test

Add tokenizer unit test + vocab-only data for tests

ecd982dd

Revert back to C++11, avoid std::string_view in the tokenizer

a0d00bd8

ggerganov merged eb34620a into master 3 years ago

ggerganov deleted the tokenizer-test branch 3 years ago

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone