openvino_tokenizers
fcfa71ed - Add UTF-8 validation (#254)

Commit
1 year ago
Add UTF-8 validation (#254) * initial * add specifying replace mode from CLI * separate layer_tests.py * fix 'replace' mode * extend test cases, add layer_tests to precommit * remove redundant lines * fix import of constants * from openvino_tokenziers.... import ...
Author
Parents
Loading