openvinotoolkit/openvino_tokenizers

Pull Requests Commits
switch build to released tarballs (#179)
mryzhov committed 1 year ago
Verified 96e7f816
Configure cmake similar to GenAI (#175)
Wovchena committed 1 year ago
Verified f0579e67
Cherry Pick For Release (#174)
apaniukov committed 1 year ago
Verified c615ec5a
Fixed conda-forge on Windows (#164)
ilya-lavrenov committed 1 year ago
Verified e5cb83bc
Build release by default (#162)
ilya-lavrenov committed 1 year ago
Verified 4a035243
Package into correct dirs (#155)
Wovchena committed 1 year ago
Verified e20e7c92
Switch default skip tokens flag behavior (#161)
slyalin committed 1 year ago
Verified 395992cd
Fix usage of protobuf_MODULE_COMPATIBLE (#159)
ilya-lavrenov committed 1 year ago
Verified 0b406fd6
Set cmake policies (#156)
ilya-lavrenov committed 1 year ago
Verified 93efb244
Propagate _GLIBCXX_USE_CXX11_ABI (#151)
Wovchena committed 1 year ago
Verified bbec3c49
Ability to use sentensepiece from conda-forge (#150)
ilya-lavrenov committed 1 year ago
Verified 200cffc1
Specify decoding for Windows
apaniukov committed 1 year ago
adfeffe6
Update Node Constructors
apaniukov committed 1 year ago
3a3f387a
Add RegexSplit Constructor
apaniukov committed 1 year ago
c82da5f2
Fix Style
apaniukov committed 1 year ago
3de62929
Update For New HF Breaking Changes
apaniukov committed 1 year ago
aa22e6a7
Make BPE and RegexSplit backward compatible
apaniukov committed 1 year ago
90bb3363
Update TF Translators
apaniukov committed 1 year ago
da6b9d8b
Add GPT-4o Tokenizer To Tests
apaniukov committed 1 year ago
7950759d
Update pass-rate and README.md
apaniukov committed 1 year ago
1098c451
Revert "Revert "Added Tokens Support for BPE Tokenizers" (#146)"
apaniukov committed 1 year ago
603d2be5
Support Punctuation Split (#145)
apaniukov committed 1 year ago
Verified fe390da6
Revert "Added Tokens Support for BPE Tokenizers" (#146)
rkazants committed 1 year ago
Verified c58b0a53
Added Tokens Support for BPE Tokenizers
apaniukov committed 1 year ago
d9789c25
WiP
apaniukov committed 1 year ago
a3e2af0d
wip
apaniukov committed 1 year ago
a70554bf
Fixed tokenizers on Windows
ilya-lavrenov committed 1 year ago
2c0c4470
Ported conda-forge Windows linkage fixes (#140)
ilya-lavrenov committed 1 year ago
Verified ad82aac1
detect conda env
mryzhov committed 1 year ago
643ba1e3
freeze scikit-build-core version
mryzhov committed 1 year ago
6bda1b8f
Terms Of Use Privacy Policy Refund Policy Impressum