openvinotoolkit/openvino_tokenizers

Pull Requests Commits
Conda forge fixes for 2024.3 (#213)
ilya-lavrenov committed 1 year ago
Verified b1b6f9c5
Гse PCRE2 fallback for RegexNormalization'
pavel-esir committed 1 year ago
Verified fb0157c3
Support GLM-4 Tokenizer (#202)
apaniukov committed 1 year ago
Verified a0d82034
Add Separate Special Token Handling To Sentencepiece (#197) (#198)
apaniukov committed 1 year ago
Verified e5b51dbc
reduced icudt.dll (#196) (#201)
mryzhov committed 1 year ago
Verified 49312afa
[CI] Fixed Jenkins artifacts (#195)
mryzhov committed 1 year ago
Verified 04795c1b
Update rt_info (#191)
apaniukov committed 1 year ago
Verified 2fb70004
Truncate Left Side When Left Padding Is Used (#192)
apaniukov committed 1 year ago
Verified df4fb149
Update Remaining Inputs Detection During Model Connection
apaniukov committed 1 year ago
Verified cc3e4449
permissions read-all (#189)
mryzhov committed 1 year ago
Verified 5f9ad53f
Sentencepiece Left Padding (#186)
apaniukov committed 1 year ago
Verified 0bb23c57
[CI] Build from OV wheel (#183)
mryzhov committed 1 year ago
Verified 6ff1bb0e
Fix Decoding For Long Tokens (#187)
apaniukov committed 1 year ago
Verified 41cdfe36
set BUILD_FAST_TOKENIZERS in command line
mryzhov committed 1 year ago
08aceeaa
renamed jobs
mryzhov committed 1 year ago
cc6cdf42
test matrix
mryzhov committed 1 year ago
f3d365bb
set BUILD_FAST_TOKENIZERS
mryzhov committed 1 year ago
ee3687b5
revert centos build
mryzhov committed 1 year ago
e3762d7d
wa to set external cmake option
mryzhov committed 1 year ago
b46dc2eb
make BUILD_FAST_TOKENIZERS option visible
mryzhov committed 1 year ago
33e57eb8
nvm install 17
mryzhov committed 1 year ago
386ede0e
install nvm
mryzhov committed 1 year ago
f8561815
manually install node.js
mryzhov committed 1 year ago
29790bbf
disable ccache
mryzhov committed 1 year ago
2f8833ae
remove python alias
mryzhov committed 1 year ago
689a0207
force-reinstall
mryzhov committed 1 year ago
8e132c08
replace upgrade pip version
mryzhov committed 1 year ago
95832b28
removed unused clone
mryzhov committed 1 year ago
f1361fd4
upgrade pip version
mryzhov committed 1 year ago
3ada7273
Select python version
mryzhov committed 1 year ago
21b083b0
Terms Of Use Privacy Policy Refund Policy Impressum