openvinotoolkit/openvino_tokenizers

Pull Requests Commits
Split implementation without FastTokenizer (to master) (#208)
pavel-esir committed 1 year ago
Verified 738a190e
Prot https://github.com/openvinotoolkit/openvino_tokenizers/pull/195 (#204)
Wovchena committed 1 year ago
Verified b89d05b7
use PCRE2 fallback for RegexNormalization (#193)
pavel-esir committed 1 year ago
Verified cb03cdd7
Add Reduce Size Build Example
apaniukov committed 1 year ago
c8d6f078
Support GLM-4 Detokenizer
apaniukov committed 1 year ago
5c93773b
Support GLM-4 Tokenizer
apaniukov committed 1 year ago
9046fa9a
reduced icudt.dll (#200)
mryzhov committed 1 year ago
Verified 3bbbf65c
Update src/CMakeLists.txt
mryzhov committed 1 year ago
0e9ffb96
support all oses
mryzhov committed 1 year ago
39fd99db
updated readme
mryzhov committed 1 year ago
eef80b2c
set filter over env
mryzhov committed 1 year ago
7b4d9036
test -DICU_DATA_FILTER_FILE
mryzhov committed 1 year ago
f090ac24
fixed names
mryzhov committed 1 year ago
a725f343
build win using msvc
mryzhov committed 1 year ago
86e49de0
copy icu.dll
mryzhov committed 1 year ago
85174dc1
set initial var
mryzhov committed 1 year ago
8b48b956
copy 3rd parties
mryzhov committed 1 year ago
4dbe8ed1
update readme
mryzhov committed 1 year ago
73e7ad35
set OpenVINO_DIR
mryzhov committed 1 year ago
a887f770
install build deps
mryzhov committed 1 year ago
5d824ec2
openvino_tokenizers_wheel
mryzhov committed 1 year ago
8ac579e7
cpack build on win
mryzhov committed 1 year ago
de3afea4
fixed win build
mryzhov committed 1 year ago
b9456579
set separete build folder
mryzhov committed 1 year ago
9ba5d2fb
updATE PATHS
mryzhov committed 1 year ago
0d030405
changed build paths
mryzhov committed 1 year ago
b1f4c7b7
reduced paths on windows
mryzhov committed 1 year ago
03d6e002
reduce paths
mryzhov committed 1 year ago
8142c3ac
fixed --wheel-dir
mryzhov committed 1 year ago
ecaa75db
changed filter path
mryzhov committed 1 year ago
a100a6f1
Terms Of Use Privacy Policy Refund Policy Impressum