openvinotoolkit/openvino_tokenizers

Pull Requests Commits
Update Flag
apaniukov committed 1 year ago
4fec5bc3
Update Fuzzing Script
apaniukov committed 1 year ago
3ee1fb59
Add `not_add-special-tokens` flag
apaniukov committed 1 year ago
c3c1a615
Add Sequence Post-Processor Support
apaniukov committed 1 year ago
85c491a5
Add FPS to the Plot
apaniukov committed 1 year ago
2f1e44e9
Update Readme
apaniukov committed 1 year ago
ec6ea7a7
Update Plot
apaniukov committed 1 year ago
c47ff792
Update Readme
apaniukov committed 1 year ago
45bf0526
Benchmark Update
apaniukov committed 1 year ago
8a89dce0
Update Benchmark
apaniukov committed 1 year ago
44263a36
Add CI-friendly llama3 tokenizer
apaniukov committed 1 year ago
4149b95d
Del llama3 - not available in CI
apaniukov committed 1 year ago
88987dc7
Add arguments to benchmark
apaniukov committed 1 year ago
e6cb5580
Formatting
apaniukov committed 1 year ago
4eb4c2e4
Add Benchmark
apaniukov committed 1 year ago
cb85dbe4
Update Readme With Llama3 Results
apaniukov committed 1 year ago
1e2ebf11
Add llama3 tokenizer to tests
apaniukov committed 1 year ago
f68414e8
[TF FE] Support StringSplitV2 with positive maxplit attribute (#121)
rkazants committed 1 year ago
Verified 3b73edef
[TF FE] Extend support for LookupTableFindV2 with string keys (#120)
rkazants committed 1 year ago
Verified d8a8ac4f
Optimize EqualStr operation (#117)
rkazants committed 1 year ago
Verified ae5560b2
set 2024.2 openvino version
mryzhov committed 1 year ago
00277c59
Add Missing .model File Error
apaniukov committed 1 year ago
6fa56295
[TF FE] Support StringToHashBucketFast operation (#115)
rkazants committed 1 year ago
Verified f1a34f01
Fix Test
apaniukov committed 1 year ago
fd220db1
Fix Parse
apaniukov committed 1 year ago
0298fea0
Parse QWEN EOD_TOKEN_ID
apaniukov committed 1 year ago
029a341f
Install torch CPU
ilya-lavrenov committed 1 year ago
dcc5fcdd
Add Cache to VocabEncoder
apaniukov committed 1 year ago
fd32aa50
Revert "removed version of ov"
mryzhov committed 1 year ago
1486252b
removed version of ov
mryzhov committed 1 year ago
020ed5a3
Terms Of Use Privacy Policy Refund Policy Impressum