openvinotoolkit/openvino_tokenizers

Pull Requests Commits

Bump OV version to 2024.4.1 (#266)

akladiev committed 1 year ago

Verified 74186cda

fix memory issues (#257)

pavel-esir committed 1 year ago

Verified 3dde884b

Fixed manylinux build (#255)

ilya-lavrenov committed 1 year ago

Verified c990f9c4

Fix constants import in cli.py (#252)

pavel-esir committed 1 year ago

Verified 145f3cf1

Add UTF-8 validation (#242)

pavel-esir committed 1 year ago

Verified 88df85ba

Fixed BUILD_FAST_TOKENIZERS condition (#248)

mryzhov committed 1 year ago

Verified def3a2dd

Add Building Without FastTokenizer Info (#247) (#250)

apaniukov committed 1 year ago

Verified d082d102

Switch To BPE Backend (#235)

apaniukov committed 1 year ago

Verified 45ddc7fd

Win debug build (#218)

mryzhov committed 1 year ago

Verified f9d8b7e3

Enabled build w/o FastTokenizers (#237)

ilya-lavrenov committed 1 year ago

Verified 018a7b20

[Cmake] Usage of explicit openvino::threading (#236)

ilya-lavrenov committed 1 year ago

Verified 9a6090d7

Fix infinite loop in split for empty split_pattern (#233)

pavel-esir committed 1 year ago

Verified e4c575b4

[TF FE][Tokenizers] Optimize TF FE extensions (#232)

rkazants committed 1 year ago

Verified 612cba80

fix segfault (#231)

pavel-esir committed 1 year ago

Verified f7cd828f

reimplement BPE tokenizer (#220)

pavel-esir committed 1 year ago

Verified 590fc064

Add Truncation To Sentencepiece (#225)

apaniukov committed 1 year ago

Verified c8083652

Use OV threading backend instead of direct TBB (#230)

ilya-lavrenov committed 1 year ago

Verified 016aa8e3

Ensure that oneTBB is found (#229)

ilya-lavrenov committed 1 year ago

Verified aa3c0324

[TF FE][Tokenizers] Avoid dependency from TF FE in tokenizers (#227)

rkazants committed 1 year ago

Verified b2934e78

Fix segfault in PCRE2 (#223)

pavel-esir committed 1 year ago

Verified 9a4d2a9c

Revert "Revert "Ops Optimization (#219)" (#222)" (#226)

apaniukov committed 1 year ago

Verified ae175fa2

Revert "Ops Optimization (#219)" (#222)

akladiev committed 1 year ago

Verified 5aeec7e0

Ops Optimization (#219)

apaniukov committed 1 year ago

Verified 6f5a01ab

Use setupvars.ps1 on Windows (#221)

ilya-lavrenov committed 1 year ago

Verified 79bc9480

Update Data Type For Windows (#215)

apaniukov committed 1 year ago

Verified 01070960

Conda forge fixes for 2024.3 (#213) (#217)

ilya-lavrenov committed 1 year ago

Verified b9234ce2

Add py chaches to gitignore (#210)

as-suvorov committed 1 year ago

Verified f8cb2ef3

Set minimal supported OpenVINO version (#212)

ilya-lavrenov committed 1 year ago

Verified 2f7bfb08

Avoid installation of PRCE2 artifacts together with OpenVINO Tokenizers (#211)

ilya-lavrenov committed 1 year ago

Verified f63b9629

Align Sentencepiece Model Vocab (#205)

apaniukov committed 1 year ago

Verified e8d13fd9

Older