openvinotoolkit/openvino_tokenizers

Pull Requests Commits

GHA: use official 2024.6.0 (#376)

ilya-lavrenov committed 1 year ago

Verified d99420c6

ilya-lavrenov committed 1 year ago

Verified 09bea4a4

GitHub workspace w/a on Windows (#342) (#343)

akladiev committed 1 year ago

Verified fb845e6a

[MERGE][2024.6]Reverted w/a with hardcoded paths #329 (#341)

mryzhov committed 1 year ago

Verified da2aaf09

Bump product version to 2024.6 (#336)

akladiev committed 1 year ago

Verified 26c5ec0b

print debug errors only if ENV VAR is set (#334)

pavel-esir committed 1 year ago

Verified 47852c10

Support Already Converted Llava Tokenizers (#333)

apaniukov committed 1 year ago

Verified d48da092

Utf8 turn on (#326)

pavel-esir committed 1 year ago

Verified 7ce02af4

Update Prepend Regex (#317) (#319)

apaniukov committed 1 year ago

Verified 5397aa31

Port Fixes from Master to Release (#315)

apaniukov committed 1 year ago

Verified e30c99f5

Bump actions/dependency-review-action from 4.3.5 to 4.4.0 (#306)

dependabot[bot] committed 1 year ago

Verified 92bec551

Use PCRE2 Compatible Regex Pattern (#307)

apaniukov committed 1 year ago

Verified 16da7f39

Build w/lo FastTokenizers (#305)

ilya-lavrenov committed 1 year ago

Verified 1ddb12ed

Bump actions/setup-python from 5.2.0 to 5.3.0 (#304)

dependabot[bot] committed 1 year ago

Verified 7213eac3

fix evaluate in concurrent scenario (#303)

pavel-esir committed 1 year ago

Verified 58cf87e4

Bump actions/dependency-review-action from 4.3.4 to 4.3.5 (#297)

dependabot[bot] committed 1 year ago

Verified 8ac82af0

[GHA] Use docker and Azure runners (#299)

akladiev committed 1 year ago

Verified d323ec40

Fix RegexSplit Segfault (#300)

apaniukov committed 1 year ago

Verified 6943dddf

Bump actions/checkout from 4.2.1 to 4.2.2 (#302)

dependabot[bot] committed 1 year ago

Verified ec351af5

Reimplement WordPiece tokenization (#298)

pavel-esir committed 1 year ago

Verified ffff67e2

Drop 3.8 python and add 3.13 (#296)

ilya-lavrenov committed 1 year ago

Verified 50e49b90

disable UTF validation in PCRE2 (#295)

pavel-esir committed 1 year ago

Verified 57373890

Improve BPE (#281)

pavel-esir committed 1 year ago

Verified 17fc3c1c

Remove re2 to fix segfault & allocate enough memory (#294)

pavel-esir committed 1 year ago

Verified 57b236f1

add packages versions to rt_info (#292)

pavel-esir committed 1 year ago

Verified 57f0082a

Revert "hotfix for RE2 segfaults (#284)" (#293)

rkazants committed 1 year ago

Verified b0b0dc9e

hotfix for RE2 segfaults (#284)

pavel-esir committed 1 year ago

Verified ce9d51c8

Add Charsmap Operation (#267)

apaniukov committed 1 year ago

Verified 0cb064ae

fix "Fix Split Mode Names #269" degradation (#291)

pavel-esir committed 1 year ago

Verified ebcf5c18

Bump aquasecurity/trivy-action from 0.27.0 to 0.28.0 (#290)

dependabot[bot] committed 1 year ago

Verified 318bff56

Older