PR #422 Add Support For Two String Inputs

Add Support For Two String Inputs #422

pavel-esir merged 13 commits into openvinotoolkit:master from pavel-esir:add_several_inputs

solution with concat

42cee5ff

with Reshape

dc7ad0a9

successfully works on tokenizers

8a99e981

pavel-esir changed the title ~~Add several inputs~~ Add Support For Two String Inputs 325 days ago

github-actions added tokenizers_cpp

github-actions added tokenizers_py

add max_length truncation for pair input

9a840227

add check_tokenizer.ipynb

b3d5c700

corrected division by half

78c5097d

use transformation instead of modifying pipeline

dcf342bc

add tests

6de45660

github-actions added tokenizers_test

pavel-esir commented on 2025-03-04

extend tests, revert benchmark workarounds

ac55a8d5

github-actions removed tokenizers_cpp

pavel-esir marked this pull request as ready for review 314 days ago

pavel-esir requested a review from

apaniukov 314 days ago

apaniukov commented on 2025-03-05

apply comments, fix segment_ids/token_type_ids

0ba6ed34

pavel-esir requested a review from

apaniukov 313 days ago

apaniukov approved these changes on 2025-03-05

Merge branch 'master' into add_several_inputs

1ab447d6

fix tests, run ruff format

acee4fa1

update pass_rates.json and stats.json

5569ea5b

pavel-esir enabled auto-merge 313 days ago

pavel-esir merged 26d9676e into master 312 days ago

pavel-esir deleted the add_several_inputs branch 312 days ago

Reviewers

apaniukov

Assignees

No one assigned

Labels

tokenizers_py tokenizers_test

Milestone

No milestone

openvino_tokenizers Add Support For Two String Inputs #422 Merged

Add Support For Two String Inputs #422

openvino_tokenizers
Add Support For Two String Inputs
#422

Merged