Add token type ids to CodeGenTokenizer #29265
ArthurZucker
merged
14 commits into
huggingface:main
from
st81:add_token_type_ids_to_code_gen_tokenizer
st81
changed the title
ArthurZucker
commented
on 2024-02-28
ArthurZucker
commented
on 2024-03-05
ArthurZucker
commented
on 2024-03-06
ArthurZucker
commented
on 2024-03-07
ArthurZucker
commented
on 2024-03-25
Add create token type ids to CodeGenTokenizer
220cd007
Fix inconsistent length of token type ids
cec6ba47
Format source codes
428b0e85
Fix inconsistent order of methods
a92d1bab
Update docstring
fe6855e9
add test_tokenizer_integration test
ec35162e
Format source codes
92ca86d9
Add `copied from` comment to CodeGenTokenizerFast
a611eefe
Add doc of create_token_type_ids_from_sequences
91a9f875
Make return_token_type_ids False by default
067720a6
Make test_tokenizer_integration as slow test
2ed831c2
Add return_token_type_ids to tokenizer init arg
bbbdebc6
Add test for tokenizer's init return_token_type_ids
ce599f76
Format source codes
8fa1d286
ArthurZucker
approved these changes
on 2024-04-17
Login to write a write a comment.
Login via GitHub