transformers
feat(tokenization): add encode_message to tokenize messages one by one
#39507
Merged
Merged
feat(tokenization): add encode_message to tokenize messages one by one #39507
ArthurZucker
merged
12 commits into
huggingface:main
from
pco111:fix/issue-39417-incremental-tokenization
feat(tokenization): add encode_message to tokenize messages one by one
7166f1b6
Fix the `encode_message` method, remove the `add_generation_prompt` p…
c0c65ba5
Optimize the `encode_message` method, improve the processing logic of…
423287be
The `_encode_message` method is deleted, the message coding logic is …
7ab08034
Docs fix
b7e57b9c
Revert changes in docstring of pad()
f39d67a1
Revert changes in docstring
f14a3ee3
Rocketknight1
approved these changes
on 2025-07-28
ArthurZucker
approved these changes
on 2025-07-29
Update src/transformers/tokenization_utils_base.py
4d59bc5b
Repair the call of the `encode_message` method, update it to `encode_…
d70fbec9
Optimize the call format of the `apply_chat_template` method, and mer…
e859b529
Merge branch 'main' into fix/issue-39417-incremental-tokenization
89dbbaf3
Merge branch 'main' into fix/issue-39417-incremental-tokenization
ec454c8e
Reviewers
Assignees
No one assigned
Labels
Chat Template
Milestone
No milestone
Login to write a write a comment.
Login via GitHub