Separate chat templates into a single file #33957
Rocketknight1
merged
33 commits into
main
from
chat_template_files
zucchini-nlp
commented
on 2024-10-11
ArthurZucker
commented
on 2024-10-29
Initial draft
020b6d90
Add .jinja file loading for processors
b8f9e0f1
Add processor saving of naked chat template files
ca101e65
make fixup
82c357e2
Add save-load test for tokenizers
3775e2b7
Add save-load test for tokenizers
91483a0b
stash commit
47205a26
Try popping the file
820990b9
make fixup
2aeb3199
Pop the arg correctly
a92dc109
Pop the arg correctly
1d54b7d1
Add processor test
5b267487
Fix processor code
743a4a50
stash commit
455a297c
Processor clobbers child tokenizer's chat template
52614f3e
Processor clobbers child tokenizer's chat template
a200a427
make fixup
5c792bc8
Split processor/tokenizer files to avoid interactions
5305a87b
fix test
36564683
Expand processor tests
e412619b
Rename arg to "save_raw_chat_template" across all classes
bd816821
Update processor warning
50d374a1
Move templates to single file
a81a7e15
Move templates to single file
f9127c17
Improve testing for processor/tokenizer clashes
3878828d
Improve testing for processor/tokenizer clashes
74cd295a
Extend saving test
cf577e1a
Test file priority correctly
b97e76de
make fixup
e5b8b76a
Don't pop the chat template file before the slow tokenizer gets a look
5b40f040
Remove breakpoint
2159cdc7
make fixup
5b88768b
Fix error
1187047d
ArthurZucker
approved these changes
on 2024-11-22
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub