Add support for chat templates #408
xenova
merged
60 commits into
main
from
chat-templates
Add basic support for chat templates
1697c3d9
Cleanup
7878ded4
JSDoc improvements
03389ac8
Support conversion of user-defined functions
d91e022d
Cleanup
466d1e43
Fix function creation
449a027c
Add unit tests for templates
d7700a05
Cleanup
ee5af8d7
Merge branch 'main' into chat-templates
5f37eebe
Improve JSDoc
69271365
Add missing return types
3b088277
Add chat templates docs to table of contents
ffab125c
Add support for logical negation
c5629b44
Fix nested logical negation
aabe4bec
Add unit tests for logical operators
5f4d7af8
Add loop variables
c3a6f081
Add support for `RuntimeValue` built-in functions
1bbf8828
Add unit tests for string instance methods
a5fafe8e
Fix conversion of normal function to `FunctionValue`
6559f814
Update object method unit tests
468e7df8
Save chat template to tokenizer_config.json during conversion
abaf579e
Fix `raise_exception` error
be49ef64
Add `!=` operator for booleans
40dfca45
Remember to increment loop index
4445766f
Cleanup for loop evaluator
6be2463b
Use `is` helper function
d4a37ae6
Add support for text nodes
6d826222
Add auto-generated templating tests
eb88df1b
Update unit tests
7fc877ab
Remove unused function
c41550e7
Add default chat templates
fe08e3f2
Use repo with up-to-date tokenizer config
b1db4a25
Temporarily disable zephyr test
0ae00fe8
Delete templates.test.js
68ac91c8
Move Jinja functionality to `@huggingface/jinja`
e6508356
Fix template cache type
0e6b945e
Update chat template unit tests
27762f1d
Update `@huggingface/jinja` version
4ca26ffc
Merge branch 'main' into chat-templates
cd8ab6f2
Fix default llama2 system prompt usage
d9b63aec
Add unit test for llama2 w/o chat template set
ec4d0c69
Update jinja version
6560d68c
Update jinja version
d9a91711
Add unit test for user-defined chat templates
147dc789
Add `AddedToken` for improved tokenization
3ee65c66
Add example usage for chat templates
149194b5
Add 'first' Metaspace pretokenizer prepend scheme
bf9ec96c
Formatting
f5c6edb2
Update wav2vec2 converter special tokens whitespace split
ec5e2872
Fix Metaspace pretokenizer split criteria
7a3b3aa4
Update inputs of `PreTokenizerSequence`
073ec93b
Improve Metaspace pretokenizer
6b5e064f
Update llama tokenizer tests
a2316207
Improve handling of legacy llama tokenizer
30a45acf
Re-enable SPM tests
c2c10d3d
Add static tokenizer test cases
4ddd1c8a
Add llama2 static tests
29589849
Allow user to override legacy tokenizer behaviour in `.from_pretrained`
e4f5cb16
Add legacy tokenizer unit tests
56af9773
Bump jinja version to 0.1.0
56820b31
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub