xenova/transformers.js

Pull Requests Commits

Bump jinja version to 0.1.0

xenova committed 1 year ago

56820b31

Add legacy tokenizer unit tests

xenova committed 1 year ago

56af9773

Allow user to override legacy tokenizer behaviour in `.from_pretrained`

xenova committed 1 year ago

e4f5cb16

Add llama2 static tests

xenova committed 1 year ago

29589849

Add static tokenizer test cases

xenova committed 1 year ago

4ddd1c8a

Re-enable SPM tests

xenova committed 1 year ago

c2c10d3d

Improve handling of legacy llama tokenizer

xenova committed 1 year ago

30a45acf

Update llama tokenizer tests

xenova committed 1 year ago

a2316207

Improve Metaspace pretokenizer

xenova committed 1 year ago

6b5e064f

Update inputs of `PreTokenizerSequence`

xenova committed 1 year ago

073ec93b

Fix Metaspace pretokenizer split criteria

xenova committed 1 year ago

7a3b3aa4

Update wav2vec2 converter special tokens whitespace split

xenova committed 1 year ago

ec5e2872

xenova committed 1 year ago

f5c6edb2

Add 'first' Metaspace pretokenizer prepend scheme

xenova committed 1 year ago

bf9ec96c

Add example usage for chat templates

xenova committed 1 year ago

149194b5

Add `AddedToken` for improved tokenization

xenova committed 1 year ago

3ee65c66

Add unit test for user-defined chat templates

xenova committed 1 year ago

147dc789

Update jinja version

xenova committed 1 year ago

d9a91711

Update jinja version

xenova committed 1 year ago

6560d68c

Add unit test for llama2 w/o chat template set

xenova committed 1 year ago

ec4d0c69

Fix default llama2 system prompt usage

xenova committed 1 year ago

d9b63aec

Merge branch 'main' into chat-templates

xenova committed 1 year ago

cd8ab6f2

Update `@huggingface/jinja` version

xenova committed 1 year ago

4ca26ffc

Update chat template unit tests

xenova committed 1 year ago

27762f1d

Fix template cache type

xenova committed 1 year ago

0e6b945e

Move Jinja functionality to `@huggingface/jinja`

xenova committed 1 year ago

e6508356

Delete templates.test.js

xenova committed 1 year ago

68ac91c8

[version] Update to 2.11.0

xenova committed 1 year ago

6129e45b

Add support for ChineseCLIP models (#455)

xenova committed 1 year ago

Verified 2de085b6

Add support for ViTMatte models (#448)

xenova committed 1 year ago

Verified b978ff8c

Older