xenova/transformers.js

Pull Requests Commits
Add emoji unit tests for bert/t5
xenova committed 1 year ago
ca4b5b98
Add more spm normalization characters
xenova committed 1 year ago
f148afd6
Handle ZERO WIDTH JOINER (U+200D) characters
xenova committed 1 year ago
67977718
Fix BertNormalizer (strip `Mn` unicode characters)
xenova committed 1 year ago
14b4bd4a
Remove unused code
xenova committed 1 year ago
072cbbce
Attempt to fix encoder-decoder memory leak
xenova committed 1 year ago
969d10e1
Add emoji tokenizer test cases for LlamaTokenizer
xenova committed 1 year ago
a912d796
[version] Update to 3.0.0-alpha.15
xenova committed 1 year ago
54f1f214
Bump versions
xenova committed 1 year ago
dabe6ae3
Code formatting
xenova committed 1 year ago
1a388048
Move whisper-specific unit tests to subfolder
xenova committed 1 year ago
79a74095
Update emoji test string names
xenova committed 1 year ago
c6c3ae18
Add llama tokenizer unit tests
xenova committed 1 year ago
4798755c
Fix `fuse_unk` for tokenizers with `byte_fallback=true` but no byte fallback tokens
xenova committed 1 year ago
cc258c23
Run unit tests in parallel
xenova committed 1 year ago
0d3cd309
Add ESM tokenizer tests
xenova committed 1 year ago
710816ef
Run prettier
xenova committed 1 year ago
fd8b9a25
Add falcon tokenizer tests
xenova committed 1 year ago
4765dd63
Add t5 tokenizer tests
xenova committed 1 year ago
36ad144b
Add blenderbot tokenizer tests
xenova committed 1 year ago
89ddccf5
Fix `.tokenize` with `fuse_unk=true`
xenova committed 1 year ago
a519379b
Rather make a copy of `this.added_tokens`
xenova committed 1 year ago
d7df5758
Sort `this.added_tokens` before creating regex (`.toSorted` is not available in Node.js < 20)
xenova committed 1 year ago
30315b21
Merge branch 'main' into v3
xenova committed 1 year ago
c40a1512
Merge branch 'main' into v3
xenova committed 1 year ago
4f17e954
Bump `onnxruntime==1.19.2` in scripts/requirements.txt
xenova committed 1 year ago
ba58ea24
Bump onnxruntime-web version
xenova committed 1 year ago
a10828f4
move certain tests to utils subfolder
xenova committed 1 year ago
48fa226e
Run prettier on `tests/**`
xenova committed 1 year ago
deec3504
Fix CI
xenova committed 1 year ago
2f9b2ed9
Terms Of Use Privacy Policy Refund Policy Impressum