[v4] Tokenizers.js migration #1501
xenova
merged
17 commits into
v4
from
v4-tokenizers-migration
Remove legacy tokenizer tests
213cbd63
Update utils.test.js
34d7a508
Update unit tests
ca2c90f9
Refactor tokenizers.js
68b521db
Update streamers.js
6b52b45a
Update imports
7343bb60
Initial tokenization migration
24ad2c0e
Remove unnecessary index.js files
14ed5406
Do not export PretrainedMixin
8d901c09
Refactor models.js/tokenizers.js
84e3e9be
Update imports
53fccd99
Update more imports
e571778f
Update folder structure and imports
34f1ca29
Delete tokenization_code_xlm_roberta.js
121156ae
nico-martin
commented
on 2026-01-21
Fix import path
ebe7d3df
Fix typos
a46a66f1
Update import path
f0668b52
Login to write a write a comment.
Login via GitHub