transformers.js
66cf69cb - [v4] Tokenizers.js migration (#1501)

Commit
27 days ago
[v4] Tokenizers.js migration (#1501) * Remove legacy tokenizer tests * Update utils.test.js * Update unit tests * Refactor tokenizers.js * Update streamers.js * Update imports * Initial tokenization migration * Remove unnecessary index.js files * Do not export PretrainedMixin * Refactor models.js/tokenizers.js * Update imports * Update more imports * Update folder structure and imports * Delete tokenization_code_xlm_roberta.js * Fix import path * Fix typos * Update import path
Author
Parents
Loading