transformers.js
ebd53353 - Add support for cross-encoder models (+fix token type ids) (#501)

Comment changes are shownComment changes are hidden
Commit
1 year ago
Add support for cross-encoder models (+fix token type ids) (#501) * Formatting * Update ESM pair template * Fix token type ids * Update JSDoc * Cleanup * Remove unused `prepare_model_inputs` function * Move pad and truncate logic to helper functions * Add static padding/truncation unit tests * Fix padding/truncation * Remove unused `add_token_types` function * Reduce duplication * `let` -> `const` where possible * Add cross-encoder models
Author
Parents
  • scripts
    • extra
      • File
        esm.py
    • File
      supported_models.py
  • src
    • File
      models.js
    • File
      tokenizers.js
  • tests
    • File
      generate_tests.py
    • File
      tokenizers.test.js
Loading