transformers.js

c8bbdd41 - Implement max character check for WordPiece tokenizer (#398)

Implement max character check for WordPiece tokenizer (#398) * Implement max character check per token * Update maxInputCharsPerWord to max_input_chars_per_word Co-authored-by: Joshua Lochner <admin@xenova.com> * Update maxInputCharsPerWord to max_input_chars_per_word Co-authored-by: Joshua Lochner <admin@xenova.com> * Update to ?? Co-authored-by: Joshua Lochner <admin@xenova.com> --------- Co-authored-by: Joshua Lochner <admin@xenova.com>