transformers.js
b6bd6083 - Update falcon tokenizer (#344)
Commit
View On GitHub
Comment changes are shownComment changes are hidden
Commit
1 year ago
Update falcon tokenizer (#344) * Update generate_tests.py * Do not add token types for `FalconTokenizer`
Author
xenova xenova
Parents
5b311292
  • src
    • File
      tokenizers.js
  • tests
    • File
      generate_tests.py
FAQ Terms Privacy Refunds Impressum