text-generation-inference
Add a new `/tokenize` route to get the tokenized input
#1471
Merged
Merged
Add a new `/tokenize` route to get the tokenized input #1471
Narsil
merged
8 commits into
main
from
tokenizer_route
Adding tokenizer route.
4f7f617e
Tokenization route.
c12ff389
Remove special, it's not correct enough (and not necessarily useful).
048bc5b4
Updating openapi docs.
b1ffd253
drbh
dismissed these changes
on 2024-01-23
Go back to the default of adding special tokens in validation.
bfe062ee
Don't actually modify the inputs.
168ec6b1
Original truncate behavior.
211b9681
Fmt.
46ca08a8
OlivierDehaene
commented
on 2024-01-26
Login to write a write a comment.
Login via GitHub