Adding more token encoding types (#1254)
* add more token encoing types
* add more token encoing types
* add tests
* add tests
* ft support, tests
* linting is shortening my lifespan
* linting is shortening my lifespan
* long tensor
* long tensor
* long tensor
* feedbacc
* import
* import
---------
Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>
