PR #2527 We can have a tokenizer anywhere.

We can have a tokenizer anywhere. #2527

Narsil merged 12 commits into main from omni_tokenizer

OlivierDehaene commented on 2024-09-17

OlivierDehaene dismissed these changes on 2024-09-17

Narsil dismissed their stale review via c88e9d47 1 year ago

Narsil force pushed from cd806b22 to c88e9d47 1 year ago

Narsil force pushed from 8f5e11db to 2bba0274 1 year ago

Narsil force pushed from ce2d951c to 6bf4ea6e 1 year ago

Narsil force pushed from 0f0e6a75 to 982f4741 1 year ago

Narsil force pushed from e4f85461 to 905731b3 1 year ago

OlivierDehaene commented on 2024-10-17

We can have a tokenizer anywhere.

5ba7805f

Handling potential lack of offsets (python tokenizer)

9d702bcd

Remove redundancy.

b89b9fd0

Fixing the tests.

5bc1fe84

Flake.lock update ?

c0151cc1

Fixing the GIL locking.

9d7a95b2

Fixing mamba by using the transformers version.

cd355d08

Adding the legacy handle.

f20ef614

Ellide lifetime.

b07935b0

Lint.

d4d43218

Deprecation message.

bbbd9a6d

Narsil force pushed from 8350797f to bbbd9a6d 1 year ago

Fixing bad rebase.

123ff3a8

Narsil requested a review from

OlivierDehaene 1 year ago

Narsil merged 90b226db into main 1 year ago

Narsil deleted the omni_tokenizer branch 1 year ago

Reviewers

OlivierDehaene

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

text-generation-inference We can have a tokenizer anywhere. #2527 Merged

We can have a tokenizer anywhere. #2527

text-generation-inference
We can have a tokenizer anywhere.
#2527

Merged