PR #3728 Add test for MPT tokenization

Add test for MPT tokenization #3728

goerch merged 4 commits into ggml-org:master from test-#3604

Add test for MPT tokenization

6a94ae6d

ggerganov approved these changes on 2023-10-22

Revert code motion

8ee73637

Remove unnecessary restriction in test case

1dc13168

cebtenzzre commented on 2023-10-22

Clarify logic in conversion

74204ccb

goerch merged 9e70cc03 into master 2 years ago

goerch deleted the test-#3604 branch 2 years ago

Reviewers

ggerganov

cebtenzzre

Assignees

No one assigned

Labels

None yet

Milestone

No milestone