llm-foundry
Make TiktokenTokenizerWrapper compatible with convert_composer_to_hf.py
#730

Merged

Make TiktokenTokenizerWrapper compatible with convert_composer_to_hf.py #730

irenedea merged 7 commits into mosaicml:main from irenedea:tiktoken-hf-checkpoints

irenedea commented on 2023-11-10

irenedea requested a review from

dakinggg 2 years ago

Add trust_remote_code argument to conversion scripts

16c76860

irenedea force pushed to 16c76860 2 years ago

make argument better

e164b9eb

irenedea marked this pull request as draft 2 years ago

Catch error and give a nicer message

7f23c06a

irenedea force pushed to 7f23c06a 2 years ago

irenedea marked this pull request as ready for review 2 years ago

fix tests

3111d3eb

irenedea force pushed to 3111d3eb 2 years ago

Merge branch 'main' into tiktoken-hf-checkpoints

e4ed2846

fix other tests

b5071d73

dakinggg approved these changes on 2023-11-11

Merge branch 'main' into tiktoken-hf-checkpoints

d16013d1

irenedea merged d11ba820 into main 2 years ago

Reviewers

dakinggg

Assignees

No one assigned

Labels

None yet

Milestone

No milestone