llama.cpp
Make tokenize CLI tool have nicer command line arguments.
#6188

Merged

Make tokenize CLI tool have nicer command line arguments. #6188

mofosyne merged 5 commits into ggml-org:master from Noeda:tokenizer-nicer-args

ggerganov approved these changes on 2024-03-21

Make tokenizer.cpp CLI tool nicer.

cd7b5f7f

Noeda force pushed to cd7b5f7f 2 years ago

Noeda marked this pull request as ready for review 2 years ago

Noeda requested a review from

ggerganov 2 years ago

cebtenzzre commented on 2024-03-26

style fix: strlen(str) == 0 --> *str == 0

a8376497

ggerganov approved these changes on 2024-03-27

Simplify tokenize.cpp; by getting rid of handling positional style ar…

71a08675

jdh1166 approved these changes on 2024-04-25

Merge branch 'master' into tokenizer-nicer-args

877f0590

ggerganov commented on 2024-05-09

mofosyne added enhancement

mofosyne added Review Complexity : Medium

tokenize.cpp: iostream header no longer required

4aaeb425

github-actions added examples

mofosyne added merge ready

mofosyne merged 57684331 into master 2 years ago

Reviewers

ggerganov

jdh1166

mofosyne

cebtenzzre

Assignees

No one assigned

Labels

enhancement examples Review Complexity : Medium merge ready

Milestone

No milestone