llama.cpp
Various script cleanups/fixes + convert merges and special token handling
#2842

Merged

Various script cleanups/fixes + convert merges and special token handling #2842

ggerganov merged 17 commits into ggml-org:master from KerfuffleV2:feat-scripts-improvements

KerfuffleV2 added script

KerfuffleV2 changed the title ~~Feat scripts improvements~~ Various script cleanups/fixes + convert merge and special token handling 2 years ago

KerfuffleV2 changed the title ~~Various script cleanups/fixes + convert merge and special token handling~~ Various script cleanups/fixes + convert merges and special token handling 2 years ago

KerfuffleV2 marked this pull request as ready for review 2 years ago

KerfuffleV2 commented on 2023-08-27

KerfuffleV2 commented on 2023-08-28

cebtenzzre commented on 2023-08-28

ggerganov approved these changes on 2023-08-28

ggerganov requested a review from

monatis 2 years ago

ggerganov requested a review from

klosax 2 years ago

cebtenzzre commented on 2023-08-28

KerfuffleV2 commented on 2023-08-28

KerfuffleV2 force pushed from 60a5f141 to 3dfecac0 2 years ago

KerfuffleV2 requested a review from

ggerganov 2 years ago

convert: Fix permute calls and method/func definitions

1793f25c

Cleanups for gguf-py

084dd216

Minor types cleanups.

795c0c6e

Initial implementation of handling merges and special tokens

ea432678

convert: Handle special tokens and merges in vocab only mode

c7b0952e

gguf: Refactor tensor name mapping

531746e9

convert: Fix type hint for special_token_types in SpecialVocab

120ed645

Use common special vocab handling in various conversion scripts

bb6b64d5

First pass at implementing suggested changes

f82aec99

Second pass

4a3d783d

gguf: SpecialVocab: Fix issue with special token content not in a dict

8534197f

convert-falcon-hf-to-gguf: Support --vocab-only option, bail out if n…

61911ca4

convert-gptneox-hf-to-gguf and convert: Only handle merges for BPE to…

0c620ef6

gguf: SpecialVocab: Actually set load_merges in object

2ea13389

Uniform args parsing and vocab only mode for convert examples

58fa4dc8

KerfuffleV2 force pushed from 3dfecac0 to 58fa4dc8 2 years ago

convert.py: Set gpt2 as tokenizer model when using BPE

ce005285

klosax approved these changes on 2023-08-29

Squish last type warning in gguf.py - yay!

d77b74b4

ggerganov approved these changes on 2023-08-30

ggerganov merged dc07dc49 into master 2 years ago

cebtenzzre commented on 2023-09-25

KerfuffleV2 deleted the feat-scripts-improvements branch 1 year ago

Reviewers

ggerganov

klosax

cebtenzzre

monatis

Assignees

No one assigned

Labels

script

Milestone

No milestone

llama.cpp Various script cleanups/fixes + convert merges and special token handling #2842 Merged

Various script cleanups/fixes + convert merges and special token handling #2842

llama.cpp
Various script cleanups/fixes + convert merges and special token handling
#2842

Merged