llama.cpp
convert-*.py: GGUF Naming Convention Refactor and Metadata Override Refactor
#7499
Merged
Merged
convert-*.py: GGUF Naming Convention Refactor and Metadata Override Refactor #7499
mofosyne
merged
66 commits into
ggml-org:master
from
mofosyne:refactor-convert-py
compilade
commented
on 2024-05-24
compilade
commented
on 2024-05-24
compilade
commented
on 2024-05-28
compilade
commented
on 2024-05-30
compilade
commented
on 2024-05-30
teleprint-me
commented
on 2024-05-31
teleprint-me
commented
on 2024-05-31
Galunid
commented
on 2024-06-03
Galunid
commented
on 2024-06-03
compilade
commented
on 2024-06-03
julien-c
commented
on 2024-06-03
julien-c
commented
on 2024-06-03
Vaibhavs10
commented
on 2024-06-03
ggerganov
commented
on 2024-06-04
teleprint-me
commented
on 2024-06-06
teleprint-me
commented
on 2024-06-06
teleprint-me
commented
on 2024-06-06
compilade
commented
on 2024-07-14
convert-*.py: licence -> license
d3a936fd
convert-*.py: add --get-outfile command and refactor
dbb1b471
convert-*.py: add basename and finetune metadata
a42c2b7e
convert-*.py: model card metadata
916872f7
convert-*.py: metadata class moved to utility
4d5f18a0
convert-*.py: encoding_scheme --> output_type
5c263cb2
convert-*.py: parse model card in metadata util. Add license_link and…
b36e391b
convert-*.py: add base_version and add tags
8f734083
convert-*.py: add parameter size class
0f1d50fa
convert-*.py: add datasets and language to KV store
684c604e
convert-*.py: move per model weight estimation away from util back to…
b1927eed
convert-*.py: enable --model-name direct metadata override
f7c20793
convert-*.py: add general.organization to kv store
5a86dfaa
convert-*.py: add quantized_by and enhance heuristics
dd157121
convert-*.py: adjust help message
b0553f42
convert-*.py: use heuristics to parse _name_or_path
4d5cd067
convert-*.py: base_model is actually in spec for model cards
32e80e09
convert-*.py: refactor parameter weight class
54918ad1
convert-*.py: need to include self in per_model_weight_count_estimati…
39472a09
convert-*.py: add heuristic to directory name fallback
3625a420
convert-*.py: add unittest to metadata class
91e65d94
convert-*.py: adjusted authorship KV store
d060fcdb
convert-*.py: separated unit test, hf_repo to repo_url
eaa47f55
convert-*.py: Remove self.model_name that was left in since last rebase
e9734434
convert_hf_to_gguf.py: optional, dataclass removed from type as it wa…
5011eefe
convert_hf_to_gguf.py: rebase error correction
2f23927d
convert_hf_to_gguf.py: Remove code that is already in fill_templated_…
4dc8ddd3
gguf_writer.py: generate tensor uuid if missing
007708e3
test: remove test_gguf.py and remove test_generate_any_missing_uuid()
7ecb8f00
convert-*.py: autogenerate general.uuid if missing
fdc5a3fc
convert-*.py: write_tensors() --> prepare_tensors_for_writing()
2a976e12
convert-*.py: refactor per model weight count estimation
59a01df7
convert-*.py: pyright type fixes
dd14b8fd
Apply suggestions from code review
74383ba6
convert-*.py: cast not required if Metadata.load_metadata_override re…
4c91d077
convert-*.py: Removing the redundant metadata is not None from all co…
6eb08ac8
convert-*.py: parameter_class_attribute --> size_label
f8b59311
convert-*.py: remove redundant gguf_writer.add_name() calls
64707b62
convert-*.py: prepare_tensors_for_writing() --> prepare_tensors()
04c4fffd
convert-*.py: import cast from typing and other refactor
f2b425c5
convert-*.py: remove autogenerated uuid
ad217d72
Update convert_hf_to_gguf.py
60278e4f
Update convert_hf_to_gguf.py
aa4e5892
Update constants.py : spacing correction
2c060303
constants.py : Revert removal of backward compatibility KEY_GENERAL_S…
8156835d
convert-*.py: remove reference to uuid generation
ccff6c7f
Apply suggestions from code review
455c0e53
convert-*.py: dict_item --> Iterable
5ab1a840
convert-*.py: update nix package to add python frontmatter
5cdb03b2
convert-*.py: add logger and refactor load_model_card()
9954b648
convert-*.py: quantized_by in model card is not relevant for converte…
abc351c2
convert-*.py: pathlib.Path exist() --> is_file() or is_dir()
144a7ec3
covert-*.py: per_model_weight_count_estimation() tensor arg type is I…
8629b7bd
covert-*.py: flake8 newline missing
4e376110
convert-*.py: more rigorous regexp for get_model_id_components()
f98f1098
convert-*.py: flake8 remove blank line
3b1766a9
gguf-py : use pyyaml instead of python-frontmatter
78a42fbe
convert_hf : use GGUFWriter to count model parameters
417d7a7c
metadata.py: account for decimal point in size label within model id …
9a925b56
compilade
commented
on 2024-07-15
Update convert_hf_to_gguf.py
c7b36164
Merge branch 'master' into refactor-convert-py
5da16bb1
convert-*.py: Add naming_convention_vocab_only()
eb0bf6b9
convert_lora_to_gguf.py: remove model_name parameter. Doesn't exist i…
7e9271ca
mofosyne
dismissed their stale review
2 years ago
Had a look again at all the review requested marked code, looked to all be resolved already
gguf-py : extract metadata from model name more resiliently
2c18a9a4
gguf-py : fix flake8 lint
4c9932c1
gguf-py : handle more name metadata extraction edge cases
73899f74
Login to write a write a comment.
Reviewers
Assignees
Labels
help wanted
refactoring
nix
examples
python
Review Complexity : Medium
devops
merge ready
Milestone
No milestone