llama.cpp
Implement backward passes for llama with small training llama from scratch example
#1360
Merged
Merged
Implement backward passes for llama with small training llama from scratch example #1360
ggerganov
merged
110 commits into
ggml-org:master
from
xaedes:train-example
implement 8 of 14 missing backward pass operations used by llama
73ac18d8
implement 5 of 6 missing backward pass operations used by llama
b1643435
norm & rms_norm can not be threaded:
b9080074
remove already resolved TODO
36d8a051
implement backward pass of ggml_rope and ggml_rope_back
488decfd
implement backward pass for ggml_get_rows and for new operation ggml_…
4e1f81d3
add test-grad0.c
0da26753
use GGML_PRINT_DEBUG for debug messages which will otherwise flood th…
20e3c1d2
test both gradients of mul_mat
9345f4c3
disable graph dot export as it floods console
9d6fc28f
bug fixes for silu_back
6fb08b45
successfully test silu backward
671e5922
bug fix for scale backward pass
a367eb9e
successfully test scale backward
0197bcb0
improve performance of sum backward pass
bfe50721
improve performance of sqr backward pass
b583136c
successfully test rope backward
75711472
bug fix for cpy backward pass
0ea8201c
successfully test cpy backward
b2bd8222
bug fix for reshape backward pass
c483a7da
successfully test reshape backward
ecf949b1
add test-opt.c
54ab300c
correctly implement softmax backward pass using new operation ggml_diag
1a80e9a0
successfully test soft_max backward
fea42be4
align shape annotations
93106504
add shape annotations for llama
38675e53
de-duplicate ggml_forward_dup code taking care of contiguous tensors …
c1a8893d
fix ggml_compute_forward_dup_same_cont for when nelements < nthreads
83fa6b3b
bug fix for add_at forward
cecd6c76
successfully test view backward
124fdca9
minor code format improvement
410a47a7
fix ggml_forward_add functions to work correctly with transposed tensors
b9416d71
fix ggml_forward_add1 functions to work correctly with transposed ten…
339b2adf
test-grad0.c : add print_elements to help with debugging
86b44a02
successfully test permute backward
a7a83704
some minor test-grad0 fixes
b0555fce
fix sub, mul and div functions to work correctly with transposed tensors
02d3fd08
implement ggml_cont backward pass
3d21f264
successfully test transpose backward and permute for all permutations
c601df97
test-grad0.c add TODO for view_2d and view_3d
1997152f
fix comments
d42531fa
successfully test diag_mask_inf and diag_mask_zero backward
19f51592
test-grad0 : fix test for div
b9920e5c
fix diag_mask to work with non-inplace input
3dbd649c
move dup call into the actual add_at functions
7281f605
fix get rows backward pass
96e773bb
successfully test get_rows backward
f0302fa7
fix view backward pass
84436383
successfully test backward pass of view_1d, view_2d and view_3d
b18b72da
fix backward pass for rms_norm
84a4b399
successfully test backward pass of rms_norm
2ecc6909
add todos for llama backward pass
22770538
add operation ggml_sum_rows
c4539ede
add missing GGML_OP_SUM_ROWS
ba62c79b
fix backward pass for repeat
8b5b2f08
successfully test backward pass of repeat
72bcfb50
update quantization types in switch-case of add_at and add1
1c4dc1e4
add baby-llama example training a very small llama model from scratch…
8fde656d
fix softmax in baby-llama example
29a0f8b9
switching from training with adam to lbfgs produces much better resul…
5f23052e
train with two examples, creating new tensors each time..
bc1c13bb
fix bug when using ggml_opt to optimize params in one context and use…
83ee1cd7
train on multiple examples, generate & print tokens with trained mode…
f1d51d14
add ggml_reshape_1d, ggml_reshape_4d and ggml_view_4d
b4c273f7
fix soft_max backward pass for input->ne[1] != 1
8cf04fec
add ggml_log operation necessary for cross entropy loss
65d9f734
add test for ggml_log gradients
5724628d
implement backward pass for ggml_sum_rows, necessary for cross entrop…
7a15a837
implement ggml_repeat support for rank > 2 tensors
e6186d98
add test for ggml_sum_rows gradients
80223d98
fix training get_example_targets
73fd66e9
add square_error_loss and cross_entropy_loss functions
7a5dec24
optimize loss over multiple samples
226521a4
fix backward pass for add_at and change arguments to have same order …
48bcc4dc
add ggml_set(ctx, a, b) to set b in view of a and return modified a
47561de7
fix kv_self gradients for training
956511b2
replace inplace operations for training with copying operations to al…
561fbe0d
add GGML_ASSERT to catch ggml_rope and back value errors
e91b83b8
add trainable lora-only model with all big matrices C split into A,B …
93201abd
vastly improve training results
49d6daa1
shorten code using a variable
e0de09d7
change name of GGML_OP_ADD_AT to GGML_OP_ACC
47648421
Merge branch 'master' into train-example
ee565f34
smaller default values for baby llama model parameters
e643fa16
update static assert of GGML_OP_COUNT
d20ba6f6
remove shape annotations in llama_eval_internal
5d9fed7e
revert disabling of threading for rms_norm and norm
47ad1866
rename print functions in baby-llama example
9dd8e405
fix call to ggml_set_name
660836f0
add missing include for strcmp, etc
7c8768f8
remove trailing whitespace
2936dd60
reduce number of test-grad0 iterations
4997bc58
remove busy loop that was used as sleep for slower sinus wave generation
f5301061
disable slow tests grad0 and opt to avoid exceeding timeouts
1ecbece7
c++ in baby-llama example
dea9c935
c++ in baby-llama example
0d72207a
ggml : fix compiler warnings + cosmetic changes
78af3e92
ggml : fix nullptr derefs in GGML_OP_CONT and GGML_OP_RESHAPE back
6cc42ded
swap arguments to vDSP_vdiv call
cafbb785
swap arguments to vDSP_vdiv call
9c3fe4eb
ggml : swap vDSP_vsub args as per documentation
6ca682b1
add parallel batched forward function for baby-llama training
3e3ed956
cleanup code for batched training
581e5eb9
remove trailing whitespace
b9ef08cc
ggerganov
commented
on 2023-05-12
minor : fix compiler warnings + indentation style
f977243d
ggml : fix null ptr deref in backward pass
33034cfe
Merge remote-tracking branch 'origin/master' into HEAD
092913ec
ggml : remove Q4_2 remnants
95a487a1
ggml : fix clang-tidy warnings
ef3d42a3
baby-llama : couple of clang-tidy warnings
dae6ba2a
ggerganov
approved these changes
on 2023-05-13
Reviewers
Assignees
No one assigned
Labels
high priority
Milestone
No milestone
Login to write a write a comment.
Login via GitHub