PR #1360 Implement backward passes for llama with small training llama from scratch example

implement 8 of 14 missing backward pass operations used by llama

xaedes committed 3 years ago

implement 5 of 6 missing backward pass operations used by llama

xaedes committed 3 years ago

norm & rms_norm can not be threaded:

xaedes committed 3 years ago

remove already resolved TODO

xaedes committed 3 years ago

implement backward pass of ggml_rope and ggml_rope_back

xaedes committed 3 years ago

implement backward pass for ggml_get_rows and for new operation ggml_get_rows_back

xaedes committed 3 years ago

add test-grad0.c

xaedes committed 3 years ago

use GGML_PRINT_DEBUG for debug messages which will otherwise flood the console

xaedes committed 3 years ago

test both gradients of mul_mat

xaedes committed 3 years ago

disable graph dot export as it floods console

xaedes committed 3 years ago

bug fixes for silu_back

xaedes committed 3 years ago

successfully test silu backward

xaedes committed 3 years ago

bug fix for scale backward pass

xaedes committed 3 years ago

successfully test scale backward

xaedes committed 3 years ago

improve performance of sum backward pass

xaedes committed 3 years ago

improve performance of sqr backward pass

xaedes committed 3 years ago

successfully test rope backward

xaedes committed 3 years ago

bug fix for cpy backward pass

xaedes committed 3 years ago

successfully test cpy backward

xaedes committed 3 years ago

bug fix for reshape backward pass

xaedes committed 3 years ago

successfully test reshape backward

xaedes committed 3 years ago

add test-opt.c

xaedes committed 3 years ago

correctly implement softmax backward pass using new operation ggml_diag

xaedes committed 3 years ago

successfully test soft_max backward

xaedes committed 3 years ago

align shape annotations

xaedes committed 3 years ago

add shape annotations for llama

xaedes committed 3 years ago

de-duplicate ggml_forward_dup code taking care of contiguous tensors of same type.

xaedes committed 3 years ago

fix ggml_compute_forward_dup_same_cont for when nelements < nthreads

xaedes committed 3 years ago

bug fix for add_at forward

xaedes committed 3 years ago

successfully test view backward

xaedes committed 3 years ago

minor code format improvement

xaedes committed 3 years ago

fix ggml_forward_add functions to work correctly with transposed tensors

xaedes committed 3 years ago

fix ggml_forward_add1 functions to work correctly with transposed tensors

xaedes committed 3 years ago

test-grad0.c : add print_elements to help with debugging

xaedes committed 3 years ago

successfully test permute backward

xaedes committed 3 years ago

some minor test-grad0 fixes

xaedes committed 3 years ago

fix sub, mul and div functions to work correctly with transposed tensors

xaedes committed 3 years ago

implement ggml_cont backward pass

xaedes committed 3 years ago

successfully test transpose backward and permute for all permutations

xaedes committed 3 years ago

test-grad0.c add TODO for view_2d and view_3d

xaedes committed 3 years ago

fix comments

xaedes committed 3 years ago

successfully test diag_mask_inf and diag_mask_zero backward

xaedes committed 3 years ago

test-grad0 : fix test for div

xaedes committed 3 years ago

fix diag_mask to work with non-inplace input

xaedes committed 3 years ago

move dup call into the actual add_at functions

xaedes committed 3 years ago

fix get rows backward pass

xaedes committed 3 years ago

successfully test get_rows backward

xaedes committed 3 years ago

fix view backward pass

xaedes committed 3 years ago

successfully test backward pass of view_1d, view_2d and view_3d

xaedes committed 3 years ago

fix backward pass for rms_norm

xaedes committed 3 years ago

successfully test backward pass of rms_norm

xaedes committed 3 years ago

add todos for llama backward pass

xaedes committed 3 years ago

add operation ggml_sum_rows

xaedes committed 3 years ago

add missing GGML_OP_SUM_ROWS

xaedes committed 3 years ago

fix backward pass for repeat

xaedes committed 3 years ago

successfully test backward pass of repeat

xaedes committed 3 years ago

update quantization types in switch-case of add_at and add1

xaedes committed 3 years ago

add baby-llama example training a very small llama model from scratch to output a sinusoidal wave.

xaedes committed 3 years ago

fix softmax in baby-llama example

xaedes committed 3 years ago

switching from training with adam to lbfgs produces much better results in the baby-llama example

xaedes committed 3 years ago

train with two examples, creating new tensors each time..

xaedes committed 3 years ago

fix bug when using ggml_opt to optimize params in one context and use a renewable context for eval and opt

xaedes committed 3 years ago

train on multiple examples, generate & print tokens with trained model afterwards

xaedes committed 3 years ago

add ggml_reshape_1d, ggml_reshape_4d and ggml_view_4d

xaedes committed 3 years ago

fix soft_max backward pass for input->ne[1] != 1

xaedes committed 3 years ago

add ggml_log operation necessary for cross entropy loss

xaedes committed 3 years ago

add test for ggml_log gradients

xaedes committed 3 years ago

implement backward pass for ggml_sum_rows, necessary for cross entropy loss

xaedes committed 3 years ago

implement ggml_repeat support for rank > 2 tensors

xaedes committed 3 years ago

add test for ggml_sum_rows gradients

xaedes committed 3 years ago

fix training get_example_targets

xaedes committed 3 years ago

add square_error_loss and cross_entropy_loss functions

xaedes committed 3 years ago

optimize loss over multiple samples

xaedes committed 3 years ago

fix backward pass for add_at and change arguments to have same order as in view

xaedes committed 3 years ago

add ggml_set(ctx, a, b) to set b in view of a and return modified a

xaedes committed 3 years ago

fix kv_self gradients for training

xaedes committed 3 years ago

replace inplace operations for training with copying operations to allow gradient propagation

xaedes committed 3 years ago

add GGML_ASSERT to catch ggml_rope and back value errors

xaedes committed 3 years ago

add trainable lora-only model with all big matrices C split into A,B with A*B=C

xaedes committed 3 years ago

vastly improve training results

xaedes committed 3 years ago

shorten code using a variable

xaedes committed 3 years ago

change name of GGML_OP_ADD_AT to GGML_OP_ACC

xaedes committed 3 years ago

Merge branch 'master' into train-example

xaedes committed 3 years ago

smaller default values for baby llama model parameters

xaedes committed 3 years ago

update static assert of GGML_OP_COUNT

xaedes committed 3 years ago

remove shape annotations in llama_eval_internal

xaedes committed 3 years ago

revert disabling of threading for rms_norm and norm

xaedes committed 3 years ago

rename print functions in baby-llama example

xaedes committed 3 years ago

fix call to ggml_set_name

xaedes committed 3 years ago

add missing include for strcmp, etc

xaedes committed 3 years ago

remove trailing whitespace

xaedes committed 3 years ago

reduce number of test-grad0 iterations

xaedes committed 3 years ago

remove busy loop that was used as sleep for slower sinus wave generation

xaedes committed 3 years ago

disable slow tests grad0 and opt to avoid exceeding timeouts

xaedes committed 3 years ago

c++ in baby-llama example

xaedes committed 3 years ago

c++ in baby-llama example

xaedes committed 3 years ago

ggml : fix compiler warnings + cosmetic changes

ggerganov committed 3 years ago

ggml : fix nullptr derefs in GGML_OP_CONT and GGML_OP_RESHAPE back

ggerganov committed 3 years ago

swap arguments to vDSP_vdiv call

xaedes committed 3 years ago

swap arguments to vDSP_vdiv call

ggerganov committed 3 years ago

llama.cpp Implement backward passes for llama with small training llama from scratch example #1360 Merged

llama.cpp
Implement backward passes for llama with small training llama from scratch example
#1360

Merged