llama.cpp
cuda : add batched cuBLAS GEMM for faster attention
#3749

Merged

Commits

cmake : add helper for faster CUDA builds

ggerganov committed 2 years ago
batched : add NGL arg

ggerganov committed 2 years ago
ggml : skip nops in compute_forward

ggerganov committed 2 years ago
cuda : minor indentation

ggerganov committed 2 years ago
cuda : batched cuBLAS GEMMs for src0 F16 and src1 F32 (attention ops)

ggerganov committed 2 years ago
Apply suggestions from code review

KerfuffleV2 committed 2 years ago
cuda : add ROCm / hipBLAS cublasGemmBatchedEx define

ggerganov committed 2 years ago
cuda : add cublasGemmStridedBatchedEx for non-broadcasted cases

ggerganov committed 2 years ago
cuda : reduce mallocs in cublasGemmBatchedEx branch

ggerganov committed 2 years ago
cuda : add TODO for calling cublas from kernel + using mem pool

ggerganov committed 2 years ago