PR #5021 ggml : add Flash Attention

ggml : add ggml_flash_attn_ext API

a1c004ef

ggerganov added help wanted

ggerganov added performance

ggerganov closed this 2 years ago

ggerganov reopened this 2 years ago

ggerganov marked this pull request as draft 2 years ago

ggml : fix GQA support in ggml_flash_attn_ext

fa7ebcca

Merge branch 'master' into gg/flash-attn

c3cdfffa

ggml : online attention (CPU)

a9681feb

metal : initial implementation

1173f49c

metal : f16 precision

528da751

metal : reduce branches

52ae0857

ggerganov force pushed to 52ae0857 2 years ago

metal : specialize for head size

b9732580

wip : 8 rows per simd group

8cde449b

wip : 4 rows per simd group

f31955f5

wip : template for rows per warp

a4b6341c

metal : parallelize across KV size

77d08f32

metal : parallel reduce across heads

17720fad

ggerganov force pushed to 17720fad 2 years ago

metal : efficient flash_attn_f16 implementation

1446a12b

metal : avoid redundant loads of the attention

d917746d

metal : scale and mask in matrix form

432ad04f

metal : fix comment

40ea8cd1

llama : avoid ggml_cast, use F32 query

f9ca5dcb

metal : add parallel reduce version (disabled)

6fea843b

ggerganov force pushed to 6fea843b 2 years ago

Merge branch 'master' into gg/flash-attn

b3dd7d97

metal : move output into local memory + optimize

77f6976a

ggerganov force pushed to 77f6976a 2 years ago

metal : add tests, fix scaling, support C > 32

ecc466a4

metal : improve precision

3a428a10

ggml : fix f16 mad

86128641

Merge branch 'master' into gg/flash-attn

0ad44baf

metal : minor

134c81c7

metal : support Q > 8

1db22d70

ggerganov force pushed to 1db22d70 2 years ago

tests : add ATTN tests

4794821a

metal : disable buffer allocation logs

abeaf0d9

tests : more

c6c1132e

metal : faster inner loop for C == 32

5fcb9c1c

metal : fix array initialization

d073e4f9

tests : ifdef

78df5527

Merge branch 'master' into gg/flash-attn

3d03bcb7

Merge branch 'master' into gg/flash-attn

2ddc9bbe

ggerganov force pushed 2 years ago

ggml : switch to padded F16 mask for ggml_soft_max, ggml_flash_attn_ext

8ad92dc1

ggerganov force pushed to 8ad92dc1 2 years ago

ggml : fix ggml_soft_max mask requirement

910b15bb

cuda : fix soft_max to use correct mask size

2e460137

cuda : add flash_attn kernel (wip)

5a19a9f6

Merge branch 'master' into gg/flash-attn

41d136b6

metal : optimize softmax for C > 32

56e45a23

metal : optimize softmax

cda5a60a

ggerganov force pushed to cda5a60a 2 years ago

tests : minor fix

c6769b94

cuda : avoid zeroing fragments

db1f3c48

tests : update dims

12eaa226

cuda : fix __hisinf() result check

b68a1122

cuda : avoid warp_reduce for smax

b150abe8

cuda : use int instead of int64_t

7c34655b

cuda : make loops use the same loop values

1f8a5924

cuda : unroll some of the loops

92472ea2

cuda : avoid __hisinf branches

c51f27c0

cuda : use half2 in softmax

b958151e

cuda : switch to 1 warp for bs > 16

a7b47156

cuda : speed-up reduce part of the kernel

3b1c4e76

cuda : unroll Q*K^T loop

5b263dd8

cuda : fix -INF block check

e04ff391

cuda : simplify softmax

cfd9732b

cuda : fix matrix names

ef68fac2

FSSRepo commented on 2024-02-03

cuda : minor

1846e92a

Merge branch 'master' into gg/flash-attn

6875997f

Merge branch 'master' into gg/flash-attn

31109ca0

llama : adapt to F16 KQ_pos

f249c997

Merge branch 'master' into gg/flash-attn

02a645e7

llama : adapt new models to F16 KQ_mask

6aefd112

Merge branch 'master' into gg/flash-attn

e307882c

ggml : fix F16 store (ARM NEON)

58c7f616

Merge branch 'master' into gg/flash-attn

9495d398

llama : fix type of KQ_mask and KQ_pos

3a468e6f

ggml : fix CPU soft_max

09532120

tests : add hs=256

e425810b

Merge branch 'master' into gg/flash-attn

013721df

cuda : fix build

6be02b59

metal : improve perf via smaller int registers

57c03b78

ggerganov force pushed to 57c03b78 2 years ago

Merge branch 'master' into gg/flash-attn

3e318e76

cuda : adapt soft_max to F16 mask and pos

08e69c50

CUDA: faster FlashAttention, kernel for bs == 1

75aa7b4b

16 cols for Phi-2

d59ac670

no vec for hs, no hs==256 ncols==32 for Volta

81da9198

adjust kernel selection logic

269374ed

4 warps, 256 stride for all D

cca6d027

no ncols == 64

68d793be

Multiple parallel blocks for batch size 1

3f777acf

fix compile warnings

e1ecd3b1

fix excessive KQ_b loads

bb0d51ac

fix cmake build

c63dfdf7

fix KV cache padding, NaN from INFINITY (#6438)

ee19a4ab

Merge branch 'master' into gg/flash-attn

89961dea

Merge branch 'master' into gg/flash-attn

2c41180e

llama : flash_attn cparam + fix defrag

599ce84a

server: support flash_attn param

40538572

server: bench: enable flash_attn param

5668c79e

CUDA: refactor host code, dyn. par. blocks

34f93bbb

fix flash_attn_vec_f16 race condition

6a3b8423

flush softmax exp below threshold to 0

ef9e1593

store temp KQ in registers

a5b0e2de

Calculate KQ as FP32 if KQV has GGML_PREC_F32

0bc67dd1

Add __hgt2_mask implementation for CUDA 11

2f538b95

fix KQ FP32 precision fpr parallel_blocks > 1

87968de9

llama-bench : add -fa,--flash-attn arg

260cdb2d

metal : add BS=1 kernel for flash attention (#6508)

105332cc

Merge branch 'master' into gg/flash-attn

fa9e8c66

metal : use F32 attention accumulators

c16a7c26

batched-bench : add fattn arg

9ca86987

ggerganov removed help wanted

ggerganov added need feedback

llama : simplify llama_build_kv_store

74d57f95

Merge branch 'master' into gg/flash-attn

1db66c1d

llama : adapt build_olmo to changes

e32b2817

ggml : fix arm fp16 store on windows

703c6e65

metal : clean-up

97eaece7

metal : clean-up kernel code

1a88565b

metal : minor

bc346166

Merge branch 'master' into gg/flash-attn

29f6ad8d

tests : remove benchmarks

52945429

ggml : fix avx512 const correctness

3badef1f

ggml : fix soft_max with bias on CPU

871fcb6e

ggerganov marked this pull request as ready for review 2 years ago

ggerganov requested a review from

slaren 2 years ago

slaren commented on 2024-04-21

common : print --flash-attn in help

a39217d4

ggml : fix num dimensions in ggml_flash_attn_ext

cb76d747

llama : force disable flash attention for incompatible models

c11d05fe

ggerganov force pushed 2 years ago

ggml : ggml_soft_max support F16/F32 mask/pos

f725ca90

ggerganov force pushed to f725ca90 2 years ago

cuda : uint -> uint32_t

5408d555

cuda : "constexpr dim3" -> "const dim3"

c70bfd7b

cuda : try to fix __hgt2_mask

c1293697

ggerganov force pushed from 78451ce4 to c1293697 2 years ago

ggml : add TODO's for F16/F32 mask/pos support in other backends

3864eea4

ggerganov force pushed to 3864eea4 2 years ago

slaren commented on 2024-04-23

llama : replace bool need_kq_pos with use_alibi

78d363b0

llama : prep ALiBi support for BERT models

19e8982f

llama : fix n_batch requirements

56657e52

cont

d228bf85

ggerganov commented on 2024-04-23

server : add help for --flash-attn arg

751591d5

Merge branch 'master' into gg/flash-attn

8937ec53

ggerganov force pushed from 1409defc 2 years ago

llama : disable FA for AMD

ce281b90

ggerganov force pushed to ce281b90 2 years ago

Merge branch 'master' into gg/flash-attn

1f77f497

tests : remove TMP_ATTN_BENCH

ff2c64a9

Merge branch 'master' into gg/flash-attn

cb3547ac

ggerganov requested a review from

slaren 2 years ago

llama : support save/load state with FA enabled

1fd5bc3d

slaren commented on 2024-04-25

Merge branch 'master' into gg/flash-attn

09d0381c

ci : add CUDA save-load-state tests

ac1c6d91

llama : llama_kv_cache_clear zeroes data + fix save-load seq

c225609f

llama : fix copy-paste errors, add TODO

bab346ba

llama : disallow incompatible states

0fc5c5eb

slaren commented on 2024-04-25

llama : update llama_state_get_size after v_trans field

1e590ac3

metal : remove tmp log

4f4c0249

llama : add static reminder for llama_state_get_size

9e387606

slaren approved these changes on 2024-04-26

FSSRepo approved these changes on 2024-04-28

Merge branch 'master' into gg/flash-attn

a1616e9f

Merge branch 'master' into gg/flash-attn

ca0275ce

metal : fix max nsg

e180fcd3

ci : fix arg order

c240ae23

ggerganov merged 9c67c277 into master 2 years ago

llama.cpp
ggml : add Flash Attention
#5021

Merged

ggml : add Flash Attention #5021

llama.cpp ggml : add Flash Attention #5021 Merged

ggml : add Flash Attention #5021

llama.cpp
ggml : add Flash Attention
#5021

Merged