llama.cpp
CUDA: faster Mixtral prompt processing for partial offloading
#4553

Merged

CUDA: faster Mixtral prompt processing for partial offloading #4553

JohannesGaessler merged 1 commit into ggml-org:master from JohannesGaessler:cuda-mixtral-partial-pp

JohannesGaessler force pushed 1 year ago

JohannesGaessler force pushed to 751687cc 1 year ago

slaren commented on 2023-12-21

CUDA: mul_mat_id always on GPU for batches >= 32

fcd0c2ca

JohannesGaessler force pushed from 751687cc to fcd0c2ca 1 year ago

slaren approved these changes on 2023-12-21

JohannesGaessler merged 91544948 into master 1 year ago

Reviewers

slaren

Assignees

No one assigned

Labels

None yet

Milestone

No milestone