CUDA: add stream-based concurrency (#16991)
* CUDA: add stream-based concurrency
* HIP: fix hipStreamWaitEvent define and nodiscard warnings
* ggml-cuda: fix fusion inside stream
* ggml-cuda: fix bug w.r.t first stream launch
* ggml-cuda: format
* ggml-cuda: improve assert message
* ggml-cuda: use lambda instead of duplicating code
* ggml-cuda: add some more comments
* ggml-cuda: add more detailed comments about concurrency
* ggml-cuda: rename + remove unused var
* ggml-cuda: fix condition for stream launch
* ggml-cuda: address review comments, add destructor
* common.cuh: add is_valid for concurrent events
* common.cuh: make comment better
* update comment
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* update comment
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* common.cuh: fix lower_bound condition + remove join_node data from write_ranges
* ggml-cuda: fix overlap condition + shadowing parameter
---------
Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
