text-generation-inference
feat: experimental support for cuda graphs
#1428
Merged
Overview
Commits
17
Changes
View On GitHub

feat: experimental support for cuda graphs #1428

Narsil merged 17 commits into main from feat/exp_cuda_graphs
OlivierDehaene
Narsil Narsil force pushed from f9b1404e to 96a14e69 1 year ago
HuggingFaceDocBuilderDev
Narsil Narsil force pushed from b079b09d to 5fd301c1 1 year ago
OlivierDehaene fix: use TORCH_NCCL_AVOID_RECORD_STREAMS=1
1d929a24
OlivierDehaene feat: experimental support for cuda graphs
15fdd405
OlivierDehaene fix value
9904f669
OlivierDehaene fix env var
8260dc00
OlivierDehaene add log
ca20c304
OlivierDehaene fix speculate
33e94379
OlivierDehaene fix
4fd6e626
Narsil Disable cuda graph with speculation (for now) and update the docs.
bc95292e
Narsil Update the doc.
4b524a30
Narsil Fixing all quantization kernels.
3ce42ba7
Narsil Fixing AWQ.
903fbec6
Narsil Update dockerfile.
4b06f318
Narsil Update docs after rebase.
7143130b
Narsil Narsil force pushed from 6d3a1a11 to 7143130b 1 year ago
Narsil Fix for AWQ.
72f74bcb
Narsil Upgrade the ubuntu version too.
8f93b473
Narsil Going from earlier release (newer ones has bugs in shape it seems).
9a5d9723
Narsil Fixing AMD dockerfile.
c85f7374
Narsil Narsil merged 0d794af6 into main 1 year ago
Narsil Narsil deleted the feat/exp_cuda_graphs branch 1 year ago

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum