transformers
Add CB
#38085
Merged
Overview
Commits
110
Changes
View On GitHub
Commits
  • stash for now
    McPatate committed 1 year ago
  • initial commit
    McPatate committed 1 year ago
  • small updated
    McPatate committed 1 year ago
  • up
    McPatate committed 1 year ago
  • up
    McPatate committed 1 year ago
  • works!
    McPatate committed 1 year ago
  • nits and fixes
    McPatate committed 1 year ago
  • don't loop too much
    McPatate committed 1 year ago
  • finish working example
    McPatate committed 1 year ago
  • update
    McPatate committed 1 year ago
  • fix the small freeblocks issue
    McPatate committed 1 year ago
  • feat: stream inputs to continuous batch
    McPatate committed 1 year ago
  • fix: update attn from `eager` to `sdpa`
    McPatate committed 1 year ago
  • refactor: fmt
    McPatate committed 1 year ago
  • refactor: cleanup unnecessary code
    McPatate committed 1 year ago
  • feat: add `update` fn to `PagedAttentionCache`
    McPatate committed 1 year ago
  • feat: broken optimal block size computation
    McPatate committed 1 year ago
  • fix: debugging invalid cache logic
    McPatate committed 1 year ago
  • fix: attention mask
    McPatate committed 1 year ago
  • refactor: use custom prompts for example
    McPatate committed 1 year ago
  • feat: add streaming output
    McPatate committed 1 year ago
  • fix: prefill split
    McPatate committed 1 year ago
  • fix: send decoded tokens when `prefilling_split` -> `decoding`
    McPatate committed 1 year ago
  • refactor: move logic to appropriate parent class
    McPatate committed 1 year ago
  • fix: remove truncation as we split prefilling anyways
    McPatate committed 1 year ago
  • feat: add paged attention forward
    McPatate committed 1 year ago
  • push Ggraoh>
    ArthurZucker committed 1 year ago
  • add paged sdpa
    ArthurZucker committed 1 year ago
  • Merge branch 'feat/stream_inputs_to_continuous_batch' of github.com:huggingface/transformers into feat/stream_inputs_to_continuous_batch
    ArthurZucker committed 1 year ago
  • update
    ArthurZucker committed 1 year ago
  • btter mps defaults
    ArthurZucker committed 1 year ago
  • feat: add progress bar for `generate_batch`
    McPatate committed 1 year ago
  • feat: add opentelemetry metrics (ttft + batch fill %age)
    McPatate committed 1 year ago
  • feat: add tracing
    McPatate committed 1 year ago
  • Add cuda graphs (#38059)
    ArthurZucker committed 1 year ago
  • revert llama changes
    ArthurZucker committed 1 year ago
  • fix merge conflicts
    ArthurZucker committed 1 year ago
  • fix: tracing and metrics
    McPatate committed 1 year ago
  • my updates
    ArthurZucker committed 1 year ago
  • update script default values
    ArthurZucker committed 1 year ago
  • fix block allocation issue
    ArthurZucker committed 1 year ago
  • Merge branch 'feat/stream_inputs_to_continuous_batch' of github.com:huggingface/transformers into feat/stream_inputs_to_continuous_batch
    ArthurZucker committed 1 year ago
  • fix prefill split attnetion mask
    ArthurZucker committed 1 year ago
  • no bugs
    ArthurZucker committed 1 year ago
  • add paged eager
    ArthurZucker committed 1 year ago
  • fix
    ArthurZucker committed 1 year ago
  • update
    ArthurZucker committed 1 year ago
  • style
    ArthurZucker committed 1 year ago
  • feat: add pytorch traces
    McPatate committed 1 year ago
  • fix
    ArthurZucker committed 1 year ago
  • fix
    ArthurZucker committed 1 year ago
  • refactor: remove pytorch profiler data
    McPatate committed 1 year ago
  • style
    ArthurZucker committed 1 year ago
  • Merge branch 'feat/stream_inputs_to_continuous_batch' of github.com:huggingface/transformers into feat/stream_inputs_to_continuous_batch
    ArthurZucker committed 1 year ago
  • nits
    ArthurZucker committed 1 year ago
  • Merge branch 'main' of github.com:huggingface/transformers into feat/stream_inputs_to_continuous_batch
    ArthurZucker committed 1 year ago
  • cleanup
    ArthurZucker committed 1 year ago
  • draft test file
    ArthurZucker committed 1 year ago
  • fix
    ArthurZucker committed 1 year ago
  • Merge branch 'feat/stream_inputs_to_continuous_batch' of github.com:huggingface/transformers into feat/stream_inputs_to_continuous_batch
    ArthurZucker committed 1 year ago
  • fix
    ArthurZucker committed 1 year ago
  • fix paged and graphs
    ArthurZucker committed 1 year ago
  • small renamings
    ArthurZucker committed 1 year ago
  • cleanups and push
    ArthurZucker committed 1 year ago
  • refactor: move tracing and metrics logic to utils
    McPatate committed 1 year ago
  • refactor: trace more blocks of code
    McPatate committed 1 year ago
  • nits
    ArthurZucker committed 1 year ago
  • nits
    ArthurZucker committed 1 year ago
  • Merge branch 'feat/stream_inputs_to_continuous_batch' of github.com:huggingface/transformers into feat/stream_inputs_to_continuous_batch
    ArthurZucker committed 1 year ago
  • update
    ArthurZucker committed 1 year ago
  • to profile or not to profile
    ArthurZucker committed 1 year ago
  • refactor: create new output object
    McPatate committed 1 year ago
  • causal by default
    ArthurZucker committed 1 year ago
  • Merge branch 'feat/stream_inputs_to_continuous_batch' of github.com:huggingface/transformers into feat/stream_inputs_to_continuous_batch
    ArthurZucker committed 1 year ago
  • cleanup but generations are still off for IDK what reason
    ArthurZucker committed 1 year ago
  • simplifications but not running still
    ArthurZucker committed 1 year ago
  • this does work.
    ArthurZucker committed 1 year ago
  • small quality of life updates
    ArthurZucker committed 1 year ago
  • nits
    ArthurZucker committed 1 year ago
  • updaet
    ArthurZucker committed 1 year ago
  • fix the scheduler
    ArthurZucker committed 1 year ago
  • fix warning
    ArthurZucker committed 1 year ago
  • ol
    ArthurZucker committed 1 year ago
  • fully fixed
    ArthurZucker committed 1 year ago
  • nits
    ArthurZucker committed 1 year ago
  • different generation parameters
    ArthurZucker committed 1 year ago
  • nice
    ArthurZucker committed 1 year ago
  • just style
    ArthurZucker committed 1 year ago
  • feat: add cache memory usage
    McPatate committed 1 year ago
  • feat: add kv cache free memory
    McPatate committed 1 year ago
  • feat: add active/waiting count & req latency
    McPatate committed 1 year ago
  • do the sampling
    ArthurZucker committed 1 year ago
  • Merge branch 'feat/stream_inputs_to_continuous_batch' of github.com:huggingface/transformers into feat/stream_inputs_to_continuous_batch
    ArthurZucker committed 1 year ago
  • fix: synchronize CUDA only if available and improve error handling in ContinuousBatchingManager
    ArthurZucker committed 1 year ago
  • fix on mps
    ArthurZucker committed 1 year ago
  • feat: add dashboard & histogram buckets
    McPatate committed 1 year ago
  • perf: improve waiting reqs data structures
    McPatate committed 1 year ago
  • attempt to compile, but we should only do it on mps AFAIK
    ArthurZucker committed 1 year ago
  • feat: decouple scheduling logic
    McPatate committed 1 year ago
  • just a draft
    ArthurZucker committed 1 year ago
  • + more commits ...
FAQ Terms Privacy Refunds Impressum
Loading