transformers
5e903d88 - feat: enable CB paged decode fast path for MPS hub kernels + sensible block_size default

Commit
153 days ago
feat: enable CB paged decode fast path for MPS hub kernels + sensible block_size default Two changes to make continuous batching work well on Apple Silicon (MPS): 1. initialization.py - ensure_decode_fast_path_is_available: Previously only FA3/CUDA and FA2/XPU could use the paged decode fast path (block tables + flash_attn_with_kvcache). Hub kernels like kernels-community/metal-flash-sdpa were excluded entirely, forcing max_blocks_per_request = 0 and losing the decode fast path even when the kernel ships flash_attn_with_kvcache. Now checks is_kernel(config._attn_implementation) so any hub kernel that provides flash_attn_with_kvcache is automatically picked up, regardless of device. This unblocks the metal-flash-sdpa paged-decode kernel (HF discussion #4 on kernels-community/metal-flash-sdpa) once its bindings land. 2. serve.py - MPS block_size default: The default block_size=256 causes a ~17x slowdown on MPS because the paged SDPA fallback (used when the kernel lacks block-table support) builds a full attention mask proportional to block_size, and MPS SDPA does not have the fused kernel that makes large blocks efficient on CUDA. Default to block_size=32 on MPS when not explicitly set. This is a temporary measure until flash_attn_with_kvcache ships in the metal-flash-sdpa kernel, at which point the block-table decode fast path will bypass the mask entirely. Tested on M3 Max with Qwen2.5-0.5B-Instruct: 5 sequential CB requests complete in ~4.3s each (vs 38-57s with block_size=256, or indefinite hang with the stale kernel revision on previous main).
Author
Parents
Loading