DeepSpeed
0886a10b - refactor: remove naive HF generate path from HybridEngineRollout

Commit
86 days ago
refactor: remove naive HF generate path from HybridEngineRollout The naive path (HF generate with num_return_sequences) is strictly worse than the continuous-batching path: no KV reuse, no slot replacement, and it required maintaining _fallback_generate, _build_rollout_batch, and _hybrid_engine_has_accel helpers. Remove the naive path and make continuous_batching_size default to 1. All generation now goes through CB (with or without graph capture). Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Author
Parents
Loading