transformers
4b24a197 - [generate] stop synchronizing the accelerator on every decode step (#47975)

Commit
77 days ago
[generate] stop synchronizing the accelerator on every decode step (#47975) * host non blocking * pipeline the streamer's tokens too, instead of giving up on one `streamer.put(next_tokens.cpu())` is a host read per step -- the very stall `PipelinedStopCheck` exists to avoid -- so the pipeline turned itself off whenever a streamer was attached. That is the case every chat client takes: through `transformers serve` on MPS the win was 1.05x streaming against 1.46x not. The tokens now ride in the same slot as the finished flag, behind the same event, so a streamer costs no read of its own. It emits each token one step later, the same step the stop decision already pays, and never emits one past the stop: the token that overshoots is the copy left in flight when the loop breaks, dropped from the stream exactly as it is dropped from `input_ids`. Ending on `max_length` instead leaves that copy valid, and `pending` hands it over. Measured through `transformers serve`, Qwen3.5-4B Q4_K_M on an M2 Max: streaming 47.4 -> 66.2 tok/s, within 1% of the non-streaming path. Streamed ids are identical to the returned ids, on a run that stops on eos and on one that runs to `max_length`. * better design ! * testssss * better desgin for now * remove comments * update * update * fix * fix tests * update * correctly slice * update * better design * drop the unused `_user_supplied_cache` flag Left behind when the gate stopped consulting it: `cache_is_returned` now reads `return_dict_in_generate` alone, so nothing ever read the attribute back and setting it did nothing. * less verbose/clearer comment/docstrings * simplify conditions and fix user-defined cache * slightly simplify * deactivate past recording at the end * slight fix and comment * only record the past when the cache is actually returned `activate_past_recording` was guarded on the `cache` argument rather than on `self.cache`, so recording was switched on even where the rollback had been deliberately skipped. Two things follow from that: sliding-window layers stop being trimmed, which is the shape mismatch in the sliding-window test, and a kernel that writes the conv cache in place has to stand aside for the eager path, which costs ~18% a step on a gated-delta-net model. * rm com --------- Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com>
Author
Parents
Loading