[generate] stop synchronizing the accelerator on every decode step (#47975)
* host non blocking
* pipeline the streamer's tokens too, instead of giving up on one
`streamer.put(next_tokens.cpu())` is a host read per step -- the very stall `PipelinedStopCheck` exists
to avoid -- so the pipeline turned itself off whenever a streamer was attached. That is the case every
chat client takes: through `transformers serve` on MPS the win was 1.05x streaming against 1.46x not.
The tokens now ride in the same slot as the finished flag, behind the same event, so a streamer costs
no read of its own. It emits each token one step later, the same step the stop decision already pays,
and never emits one past the stop: the token that overshoots is the copy left in flight when the loop
breaks, dropped from the stream exactly as it is dropped from `input_ids`. Ending on `max_length`
instead leaves that copy valid, and `pending` hands it over.
Measured through `transformers serve`, Qwen3.5-4B Q4_K_M on an M2 Max: streaming 47.4 -> 66.2 tok/s,
within 1% of the non-streaming path. Streamed ids are identical to the returned ids, on a run that
stops on eos and on one that runs to `max_length`.
* better design !
* testssss
* better desgin for now
* remove comments
* update
* update
* fix
* fix tests
* update
* correctly slice
* update
* better design
* drop the unused `_user_supplied_cache` flag
Left behind when the gate stopped consulting it: `cache_is_returned` now
reads `return_dict_in_generate` alone, so nothing ever read the attribute
back and setting it did nothing.
* less verbose/clearer comment/docstrings
* simplify conditions and fix user-defined cache
* slightly simplify
* deactivate past recording at the end
* slight fix and comment
* only record the past when the cache is actually returned
`activate_past_recording` was guarded on the `cache` argument rather than on
`self.cache`, so recording was switched on even where the rollback had been
deliberately skipped. Two things follow from that: sliding-window layers stop
being trimmed, which is the shape mismatch in the sliding-window test, and a
kernel that writes the conv cache in place has to stand aside for the eager
path, which costs ~18% a step on a gated-delta-net model.
* rm com
---------
Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com>