transformers
abf9ff0f - generate: drop an all-ones attention mask from model_kwargs

Commit
67 days ago
generate: drop an all-ones attention mask from model_kwargs Padding can only come from the prompt, so settle once whether the inputs are padded and, when they are not, take the mask out of `model_kwargs` entirely - the per-step `torch.cat` that grows it goes with it. `_prefill` used the mask's length to tell whether `input_ids` holds the whole sequence or only the new tokens, so that answer is recorded before the mask is dropped.
Author
Parents
Loading