pytorch

ce775118 - [DDP] Add `PackedSequence` support when `device_ids` is specified (#86614)

[DDP] Add `PackedSequence` support when `device_ids` is specified (#86614) Before this PR, if a user runs DDP with `device_ids` specified and with a `PackedSequence` input, then the execution will error with something like: ``` raise ValueError( ValueError: batch_sizes should always be on CPU. Instances of PackedSequence should never be created manually. They should be instantiated by functions like pack_sequence and pack_padded_sequences in nn.utils.rnn. https://pytorch.org/docs/stable/nn.html... ``` This is because the DDP forward calls `_to_kwargs()`, which calls `_recursive_to()`, which moves the inputs to GPU. However, `_is_namedtuple(packed_sequence)` returns `True`, leading to the branch `return [type(obj)(*args) for args in zip(*map(to_map, obj))]`, which tries to construct a `PackedSequence` directly via `type(obj)(*args)`, leading to the error. Repro for `_is_namedtuple(packed_sequence)` returning `True`: ``` import random import torch import torch.nn.utils.rnn as rnn_utils from torch.nn.parallel.scatter_gather import _is_namedtuple def _ordered_sequence(tensor_type): seqs = [tensor_type(random.randint(1, 256)) for _ in range(32)] seqs = [s.random_(-128, 128) for s in seqs] ordered = sorted(seqs, key=len, reverse=True) return ordered def _padded_sequence(tensor_type): ordered = _ordered_sequence(tensor_type) lengths = [len(i) for i in ordered] padded_tensor = rnn_utils.pad_sequence(ordered) return padded_tensor, lengths padded, lengths = _padded_sequence(torch.Tensor) packed = rnn_utils.pack_padded_sequence( padded, lengths, enforce_sorted=False) print(type(packed), packed.data.device) print(_is_namedtuple(packed)) ``` Test Plan: ``` python test/distributed/test_c10d_nccl.py -k test_ddp_packed_sequence ``` Without the fix, the added unit test fails with the expected error. Pull Request resolved: https://github.com/pytorch/pytorch/pull/86614 Approved by: https://github.com/rohan-varma