pytorch
f3f41088 - [DataPipe] Remove ZipArchiveReader's dependency on FileLoader (#64786)

Commit
3 years ago
[DataPipe] Remove ZipArchiveReader's dependency on FileLoader (#64786) Summary: Stack from [ghstack](https://github.com/ezyang/ghstack): * https://github.com/pytorch/pytorch/issues/64788 * __->__ https://github.com/pytorch/pytorch/issues/64786 This PR removes ZipArchiveReader's dependency on FileLoader DataPipe, by allowing it to use a IterDataPipe of path names as input rather than a tuple of path name and a stream. It also adds additional tests to ensure that the DataPipe is functioning properly when it is read multiple times or reset half way through reading. The whole stack fixes issues related to unclosed buffer stream (see https://github.com/pytorch/pytorch/issues/64281). cc VitalyFedyunin ejguan Pull Request resolved: https://github.com/pytorch/pytorch/pull/64786 Reviewed By: ngimel Differential Revision: D30870968 Pulled By: NivekT fbshipit-source-id: 64b04d1697b99772f2fa20fc141668e6b8e18c41
Author
Parents
Loading