webdataset prototype - ListDirFilesIterableDataset (#48944)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/48944
This is a stack PR for webdataset prototype. I am trying to make each stack a separate dataset.
To make the implementation simple, each dataset will only support the basic functionality.
- [x] ListDirFilesDataset
- [x] LoadFilesFromDiskIterableDataset
- [x] ReadFilesFromTarIterableDataset
- [x] ReadFilesFromZipIterableDataset
- [x] RoutedDecoderIterableDataset
Test Plan: Imported from OSS
Reviewed By: izdeby
Differential Revision: D25541277
Pulled By: glaringlee
fbshipit-source-id: 9e738f6973493f6be1d5cc1feb7a91513fa5807c
