Improve the quality of speech transformer (#656)
Summary:
Use original train batch size 32.
Source: https://github.com/kaituoxu/Speech-Transformer/blob/e6847772d6a786336e117a03c48c62ecbf3016f6/src/bin/train.py#L68
Added code to prefetch the data to GPU.
This model doesn't support changing inference batch size.
Profile of train:
Profile of eval:
Pull Request resolved: https://github.com/pytorch/benchmark/pull/656
Reviewed By: aaronenyeshi
Differential Revision: D33178568
Pulled By: xuzhao9
fbshipit-source-id: 153cf729be3e30fe0742812fb3f35bb838f18b3a
