Remove attention_is_all_you_need and pytorch_struct (#1833)
Summary:
This is a follow-up of https://github.com/pytorch/benchmark/pull/1831
Pull Request resolved: https://github.com/pytorch/benchmark/pull/1833
Reviewed By: janeyx99
Differential Revision: D48432770
Pulled By: xuzhao9
fbshipit-source-id: 2ba8dff02d2ab0df17703427abad126708252762
