prevent generating caffe2::mkldnn for multiple times (#25257)
Summary:
This is a similar problem to https://github.com/pytorch/pytorch/issues/25004. After the merge of https://github.com/pytorch/pytorch/issues/25167, I recompiled torch and discovered another similar bug.
ezyang please take a look
Pull Request resolved: https://github.com/pytorch/pytorch/pull/25257
Differential Revision: D17528116
Pulled By: ezyang
fbshipit-source-id: 1657d9ee6dced3548f246010b05e2b3c25c37dee
