avoid calling front() on empty working set (#21396)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/21396
ghimport-source-id: 7e57282099d2fd57c58c990b51ae933e427aecb2
Reviewed By: jamesr66a
Differential Revision: D15642663
Pulled By: suo
fbshipit-source-id: f9b467ba53f03438879bf3929da522aabaff2343
