[FX] Fix cases when we should not fuse due to more than one users of intermediate node (#66472)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/66472
A follow up of https://github.com/pytorch/pytorch/pull/66362. Same fix.
Test Plan:
```
buck test mode/dev-nosan caffe2/torch/fb/fx2trt:test_fuse_permute_matmul_trt
buck test mode/dev-nosan caffe2/torch/fb/fx2trt:test_fuse_permute_linear_trt
```
Reviewed By: wushirong, 842974287
Differential Revision: D31567662
fbshipit-source-id: 2c9e6a138fc31996d790fd4d79e0bf931507fc99
