Remove cffi dependency as it doesn't look like we're using it (#92738)
Maybe this will go horribly wrong in CI but works fine without it locally!
Pull Request resolved: https://github.com/pytorch/pytorch/pull/92738
Approved by: https://github.com/kit1980, https://github.com/seemethere
