fix misaligned access #56325 (#56403)
Summary:
CC ngimel ptrblck
ref: https://github.com/pytorch/pytorch/issues/56325
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56403
Reviewed By: mruberry
Differential Revision: D27866625
Pulled By: ngimel
fbshipit-source-id: 9dff0e9749f8de57fac6a653f685c14854611a02
