Revert "Fix bug in flatbuffer deserialization"
This reverts commit 7e72c96b104e1f3021aa491e57b36150ddffca8f.
Reverted https://github.com/pytorch/pytorch/pull/78344 on behalf of https://github.com/tugsbayasgalan due to as we need to land it in fbcode asap
