Revert "group constraints by arg (#101815)"
This reverts commit 03de15806e5d27ee4ef6d82dbcc66dac78f6e3bf.
Reverted https://github.com/pytorch/pytorch/pull/101815 on behalf of https://github.com/malfet due to it broke ExecuTorch and author was well aware about it"
