DeepNVMe: fix silent I/O failures on misaligned buffers (#8985)
## Summary
DeepNVMe does not check the result of an I/O request, so a request that
the kernel fails is reported as done. ZeRO-Infinity reads and writes
views into flat buffers at offsets that are often not 512-byte aligned,
and O_DIRECT fails those on md RAID, LVM, and before Linux 6.0 on every
block device. Swap-ins then leave stale data, swap-outs write nothing,
and training goes on with wrong weights and gradients.
This PR checks every completed request: a failed op makes `wait()`,
`sync_pread()` and `sync_pwrite()` raise `RuntimeError` after the other
ops complete. Tensors that are not contiguous, or not aligned as the
file needs (`statx`, else 512), go through the existing bounce buffer.
ZeRO-Infinity's fast init reads partitions of unaligned length through a
padded staging tensor. Also, `io_submit` errors no longer abort on an
assert, `io_pgetevents` is retried after `EINTR`, and a failed
`io_queue_init` raises.
## Fixes
Before → after, GH200, ext4 on md RAID 0, base `1a956270f`:
1. I/O on a pinned view 416 bytes past an aligned address: nothing
moved, success returned → data moved.
2. ZeRO-3, NVMe parameter offload, Qwen2.5-0.5B: 60 of 290 parameters
wrong after `initialize` → 0; the first loss now equals CPU offload's
(12.206690).
3. ZeRO-3, NVMe optimizer offload, 2 ranks, Qwen2.5 config: losses leave
CPU offload's at step 2 → equal.
4. A write past the file size limit: success with 2048 of 4096 bytes on
disk → `RuntimeError: ... File too large`.
5. A write to a read-only file, or a signal to an AIO thread during a
read: the process aborts on an `assert` → `RuntimeError: ... Bad file
descriptor`, or the read completes.
---------
Signed-off-by: Xinyu Lian <lian7@illinois.edu>
Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>