DeepSpeed
1b022dcd - DeepNVMe: fix silent I/O failures on misaligned buffers (#8985)

Commit
3 days ago
DeepNVMe: fix silent I/O failures on misaligned buffers (#8985) ## Summary DeepNVMe does not check the result of an I/O request, so a request that the kernel fails is reported as done. ZeRO-Infinity reads and writes views into flat buffers at offsets that are often not 512-byte aligned, and O_DIRECT fails those on md RAID, LVM, and before Linux 6.0 on every block device. Swap-ins then leave stale data, swap-outs write nothing, and training goes on with wrong weights and gradients. This PR checks every completed request: a failed op makes `wait()`, `sync_pread()` and `sync_pwrite()` raise `RuntimeError` after the other ops complete. Tensors that are not contiguous, or not aligned as the file needs (`statx`, else 512), go through the existing bounce buffer. ZeRO-Infinity's fast init reads partitions of unaligned length through a padded staging tensor. Also, `io_submit` errors no longer abort on an assert, `io_pgetevents` is retried after `EINTR`, and a failed `io_queue_init` raises. ## Fixes Before → after, GH200, ext4 on md RAID 0, base `1a956270f`: 1. I/O on a pinned view 416 bytes past an aligned address: nothing moved, success returned → data moved. 2. ZeRO-3, NVMe parameter offload, Qwen2.5-0.5B: 60 of 290 parameters wrong after `initialize` → 0; the first loss now equals CPU offload's (12.206690). 3. ZeRO-3, NVMe optimizer offload, 2 ranks, Qwen2.5 config: losses leave CPU offload's at step 2 → equal. 4. A write past the file size limit: success with 2048 of 4096 bytes on disk → `RuntimeError: ... File too large`. 5. A write to a read-only file, or a signal to an AIO thread during a read: the process aborts on an `assert` → `RuntimeError: ... Bad file descriptor`, or the read completes. --------- Signed-off-by: Xinyu Lian <lian7@illinois.edu> Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>
Author
Parents
Loading