xla
Fix race condition when use multi threads to transfer data in parallel Loader
#5267
Merged
Merged
Fix race condition when use multi threads to transfer data in parallel Loader #5267
JackCaoG
merged
2 commits into
pytorch:master
from
aws-tianquaw:fix-parallel-loader
Fix race condition when use > 1 threads to transfer data in parallel …
8f4b5243
JackCaoG2 years ago
@Liyang90 Do you have cycle to take a look at this one?
chandrasekhard2
approved these changes
on 2023-07-13
JackCaoG2 years ago
@aws-tianquaw DO you mind rebasing this pr? Then ci should start passing
Merge branch 'pytorch:master' into fix-parallel-loader
6cc17a01
Login to write a write a comment.
This pull request includes a fix to the issue brought by the previous pull request that added feature to increase the number of host to device transfer threads. When using > 1 workers to transfer data, a possible race condition could happen and caused some data to be lost:
Currently, each thread will call queue.close_write() when there is no more data in the loader queue. But if one thread called
queue.close_write()while others still need to to put data to the queue, these batches of data will be lost. In this case. thenext_item()call in this line will returnNonewhile it should return valid data.Ideally, we should only call
queue.close_write()after all threads have completed writing data to the queue to avoid this race condition.