cog
9c894d66 - Update Worker to support concurrent predictions

Commit
1 year ago
Update Worker to support concurrent predictions This adds a max_concurrency parameter to Worker that allows it to accept predictions while others are already in-flight. I have removed the PROCESSING WorkerState; in concurrent mode, there is no distinction between READY and PROCESSING because we might be able to accept a prediction in either case. Worker now keeps track of multiple in-flight predictions in a dictionary, keyed on tag. Tags are required if max_concurrency > 1. Otherwise tags are optional (and, if omitted, we store the prediction with tag=None). There is one awkward place which is _prepare_payload(). As I understand it, this synchronously downloads URLFiles, which will block us from processing any other updates from the child while we download the URL.
Author
Committer
Parents
Loading