transformers

461d017a - Make `td` recover instead of hanging, and free GPU memory between jobs

Make `td` recover instead of hanging, and free GPU memory between jobs Three fixes from using it: - `main()` is now optional. Importing a job script already runs its module body, so a plain top-level script is a valid job; `main()` is still called when the script defines one. - The client polled nothing: it blocked in `store.get(done:N)` against a store whose timeout is measured in days. When the pool went away without closing the socket -- a node released from the allocation, or one rank wedged inside a collective so the verdict is never written -- `td` sat there forever with the job apparently finished, and nothing further could be submitted. It now polls with a liveness check and exits with a reason. A client that dies between reserving a slot and naming a script leaves a `__skip__` placeholder, so the ranks do not park on a slot nobody will fill. - GPU memory is reclaimed between jobs. Dropping the job module makes its tensors garbage, but reference cycles wait for a gc pass and the caching allocator holds freed blocks, so a long-lived pool looked permanently full and the next job fragmented against a cache it could not use. Slurm defaults also got more accurate: the allocation is looked up when SLURM_JOB_ID is not exported (submitting from a login shell on an allocated node is normal), and per-node GPU counts come from `scontrol show job -d`, since `--gpus=4` over two nodes is a total that Slurm may hand back as 3+1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>