transformers

5489cde1 - Warm the page cache before sharded loading (opt-in via HF_SHARD_PREFETCH)

Warm the page cache before sharded loading (opt-in via HF_SHARD_PREFETCH) The loader's per-tensor read pattern pulls a network filesystem at well under 1 GiB/s while large sequential reads sustain many times that (measured: 0.26-0.7 GiB/s vs 8.5 GiB/s on Lustre). With HF_SHARD_PREFETCH=<threads>, the local ranks split the shard list and stream it into the page cache before loading; the load then runs at memory speed. Measured on GLM-4.5-Air (206 GiB, cold, 8 GPUs): 60 s baseline vs 23 s prefetch + 10 s load.