transformers
36deb0b5 - Ignore a stale best checkpoint recorded in a resumed trainer state (#48319)

Commit
1 day ago
Ignore a stale best checkpoint recorded in a resumed trainer state (#48319) * Ignore a stale best checkpoint recorded in a resumed trainer state Resuming from a checkpoint whose trainer_state.json points best_model_checkpoint at a directory that no longer exists (the usual case when checkpoints were moved to another machine or the original run directory was removed) kept the dead path all run long. If the resumed run never set a new best, training finished and then crashed in the save_total_limit=1 cleanup when os.path.samefile hit the missing path, and rotate_checkpoints counted the ghost path as protected, keeping one checkpoint beyond the configured limit for the whole run. Validate the restored path once at state load: warn and clear it so every consumer treats the run as having no best checkpoint yet. Test Plan: python -m pytest tests/trainer/test_trainer_checkpointing.py -k "stale" -q --no-header -p no:cacheprovider fail-before (validation reverted): 1 failed, FileNotFoundError from os.path.samefile inside _finalize_training pass-after: 1 passed python -m pytest tests/trainer/test_trainer_checkpointing.py -q --no-header -p no:cacheprovider 45 passed, 11 skipped (staging tests needing hub env) * Retrigger CI after runner-side checkout failures * Retrigger CI after a runner-side container kill
Author
Parents
Loading