Ignore a stale best checkpoint recorded in a resumed trainer state (#48319)
* Ignore a stale best checkpoint recorded in a resumed trainer state
Resuming from a checkpoint whose trainer_state.json points
best_model_checkpoint at a directory that no longer exists (the usual
case when checkpoints were moved to another machine or the original run
directory was removed) kept the dead path all run long. If the resumed
run never set a new best, training finished and then crashed in the
save_total_limit=1 cleanup when os.path.samefile hit the missing path,
and rotate_checkpoints counted the ghost path as protected, keeping one
checkpoint beyond the configured limit for the whole run. Validate the
restored path once at state load: warn and clear it so every consumer
treats the run as having no best checkpoint yet.
Test Plan:
python -m pytest tests/trainer/test_trainer_checkpointing.py -k "stale" -q --no-header -p no:cacheprovider
fail-before (validation reverted): 1 failed, FileNotFoundError from os.path.samefile inside _finalize_training
pass-after: 1 passed
python -m pytest tests/trainer/test_trainer_checkpointing.py -q --no-header -p no:cacheprovider
45 passed, 11 skipped (staging tests needing hub env)
* Retrigger CI after runner-side checkout failures
* Retrigger CI after a runner-side container kill