fix: handle exception when loading cache file in test_inference.py (#5802)
This PR is to fix CI failures such as
https://github.com/microsoft/DeepSpeed/actions/runs/10085903860/job/27887546470#step:8:3616
cc @tjruwase
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>