transformers
d56c55bf - [TableTransformer, PI0] Fix stale expected values and OOM in integration tests (#48198)

Commit
91 days ago
[TableTransformer, PI0] Fix stale expected values and OOM in integration tests (#48198) * [WIP] Fix integration test failures: TableTransformer, T5 compile, PI0 - TableTransformer: stale expected logit/box values for A10G (sm_86) - T5: CUDAGraphs tensor overwrite in test_compile_static_cache, likely introduced by #47731 (stop setting static cache as model attribute) - PI0: OOM in test_train_pi0_base_libero (float32 training on 22 GiB GPU) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [TableTransformer] Fix stale expected values for A10G (sm_86) Add ("cuda", 8) entries to expected_logits_data and expected_boxes_data in test_table_detection, captured on A10G (compute capability 8.6). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [PI0] Fix OOM in test_train_pi0_base_libero by using bfloat16 The test loaded pi0_base (PaLiGemma-3B-based) in float32 and ran full Trainer training, which OOMs on a 22 GiB A10G GPU. Switch to bfloat16 (torch_dtype=torch.bfloat16 + TrainingArguments(bf16=True)) to halve the memory footprint. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [TableTransformer] Use (None, None) fallback key instead of ("cuda", None) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [PI0] Fix OOM in test_train_pi0_base_libero (bfloat16 + adafactor) The test loaded pi0_base (PaLiGemma-3B backbone) in float32 with default Adam optimizer, which OOMs on a 22 GiB A10G GPU: Adam stores exp_avg + exp_avg_sq (~12 GB) on top of the model weights (~6 GB) and gradients (~6 GB), totalling ~24 GB regardless of batch size. Fix: - torch_dtype=torch.bfloat16 (halves model footprint) - optim="adafactor" (factored 2nd moment, no 1st moment; ~0.3 GB vs 12 GB) - per_device_train_batch_size=2 (reduces activation memory) - max_steps=6 (one extra step for a clearer trend) - Relax the loss assertion to allow 1 non-monotone step: with a small batch size and bf16 precision, occasional gradient noise can cause a single step to tick up even as the overall trend is downward. Verified on A10G runner (22 GiB). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [PI0] Add comment explaining why adafactor is used Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [PI0] Fix dtype mismatch on multi-GPU machines (DataParallel + bfloat16) On multi-GPU runners Trainer wraps the model in nn.DataParallel, which scatters float32 dataset inputs to each GPU replica. The model weights are bfloat16, causing a dtype mismatch in the first linear layer. Setting device_map={"": torch_device} causes from_pretrained to record model.hf_device_map, which Trainer checks to skip DataParallel wrapping. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [PI0] Extend device_map comment to clarify single-device intent Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [PI0] Fix DataParallel dtype mismatch on multi-GPU runners Replace device_map approach (which only skips DataParallel when the model spans multiple GPUs) with explicitly setting trainer.args._n_gpu=1 before training. This reliably prevents Trainer from wrapping the model in nn.DataParallel regardless of how many GPUs are available. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [PI0] Use @require_torch_non_multi_accelerator to skip on multi-GPU runners On multi-GPU machines Trainer wraps the model in nn.DataParallel, which scatters float32 dataset inputs to replicas while the model weights are bfloat16, causing a dtype mismatch. This test is not intended to cover multi-GPU training, so skip it on multi-GPU setups via the decorator. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Author
Parents
Loading