transformers
4c0dafa1 - Batch Rebalance Data Sampler (#47340)

Commit
80 days ago
Batch Rebalance Data Sampler (#47340) * Add BatchRebalanceSampler for cost-balanced distributed training New sampler that balances sequence length cost across DP ranks while minimizing padding waste. Uses Poorman's Batch Rebalance algorithm with configurable cost model. Changes: - trainer_pt_utils.py: Add BatchRebalanceSampler class - training_args.py: Add 'batch_rebalance' to train_sampling_strategy choices, batch_rebalance_alpha and batch_rebalance_max_tokens params - trainer.py: Add batch_rebalance branch in _get_train_sampler Usage: TrainingArguments(train_sampling_strategy='batch_rebalance', batch_rebalance_alpha=0.001) * Add unit tests for BatchRebalanceSampler Tests cover: - Basic correctness (coverage, micro-batch count) - No overlap between ranks within same global batch - Effective batch size invariant - max_tokens constraint enforcement - Determinism (same seed = same result) - Custom cost_fn override * Fix: remove invalid super().__init__ call in BatchRebalanceSampler PyTorch's Sampler base class doesn't accept data_source kwarg. Drop the super().__init__() call entirely (consistent with other HF samplers like LengthGroupedSampler). * Remove unused micro_batch_size param, fix silent data loss in max_tokens rebalance - Remove micro_batch_size from BatchRebalanceSampler: it was stored but never read anywhere in the class, and its name was misleading since it did not reflect the actual minimum per-group size used by the algorithm. - Fix a bug in _poormans_rebalance where counts[gi] -= 1 was applied unconditionally before searching for a donor group, causing samples to be silently dropped when max_tokens could not be satisfied and no group had spare capacity. Now the decrement only happens once a donor is found, and a warning is logged (with a fallback to exceed max_tokens rather than drop samples) when no donor is available. - Fix docstring typo (Poorman's -> Poor Man's). - Update/add tests: drop micro_batch_size from existing tests, add a regression test for the max_tokens data-loss fix, and rewrite the cost_fn test to directly assert on _cost() output instead of an unstable partition overlap comparison. * Fix batch_rebalance sampling strategy: missing import and DataLoader wiring BatchRebalanceSampler yields a full batch of sample indices per iteration (BatchSampler semantics), but _get_dataloader() was passing it via DataLoader's sampler= kwarg together with batch_size=, which expects the sampler to yield one sample index at a time. This caused 'TypeError: list indices must be integers or slices, not list' as soon as train_sampling_strategy="batch_rebalance" was used with the Trainer. Also fixes a NameError (BatchEncoding was referenced in _get_train_sampler but never imported), which was the first failure hit before reaching the DataLoader wiring bug. Together these two bugs meant the batch_rebalance strategy could never actually run end-to-end through Trainer.train(); verified with a CPU multi-process (torchrun --nproc_per_node=4) benchmark on Qwen2.5-0.5B with a synthetic variable-length dataset after the fix. * Remove unused variable flagged by ruff F841 * Fix import sort order for BatchRebalanceSampler (ruff I001) * Apply ruff format to changed files * Replace dead count-adjustment code with explicit ValueError When n < K * min_per_group, there are not enough samples to fill all groups. Previously this was silently patched by trimming counts (and a dead 'total < n' branch that could never trigger). Now it raises a clear error telling the user to reduce effective_batch_size or increase dataset size. * Fix can_transfer accuracy and iterate all lo_candidates in rebalance - Replace can_increase(cts, gi) with can_transfer(cts, from_idx, to_idx) which simulates the actual -1/+1 transfer to compute the correct start index and max_len for the token limit check. The previous implementation used stale positions that didn't account for shifts caused by the donor group shrinking. - When the lowest-cost group cannot accept a transfer due to max_tokens, iterate through remaining candidates by ascending cost instead of falling through to the nail-house branch or breaking entirely. * Scale BatchRebalanceSampler iteration budget with effective batch size Each iteration of the rebalance loop moves exactly one sample between groups, so worst-case convergence is O(n). The fixed `K * 30` cap could therefore truncate before convergence for large `effective_batch_size` with a small `dp_size * grad_accum` (K), leaving the partition at a non-optimal cost spread. Use `max(K * 30, effective_batch_size)` as the budget: the `effective_batch_size` term covers the move-dominated regime (large batch, small K), while `K * 30` remains a floor for the detection-dominated regime (tiny batch, n ~= K) where convergence is reached via `spread == 0` / `seen` rather than the move budget. `best_counts` is still tracked, so hitting the cap only degrades the result gracefully (no samples are dropped). Adds a regression test (large n=246, small K=2, heavily skewed lengths) that asserts full sample coverage across all ranks and convergence to the brute-force-optimal contiguous K=2 cost spread. * Fix BatchRebalanceSampler correctness issues in Trainer integration Three review fixes for PR #47340: defeat double-sharding by neutralizing accelerate's BatchSamplerShard wrapper after prepare; fix effective_batch_size undercount (train_batch_size // world_size undercounts since train_batch_size excludes world_size in DDP); honor drop_last=False by yielding the padded trailing partial batch and passing dataloader_drop_last from the Trainer. Adds regression tests for all three. * Fix UnboundLocalError on `sampler` for iterable datasets in Trainer._get_dataloader `sampler` was only assigned inside the non-IterableDataset branch, but the post-`accelerator.prepare` check referenced it unconditionally. Initialize `sampler` to None so IterableDataset runs no longer raise. * Address sywangyi review: effective_batch_size DP fix, bounded tail padding, DDP integration test - trainer.py: effective_batch_size = train_batch_size * grad_accum * world_size (was per_device_train_batch_size * ...), so the DP (single-process, multi-GPU) case no longer drops the n_gpu factor. Covers both DDP and DP. - trainer_pt_utils.py: when drop_last=False and the remainder < dp_size*grad_accum, pad the tail up to K (dp_size*grad_accum) instead of the full effective_batch_size, avoiding heavy over-weighting of replayed samples in the final optimizer step. - test_trainer_data.py: extend effective_batch_size regression test to cover both DDP and DP; assert tail padding yields eff_bs + K (not eff_bs + eff_bs). - Add real DDP integration test (batch_rebalance_ddp.py + test_batch_rebalance_no_double_sharding) that exercises Trainer.get_train_dataloader -> accelerator.prepare -> patch path end-to-end under accelerate launch, verifying the structure-dependent wrapper detection and exact global sample coverage (no double-sharding drops). * Print summary line in batch_rebalance_ddp worker script execute_subprocess_async (the distributed test harness) treats a subprocess that produces no stdout/stderr as a failure. When launched with a full --config_file the script was silent on success, so the test raised 'produced no output'. Emit a one-line summary on the main process. * Trim redundant rationale from effective_batch_size comment * Apply ruff format to test_trainer_distributed_ddp * Pad tail by wrapping order when dataset is smaller than K order[:pad] returns at most len(order) elements, so for len(dataset) < K the tail underfills and _poormans_rebalance(..., min_per_group=1) raises ValueError. Repeat the order (math.ceil) to yield exactly pad padding indices. Adds an n=1, K=8 regression test. * Address comments Signed-off-by: Guokai Ma <guokai.ma@gmail.com> * Remove batch_rebalance_alpha and batch_rebalance_max_tokens args Per review: no ablation was run on these knobs and there is no user demand, so they do not belong in the public API. The quadratic cost weight is fixed at 0.001 (previous default) as an internal class constant; intercept and cost_fn were unused and are deleted along with the max_tokens machinery (feasibility repair loop, can_transfer, _padded_tokens, _within_token_limit). Output is bit-identical to the previous defaults, verified against the pre-change implementation over 30 randomized inputs. Signed-off-by: Guokai Ma <guokai.ma@gmail.com> * Simplify BatchRebalanceSampler for readability Per review: document and restructure the sampler internals. - _assign: explain the snake deal (groups ranked by cost, slot k takes ranks [k*dp_size, (k+1)*dp_size) so every slot has the same cost profile). - __iter__: deduplicate the full-batch loop and drop_last=False tail block into a single loop over global batches plus a _yield_global_batch helper. - _poormans_rebalance -> _balance_groups: drop the three redundant kwargs (derivable from self), document the hill-climbing search, and extract the freeze fallback into _slot_groups (pure: which groups share the capped group's slot) and _relieve_and_freeze (top up the slot from outside groups, then freeze it). Output verified bit-identical to the previous implementation over 286 randomized inputs, including adversarial freeze-prone configurations. Signed-off-by: Guokai Ma <guokai.ma@gmail.com> * Consolidate batch_rebalance tests and extend the DDP e2e to two epochs Per review: keep it to a few unit tests and focus on the DDP integration test. - Merge basic/no_overlap/eff_batch_size/deterministic into one basic test asserting coverage, no cross-rank overlap, exact per-global-batch counts and seed determinism (original hand-picked lengths kept). - Merge the four drop_last tests into one test covering: tail dropped with drop_last=True, tail yielded as-is otherwise, padding up to K (not effective_batch_size) when the remainder is smaller, and order wrap-around when the whole dataset is smaller than K. - Drop test_batch_rebalance_defeats_double_sharding: it simulated accelerate's BatchSamplerShard by hand; the real path is covered by the DDP e2e test. - Extend the DDP e2e worker to actually train for two epochs and assert the per-epoch invariants end to end: each epoch covers all samples across ranks, set_epoch reaches the wrapped sampler (inner_epoch advances) and later epochs reshuffle. Validated on 2x RTX 5090. Every assertion from the removed tests maps into the consolidated ones. Signed-off-by: Guokai Ma <guokai.ma@gmail.com> * Reuse train.py for the batch_rebalance DDP test Drop the bespoke batch_rebalance_ddp.py worker script and its white-box assertions in favour of the shared train.py, so the sampler is exercised against a real model end to end. train.py gains a --padding option because the default max_length padding makes every sample the same length, which leaves the rebalancing with nothing to do. * [docs] Add batch_rebalance sampling strategy to the training guide * [docs] Move batch_rebalance sampling guide to trainer_recipes Move the batch_rebalance sampling strategy docs from training.md to a dedicated section in trainer_recipes.md, next to Dataloader performance, following review feedback. The new section explains the two imbalances it addresses (padding within a micro-batch and idle time across devices), how the sampler works (sort then shard), the average per_device_train_batch_size semantics, and how to skip the startup length scan with precomputed lengths. --------- Signed-off-by: Guokai Ma <guokai.ma@gmail.com> Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
Author
Parents
Loading