Batch Rebalance Data Sampler (#47340)
* Add BatchRebalanceSampler for cost-balanced distributed training
New sampler that balances sequence length cost across DP ranks while
minimizing padding waste. Uses Poorman's Batch Rebalance algorithm with
configurable cost model.
Changes:
- trainer_pt_utils.py: Add BatchRebalanceSampler class
- training_args.py: Add 'batch_rebalance' to train_sampling_strategy
choices, batch_rebalance_alpha and batch_rebalance_max_tokens params
- trainer.py: Add batch_rebalance branch in _get_train_sampler
Usage:
TrainingArguments(train_sampling_strategy='batch_rebalance',
batch_rebalance_alpha=0.001)
* Add unit tests for BatchRebalanceSampler
Tests cover:
- Basic correctness (coverage, micro-batch count)
- No overlap between ranks within same global batch
- Effective batch size invariant
- max_tokens constraint enforcement
- Determinism (same seed = same result)
- Custom cost_fn override
* Fix: remove invalid super().__init__ call in BatchRebalanceSampler
PyTorch's Sampler base class doesn't accept data_source kwarg.
Drop the super().__init__() call entirely (consistent with other
HF samplers like LengthGroupedSampler).
* Remove unused micro_batch_size param, fix silent data loss in max_tokens rebalance
- Remove micro_batch_size from BatchRebalanceSampler: it was stored but never
read anywhere in the class, and its name was misleading since it did not
reflect the actual minimum per-group size used by the algorithm.
- Fix a bug in _poormans_rebalance where counts[gi] -= 1 was applied
unconditionally before searching for a donor group, causing samples to be
silently dropped when max_tokens could not be satisfied and no group had
spare capacity. Now the decrement only happens once a donor is found, and a
warning is logged (with a fallback to exceed max_tokens rather than drop
samples) when no donor is available.
- Fix docstring typo (Poorman's -> Poor Man's).
- Update/add tests: drop micro_batch_size from existing tests, add a
regression test for the max_tokens data-loss fix, and rewrite the cost_fn
test to directly assert on _cost() output instead of an unstable partition
overlap comparison.
* Fix batch_rebalance sampling strategy: missing import and DataLoader wiring
BatchRebalanceSampler yields a full batch of sample indices per iteration
(BatchSampler semantics), but _get_dataloader() was passing it via
DataLoader's sampler= kwarg together with batch_size=, which expects the
sampler to yield one sample index at a time. This caused
'TypeError: list indices must be integers or slices, not list' as soon as
train_sampling_strategy="batch_rebalance" was used with the Trainer.
Also fixes a NameError (BatchEncoding was referenced in _get_train_sampler
but never imported), which was the first failure hit before reaching the
DataLoader wiring bug.
Together these two bugs meant the batch_rebalance strategy could never
actually run end-to-end through Trainer.train(); verified with a CPU
multi-process (torchrun --nproc_per_node=4) benchmark on Qwen2.5-0.5B with a
synthetic variable-length dataset after the fix.
* Remove unused variable flagged by ruff F841
* Fix import sort order for BatchRebalanceSampler (ruff I001)
* Apply ruff format to changed files
* Replace dead count-adjustment code with explicit ValueError
When n < K * min_per_group, there are not enough samples to fill all groups.
Previously this was silently patched by trimming counts (and a dead 'total < n'
branch that could never trigger). Now it raises a clear error telling the user
to reduce effective_batch_size or increase dataset size.
* Fix can_transfer accuracy and iterate all lo_candidates in rebalance
- Replace can_increase(cts, gi) with can_transfer(cts, from_idx, to_idx)
which simulates the actual -1/+1 transfer to compute the correct
start index and max_len for the token limit check. The previous
implementation used stale positions that didn't account for shifts
caused by the donor group shrinking.
- When the lowest-cost group cannot accept a transfer due to max_tokens,
iterate through remaining candidates by ascending cost instead of
falling through to the nail-house branch or breaking entirely.
* Scale BatchRebalanceSampler iteration budget with effective batch size
Each iteration of the rebalance loop moves exactly one sample between groups,
so worst-case convergence is O(n). The fixed `K * 30` cap could therefore
truncate before convergence for large `effective_batch_size` with a small
`dp_size * grad_accum` (K), leaving the partition at a non-optimal cost spread.
Use `max(K * 30, effective_batch_size)` as the budget: the `effective_batch_size`
term covers the move-dominated regime (large batch, small K), while `K * 30`
remains a floor for the detection-dominated regime (tiny batch, n ~= K) where
convergence is reached via `spread == 0` / `seen` rather than the move budget.
`best_counts` is still tracked, so hitting the cap only degrades the result
gracefully (no samples are dropped).
Adds a regression test (large n=246, small K=2, heavily skewed lengths) that
asserts full sample coverage across all ranks and convergence to the
brute-force-optimal contiguous K=2 cost spread.
* Fix BatchRebalanceSampler correctness issues in Trainer integration
Three review fixes for PR #47340: defeat double-sharding by neutralizing accelerate's BatchSamplerShard wrapper after prepare; fix effective_batch_size undercount (train_batch_size // world_size undercounts since train_batch_size excludes world_size in DDP); honor drop_last=False by yielding the padded trailing partial batch and passing dataloader_drop_last from the Trainer. Adds regression tests for all three.
* Fix UnboundLocalError on `sampler` for iterable datasets in Trainer._get_dataloader
`sampler` was only assigned inside the non-IterableDataset branch, but
the post-`accelerator.prepare` check referenced it unconditionally.
Initialize `sampler` to None so IterableDataset runs no longer raise.
* Address sywangyi review: effective_batch_size DP fix, bounded tail padding, DDP integration test
- trainer.py: effective_batch_size = train_batch_size * grad_accum * world_size
(was per_device_train_batch_size * ...), so the DP (single-process, multi-GPU) case
no longer drops the n_gpu factor. Covers both DDP and DP.
- trainer_pt_utils.py: when drop_last=False and the remainder < dp_size*grad_accum,
pad the tail up to K (dp_size*grad_accum) instead of the full effective_batch_size,
avoiding heavy over-weighting of replayed samples in the final optimizer step.
- test_trainer_data.py: extend effective_batch_size regression test to cover both DDP
and DP; assert tail padding yields eff_bs + K (not eff_bs + eff_bs).
- Add real DDP integration test (batch_rebalance_ddp.py + test_batch_rebalance_no_double_sharding)
that exercises Trainer.get_train_dataloader -> accelerator.prepare -> patch path end-to-end
under accelerate launch, verifying the structure-dependent wrapper detection and exact
global sample coverage (no double-sharding drops).
* Print summary line in batch_rebalance_ddp worker script
execute_subprocess_async (the distributed test harness) treats a subprocess that
produces no stdout/stderr as a failure. When launched with a full --config_file the
script was silent on success, so the test raised 'produced no output'. Emit a one-line
summary on the main process.
* Trim redundant rationale from effective_batch_size comment
* Apply ruff format to test_trainer_distributed_ddp
* Pad tail by wrapping order when dataset is smaller than K
order[:pad] returns at most len(order) elements, so for len(dataset) < K the tail
underfills and _poormans_rebalance(..., min_per_group=1) raises ValueError. Repeat the
order (math.ceil) to yield exactly pad padding indices. Adds an n=1, K=8 regression test.
* Address comments
Signed-off-by: Guokai Ma <guokai.ma@gmail.com>
* Remove batch_rebalance_alpha and batch_rebalance_max_tokens args
Per review: no ablation was run on these knobs and there is no user demand,
so they do not belong in the public API. The quadratic cost weight is fixed
at 0.001 (previous default) as an internal class constant; intercept and
cost_fn were unused and are deleted along with the max_tokens machinery
(feasibility repair loop, can_transfer, _padded_tokens, _within_token_limit).
Output is bit-identical to the previous defaults, verified against the
pre-change implementation over 30 randomized inputs.
Signed-off-by: Guokai Ma <guokai.ma@gmail.com>
* Simplify BatchRebalanceSampler for readability
Per review: document and restructure the sampler internals.
- _assign: explain the snake deal (groups ranked by cost, slot k takes ranks
[k*dp_size, (k+1)*dp_size) so every slot has the same cost profile).
- __iter__: deduplicate the full-batch loop and drop_last=False tail block into
a single loop over global batches plus a _yield_global_batch helper.
- _poormans_rebalance -> _balance_groups: drop the three redundant kwargs
(derivable from self), document the hill-climbing search, and extract the
freeze fallback into _slot_groups (pure: which groups share the capped
group's slot) and _relieve_and_freeze (top up the slot from outside groups,
then freeze it).
Output verified bit-identical to the previous implementation over 286
randomized inputs, including adversarial freeze-prone configurations.
Signed-off-by: Guokai Ma <guokai.ma@gmail.com>
* Consolidate batch_rebalance tests and extend the DDP e2e to two epochs
Per review: keep it to a few unit tests and focus on the DDP integration test.
- Merge basic/no_overlap/eff_batch_size/deterministic into one basic test
asserting coverage, no cross-rank overlap, exact per-global-batch counts
and seed determinism (original hand-picked lengths kept).
- Merge the four drop_last tests into one test covering: tail dropped with
drop_last=True, tail yielded as-is otherwise, padding up to K (not
effective_batch_size) when the remainder is smaller, and order wrap-around
when the whole dataset is smaller than K.
- Drop test_batch_rebalance_defeats_double_sharding: it simulated
accelerate's BatchSamplerShard by hand; the real path is covered by the
DDP e2e test.
- Extend the DDP e2e worker to actually train for two epochs and assert the
per-epoch invariants end to end: each epoch covers all samples across
ranks, set_epoch reaches the wrapped sampler (inner_epoch advances) and
later epochs reshuffle. Validated on 2x RTX 5090.
Every assertion from the removed tests maps into the consolidated ones.
Signed-off-by: Guokai Ma <guokai.ma@gmail.com>
* Reuse train.py for the batch_rebalance DDP test
Drop the bespoke batch_rebalance_ddp.py worker script and its white-box
assertions in favour of the shared train.py, so the sampler is exercised
against a real model end to end. train.py gains a --padding option because
the default max_length padding makes every sample the same length, which
leaves the rebalancing with nothing to do.
* [docs] Add batch_rebalance sampling strategy to the training guide
* [docs] Move batch_rebalance sampling guide to trainer_recipes
Move the batch_rebalance sampling strategy docs from training.md to a
dedicated section in trainer_recipes.md, next to Dataloader performance,
following review feedback. The new section explains the two imbalances
it addresses (padding within a micro-batch and idle time across
devices), how the sampler works (sort then shard), the average
per_device_train_batch_size semantics, and how to skip the startup
length scan with precomputed lengths.
---------
Signed-off-by: Guokai Ma <guokai.ma@gmail.com>
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>