transformers
1a55addd - test: add parity check to the CPU-only continuous batching test (#49728)

Commit
30 days ago
test: add parity check to the CPU-only continuous batching test (#49728) * Add CPU parity test for continuous batching The only CPU continuous batching test is @slow and only checks that generation runs. The parity tests against `generate` need an accelerator and are mostly @slow, so the CPU path has no accuracy coverage. Run greedy generation with both `generate_batch` and `generate` on a tiny random Llama in fp32 CPU, and assert the generated tokens match. `eos_token_id=-1` keeps both paths decoding for the full 10 steps, covering prefill and KV-cache decoding. * Check continuous batching parity in the CPU-only test The CPU-only continuous batching test was marked slow and only checked that generation ran, while the parity tests need an accelerator. Drop the slow marker, force fp32 so the comparison is stable on CPU, and assert that generate_batch returns the same greedy tokens as generate. * Update tests/generation/test_continuous_batching.py --------- Co-authored-by: Rémi Ouazan <83456801+remi-or@users.noreply.github.com>
Author
Parents
Loading