transformers

39ea714c - Reduce preprocessing_num_workers 2→1 to eliminate CoW memory spike

Reduce preprocessing_num_workers 2→1 to eliminate CoW memory spike With num_proc=2, DatasetDict.map/filter forks 2 workers that each inherit the full Python/torch process pages (~500MB). As workers process audio data, CoW triggers on many pages, causing a transient 5.8GB PSS spike for wav2vec2_pretraining despite the model being only 0.76MB and the dataset arrow table being 9.2MB. With num_proc=1, the datasets library uses the single-process path (no fork), eliminating the CoW overhead entirely. For tiny test datasets (73 examples, max_train_steps=2) there is no benefit to parallel preprocessing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>