transformers
0e04f91b - Add the AudioProcessor skeleton and its declarative spectrogram config

Commit
1 day ago
Add the AudioProcessor skeleton and its declarative spectrogram config Introduces the modality layer for audio preprocessing, without any numerics. `audio_utils` gains `StftConfig`, `MelScaleConfig` and `SpectrogramConfig`: frozen dataclasses that describe an audio frontend declaratively (framing, windowing, padding, preemphasis, mel filterbank, log scaling) instead of encoding it as a call into `spectrogram()` with a dozen positional arguments. They carry `to_dict`/`from_dict` so they round-trip through a processor config. `hertz_to_mel`, `mel_to_hertz` and `_create_triangular_filter_bank` become array-namespace generic via `_array_namespace`/`_xp_or_math`/`_clamp_min`, so one implementation serves numpy and torch. The slaney branch keeps per-backend constant spellings on purpose: librosa (numpy) and torchaudio (torch) round differently and both are matched bit-exactly. On top of that config: - `audio_processing_base.AudioProcessingMixin` is the audio counterpart of `ImageProcessingMixin`, on top of `PreprocessingMixin`. It carries the audio identity attributes, config round-tripping, and `make_legacy_audio_processor_alias` for building deprecated `XxxFeatureExtractor` aliases (with `utils.deprecation.deprecated_feature_extractor`). - `audio_processing_utils.BaseAudioProcessor` is the modality base class. It owns padding, truncation, masking, audio fetching, feature-length bookkeeping and the `preprocess` pipeline — all driven by a `SpectrogramConfig`, and all independent of how the spectrogram is actually computed. - `AudioKwargs` gains the per-call knobs this layer reads: `spectrogram_config`, `do_extract_spectrogram`, `do_batch_spectrogram`, `do_resample`, `device`. The concrete numeric backends land in the next commit; `BaseAudioProcessor` has no dependency on them, so this layer is reviewable on its own. Purely additive elsewhere: `mel_filter_bank`, `spectrogram`, `spectrogram_batch`, `power_to_db{,_batch}` and `amplitude_to_db{,_batch}` are untouched, no model is migrated, and `SequenceFeatureExtractor` is unchanged. The rewritten mel helpers were verified bit-exact against the previous implementations over 38 probes covering all three mel scales, scalar and array inputs in float32/float64, every `mel_filter_bank` option combination, `chroma_filter_bank`, and an end-to-end `spectrogram`/`power_to_db`/ `amplitude_to_db` run: all values identical. The one intentional difference is that `_create_triangular_filter_bank` now preserves a float32 input dtype instead of upcasting to float64 (it used `np.maximum(np.zeros(1), ...)`); it is private and its only caller passes float64, so no output changes.
Author
Committer
Parents
Loading