transformers
0e04f91b - Add the AudioProcessor skeleton and its declarative spectrogram config

Commit
32 days ago
Add the AudioProcessor skeleton and its declarative spectrogram config Introduces the modality layer for audio preprocessing, without any numerics. `audio_utils` gains `StftConfig`, `MelScaleConfig` and `SpectrogramConfig`: frozen dataclasses that describe an audio frontend declaratively (framing, windowing, padding, preemphasis, mel filterbank, log scaling) instead of encoding it as a call into `spectrogram()` with a dozen positional arguments. They carry `to_dict`/`from_dict` so they round-trip through a processor config. `hertz_to_mel`, `mel_to_hertz` and `_create_triangular_filter_bank` become array-namespace generic via `_array_namespace`/`_xp_or_math`/`_clamp_min`, so one implementation serves numpy and torch. The slaney branch keeps per-backend constant spellings on purpose: librosa (numpy) and torchaudio (torch) round differently and both are matched bit-exactly. On top of that config: - `audio_processing_base.AudioProcessingMixin` is the audio counterpart of `ImageProcessingMixin`, on top of `PreprocessingMixin`. It carries the audio identity attributes, config round-tripping, and `make_legacy_audio_processor_alias` for building deprecated `XxxFeatureExtractor` aliases (with `utils.deprecation.deprecated_feature_extractor`). - `audio_processing_utils.BaseAudioProcessor` is the modality base class. It owns padding, truncation, masking, audio fetching, feature-length bookkeeping and the `preprocess` pipeline — all driven by a `SpectrogramConfig`, and all independent of how the spectrogram is actually computed. - `AudioKwargs` gains the per-call knobs this layer reads: `spectrogram_config`, `do_extract_spectrogram`, `do_batch_spectrogram`, `do_resample`, `device`. The concrete numeric backends land in the next commit; `BaseAudioProcessor` has no dependency on them, so this layer is reviewable on its own. Purely additive elsewhere: `mel_filter_bank`, `spectrogram`, `spectrogram_batch`, `power_to_db{,_batch}` and `amplitude_to_db{,_batch}` are untouched, no model is migrated, and `SequenceFeatureExtractor` is unchanged. The rewritten mel helpers were verified bit-exact against the previous implementations over 38 probes covering all three mel scales, scalar and array inputs in float32/float64, every `mel_filter_bank` option combination, `chroma_filter_bank`, and an end-to-end `spectrogram`/`power_to_db`/ `amplitude_to_db` run: all values identical. The one intentional difference is that `_create_triangular_filter_bank` now preserves a float32 input dtype instead of upcasting to float64 (it used `np.maximum(np.zeros(1), ...)`); it is private and its only caller passes float64, so no output changes.
Author
Committer
Parents
Loading