transformers

ce5c8f5e - [GPTNeoX] Fix post_processor not overridden when loading from pretrained (OLMo garbage generation) (#47988)

[GPTNeoX] Fix post_processor not overridden when loading from pretrained (OLMo garbage generation) (#47988) [GPTNeoX] Fix post_processor not being overridden when loading from pretrained `_from_pretrained` strips `add_bos_token`/`add_eos_token` from init_kwargs when a tokenizer.json is present, so `TokenizersBackend.__init__` sees no explicit bos/eos kwargs and skips calling `update_post_processor()`. For GPTNeoX (which rebuilds its Rust tokenizer from scratch rather than loading it directly from tokenizer.json), the post_processor from the JSON file is applied as-is and is never overridden — causing OLMo-7B to receive an unexpected EOS token appended to every input, producing garbage generation. Restore the explicit `update_post_processor()` call at the end of `GPTNeoXTokenizer.__init__` so the Rust post_processor always reflects the `add_bos_token`/`add_eos_token` settings. Regression introduced by #42563 (commit 73a13f86f6). Fixes OlmoIntegrationTest::test_model_7b_greedy_generation producing `.1.1.1.1.1...` garbage instead of coherent text. Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>