Streamable chat parsing (#45847)
* scaffolding
* Let's do an initial commit
* Let's do an initial commit
* More cleanup
* More cleanup
* Add the prefill
* One more stash commit
* One more stash commit
* Renames/docstring cleanup
* More renames/docstring cleanup
* More renames/docstring cleanup
* Docstring cleanup
* make fix-repo
* Doc cleanup
* Doc cleanup
* make fix-repo
* No more coerce, merge content types
* Cleanup: Support tokens/arrays/lists correctly for input and prefix
* Add prefix to docs too
* Better initial_events handling, better region handling
* Allow multiple literal matches
* Allow emitting dirty regions
* Doc updates
* Drop `assemble` to just use `transform`
* Lots more doc updates
* Lots more doc updates
* Remove batched handling, lots of cleanup
* Use the new method in test_serve
* Small doc update
* Update docs/source/en/chat_response_parsing.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update docs/source/en/chat_response_parsing.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update docs/source/en/chat_response_parsing.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update src/transformers/models/gemma4/convert_gemma4_weights.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Mention the extra
* Fix the example stream so it doesn't refer to the previous example
* Remove the `strip` arg for numerical/bool types because it was meaningless
* Fix convert script bracket bug
* Add a regex compatibility section
* Update regex compatibility section
* jmespath: defeated
* jmespath: defeated
* Flesh out the Cohere example
* Update the docs with proper content_args
* Make start_anchor mandatory
* Update gemma4 template!
* Proper buffering
* Corrected prefix handling
* Type fixes
* make fix-repo
* Pipeline updates, batching changes
* Post-rebase fixes
* Tiny reorder on the version check
* Fix licences + no variable abbreviations
* Cleanups, remove assertion
* Refactor load_response_template
* Lots of little doc examples
* Fix docs to explain prefix a bit better
* Add batching
* Add batching
* Docs cleanup
* Docs cleanup, big block of vasqu comments
* Apply suggestions from code review
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* More comment fixes
---------
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
