Merge highlighted text when neighbors (#2767)
* Merge highlighted text when neighbors
In the previous implementation dictionary inputs were not correctly
processed meaning: a None token was added on the empty string between
each entity. That None entry is removed letting neighbouring entities
be merged.
* Add changelog
* Add test
* Improve API compatibility and fix test
* Fix lint
* Strip out the empty entries on merge
* Test against the right reference
* fix changelog
* Apply suggestions from code review
Co-authored-by: Abubakar Abid <abubakar@huggingface.co>
* Remove additonal merge adjacent
* Fix lint
Co-authored-by: Abubakar Abid <abubakar@huggingface.co>
