Unpin with latest transformers fixes (#6763)
Reverts #6759
Requires from transformers:
https://github.com/huggingface/transformers/pull/34816
https://github.com/huggingface/transformers/pull/34800
Todo:
- [x] Need to merge first PR to get support for torch 2.4
