transformers

6d67837f - Add Fill-in-the-middle training objective example - PyTorch (#27464)

Add Fill-in-the-middle training objective example - PyTorch (#27464) * add: initial script to train clm fim * fix: if training model from scratch, new tokens will be added and embeddings resized * fix: fixed attention_mask errors when generating FIM data * fix: file formatted using black * add: run_fim_no_trainer.py and fixed some comments in run_fim.py * add: added fim examples to the README.md and ran code fixup * fix: little bug in both fim training scripts * fix: remove comment from notebook and added a note on fim related params * fix: minor typo in README * add: suggested minor changes to README and run_fim.py * add: gradient_accumulation_steps and gradient_checkpointing args * add: improved model embedding resizing * add: pad_to_multiple_of and attn_implementation params * add: requested minor changes * add: deepspeed zero compatibility * add: resize embeddings layer with zero3 support for fim model initialization