transformers
Add Fill-in-the-middle training objective example - PyTorch
#27464
Merged
Merged
Add Fill-in-the-middle training objective example - PyTorch #27464
asartran
merged
16 commits into
huggingface:main
from
tanaymeh:add_fim_examples
add: initial script to train clm fim
f92c1db7
fix: if training model from scratch, new tokens will be added and emb…
43e180b5
fix: fixed attention_mask errors when generating FIM data
b4ec1707
fix: file formatted using black
06b8ab88
add: run_fim_no_trainer.py and fixed some comments in run_fim.py
b849c769
add: added fim examples to the README.md and ran code fixup
0b23db2e
fix: little bug in both fim training scripts
03c34fb3
fix: remove comment from notebook and added a note on fim related params
533e43a0
fix: minor typo in README
8cc79273
tanaymeh
changed the title
add: suggested minor changes to README and run_fim.py
7951865e
add: gradient_accumulation_steps and gradient_checkpointing args
bd7a18d9
add: improved model embedding resizing
b34e25ea
add: pad_to_multiple_of and attn_implementation params
98a5976c
ArthurZucker
approved these changes
on 2024-01-31
add: requested minor changes
1eab3c6d
add: deepspeed zero compatibility
3cdc2388
pacman100
approved these changes
on 2024-03-08
add: resize embeddings layer with zero3 support for fim model initial…
c58517cf
Login to write a write a comment.
Login via GitHub