Megatron-DeepSpeed

846c0879 - Adding language specific validation sets for Multilingual model training (#97)

Adding language specific validation sets for Multilingual model training (#97) * adding multiple datasets to args * Extra_valid_dataset arg & display on Tensorboard * code cleaning * code cleaning * add run script example for multilingual validation * rename extra-validation to periodic-eval * bugfix + logging amount of training data per language * change "periodic-eval" to "extra-eval" * elaborate tree sturcture in Multiling run script * initialize all_x_datasets to None if empty * clean kwargs.get * add mode two of data loading * adding option2 for data loading * fix missing argument and range of split * add multilingual run script * small cleanups + fixed bugs with test iterator * Fixed prefixlm * adding variable cutoff for last epoch rather than 80% Co-authored-by: TevenLeScao <teven.lescao@gmail.com>