llm-foundry

64212cba - Add deeper support for multi-turn chats and loss-generating tokens in finetuning (#985)

Add deeper support for multi-turn chats and loss-generating tokens in finetuning (#985) The main purpose of this PR is to support training on non-terminal responses in multi-round chats. This is achieved by tokenizing at the level of conversation "turns" and exposing some options for what turns are used as training targets (i.e. generate loss). This also adds support for treating prompt tokens as loss-generating. The script for converting a finetuning dataset to streaming has also been updated (with some bug fixes).