llm-foundry

36cee64b - Refactor build optimizer and peft models to use kwargs syntax (#525)

Refactor build optimizer and peft models to use kwargs syntax (#525) Refactor build optimizer and peft models in `train/train.py` to use `**kwargs` syntax so that config's are dictionaries, classes/functions have arguments explicitly stated in their signature, and `build_***` uses **kwargs for initializing classes. Changes: - `build_optimizer` - `build_scheduler` - `build_composer_peft_model` Additionally, moved `build_dataloader` right before `Trainer` is initialized to make it faster to catch non-data related issues in the yaml. ## Unit Tests - Added unit tests for optimizer states and build_scheduler. ## Integration Tests - Trained PEFT model, got same fine tuning scores: <img width="1635" alt="Screenshot 2023-08-16 at 5 39 17 PM" src="https://github.com/mosaicml/llm-foundry/assets/13524881/57226d52-a35e-4229-8e01-4ff3e7d51d29"> - [Before change LORA run](https://wandb.ai/mosaic-ml/chuck-runs/runs/ckod4o5r/logs?workspace=user-chuck-tang98) - [After change LORA run](https://wandb.ai/mosaic-ml/chuck-runs/runs/69aiy9uk?workspace=user-chuck-tang98) ## Issues Closed Chipping away at tech [debt](https://id.atlassian.com/login/authorize?continue=https%3A%2F%2Fmosaicml.atlassian.net%2Fbrowse%2FRESEARCH-717&token=eyJraWQiOiJtaWNyb3Mvc2lnbi1pbi1zZXJ2aWNlL2FlN21rM2I2cGxzOWhtaDkiLCJhbGciOiJSUzI1NiJ9.eyJtYXJrZWRWZXJpZmllZCI6ImZhbHNlIiwibG9naW5UeXBlIjoic2Vzc2lvblJlZnJlc2giLCJpc3MiOiJtaWNyb3Mvc2lnbi1pbi1zZXJ2aWNlIiwidXNlcklkIjoiNzEyMDIwOjVkY2IyNmIzLWYzNjctNDc2Yy1hYTJlLTNjNzg3NDhiZjgyNiIsImlzU2xhY2tBcHBTb3VyY2UiOiJmYWxzZSIsImF1ZCI6Imxpbmstc2lnbmF0dXJlLXZhbGlkYXRvciIsIm5iZiI6MTY5MjEzNTk3MCwic2NvcGUiOiJMb2dpbiIsImV4cCI6MTY5MjEzNjA5MCwiaWF0IjoxNjkyMTM1OTcwLCJqdGkiOiJmNTA2MjY1My05YTZmLTRjNTQtOWViMC0yYjhiOWQ4MjQ2ODMiLCJoYXNoZWRDc3JmVG9rZW4iOiIxMDYzYTZkYjY2NTVmZDMyZGYwMjQ5N2FmZjllZWFmYmFhOTU5NzA4ZDA4ZjdmMzg4MDk4YjBjMGJlNWU3Yjg2In0.abK8iEzpDsAT9ca3zLtWLCgkEiYyz0Byng6_Zl68jhrMkR0_w-EXp8MEkWVrWXQWEdpNVRjyb9rxq32VN-STEdR3Y1wXYRI6-7z-V7Uwpk0ZxRNproba8kthUFtu22atf-II_P_zjvE4abkVaeENXCq3i9v5jX1MDgYlF_9t72BnQ_-rFpOYDGhme8NXu5w2Z2G_M3AhTzGSSxi52_0JcCAHwMN39RWQKQR3gsHPFDPC6MGdjnN_dnNB18iYNLzp2BGGBBobCPgruuvMsu9r2wDOC0W34pdFnBkXvTnoCtVp-3jIKPvVDphkuSYl28Y6-rvck86bq_Skwz76zg6DQg&state=eyJoYXNoZWRDc3JmVG9rZW4iOiJhNDAxMTdhYWZmMjkzOWNjNWI5YTE0ZDRjOTMyN2ZlYjAxNGUyNjg4ZWNhMTkzYzhhZDcyNzA5YzJlMWIzNTExIn0%3D)