pytorch-lightning

[Not finished] Allow customized parameter grouping for automatic optimzier configuration.

#20742

Open

Make a minor change during automatic optimizer configuration

• replace self.model.parameters() with an additional method
• The behavior is stay unchanged, but this change may allow users customizing their own CLI, with different parameters given to to optimizer.

E.g.
If one want to add weight decay to weight groups only (L2 regularization), he/she can usually use named_parameters to iterate over the model parameter to determine if one parameter should be put into the weighted group, or remain unregulated. However, this seems impossible when using CLI for automatic configuration. Although one can still write the configure_optimizers him/herself, but I think making this minor change would give users a faster path to do such things without creating a bunch of codes.

What does this PR do?

As shown in the commit message, this PR changes the src/lightning/pytorch/cli.py, adding a new separate private object method, LightningCLI._get_model_parameters(). The return of this method is exactly self.model.parameters(). And during the automatic optimizer configuration, instead of getting model parameters directly, CLI object will use this method to obtain the model parameters.

The default behavior is just as the original: get model parameters directly.

However, if one would like to create an optimizer receiving named_parameters (e.g. giving learning rate or L2 factor to different parameter groups), it could customize its own CLI class, just overriding this method, and write his own optimizer. At the same time, it does not need to write a bunch of codes in configure_optimizers of the LightningModule, nor caring about parameter passing in.

And for users who do not rely on this functionality, the default behavior remains unchanged.

• Was this discussed/agreed via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

📚 Documentation preview 📚: https://pytorch-lightning--20742.org.readthedocs.build/en/20742/

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you need further help see our docs: https://lightning.ai/docs/pytorch/latest/generated/CONTRIBUTING.html#pull-request or ask the assistance of a core contributor here or on Discord. Thank you for your contributions.