Bump pytorch-optimizer from 3.5.1 to 3.6.1

[dependabot]

Bumps pytorch-optimizer from 3.5.1 to 3.6.1.

Release notes

Sourced from pytorch-optimizer's releases.

pytorch-optimizer v3.6.1

Change Log

Feature

• Implement more cooldown types for WSD learning rate scheduler. (#382, #386)
• Implement AdamWSN optimizer. (#387, #389)
  • Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees
• Implement AdamC optimizer. (#388, #390)
  • Why Gradients Rapidly Increase Near the End of Training

Update

• Change the default range of the beta parameter from [0, 1] to [0, 1). (#392)

Fix

• Fix to use momentum buffer instead of the gradient to calculate LMO. (#385)

pytorch-optimizer v3.6.0

Change Log

Feature

• Implement Fira optimizer. (#376)
  • Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?
• Implement RACS and Alice optimizers. (#376)
  • Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension
• Implement VSGD optimizer. (#377, #378)
  • Variational Stochastic Gradient Descent for Deep Neural Networks
• Enable training with complex parameters. (#370, #380)
  • will raise NoComplexParameterError for unsupported optimizers, due to its design or not-yet-implemented.
• Support maximize parameter. (#370, #380)
  • maximize: maximize the objective with respect to the params, instead of minimizing.
• Implement copy_stochastic() method. (#381)

Update

• Support 2D< Tensor for RACS and Alice optimizers. (#380)
• Remove the auxiliary variants from the default parameters of the optimizers and change the name of the state and parameter. (#380)
  • use_gc, adanorm, cautious, stable_adamw, and adam_debias will be affected.
  • You can still use these variants by passing the parameters to **kwargs.
  • Notably, in case of adanorm variant, you need to pass adanorm (and adanorm_r for r option) parameter(s) to use this variant, and the name of the state will be changed from exp_avg_norm to exp_avg_adanorm.
• Refactor reset() to init_group() method in the BaseOptimizer class. (#380)
• Refactor SAM optimizer family. (#380)
• Gather AdamP, SGDP things into pytorch_optimizer.optimizer.adamp.*. (#381)
  • pytorch_optimizer.optimizer.sgdp.SGDP to pytorch_optimizer.optimizer.adamp.SGDP
  • pytorch_optimizer.optimizer.util.projection to pytorch_optimizer.optimizer.adamp.projection
  • pytorch_optimizer.optimizer.util.cosine_similarity_by_view to pytorch_optimizer.optimizer.adamp.cosine_similarity_by_view
• Remove channel_view() and layer_view() from pytorch_optimizer.optimizer.util. (#381)

... (truncated)

Commits

• 77098e9 [Release] v3.6.1 (#392)
• 2a4423d [Feature] Implement AdamC optimizer (#390)
• 030d303 [Feature] Implement AdamWSN optimizer (#389)
• 02fc0af [Feature] Implment more cooldown types for WSD lr scheduler (#386)
• 5aa4d13 Merge pull request #385 from kozistr/fix/scion-optimizer
• 64a7e7d docs: v3.6.1 changelog
• 2a74a72 update: recipe
• ab73a77 update: recipe
• da1dec2 update: recipe
• 6b3dd0e fix: momentum buffer
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Superseded by #181.