Bump pytorch-optimizer from 3.5.1 to 3.8.0

[dependabot]

Bumps pytorch-optimizer from 3.5.1 to 3.8.0.

Release notes

Sourced from pytorch-optimizer's releases.

pytorch-optimizer v3.8.0

Change Log

Feature

• Implement EmoNeco and EmoZeal optimizers. (#407)
• Implement Refined Schedule-Free AdamW optimizer. (#409, #414)
  • Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training
  • You can use this variant by setting decoupling_c parameter in the ScheduleFreeAdamW optimizer.
• Add more built-in optimizers, NAdam, RMSProp, and LBFGS optimizers. (#415)
• Support cautious variant for Muon optimizer. (#417)
• Separate distributed functionality from Muon to DistribtuedMuon optimizer. (#418)
• Implement StochasticAccumulator, which is a gradient hook. (#418)
  • stochastic optimizer

Update

• Re-implement Muon and AdaMuon optimizers based on the recent official implementation. (#408, #410)
  • Their definitions have changed from the previous version, so please check out the documentation!
• Update the missing optimizers from __init__.py. (#415)
• Add the HuggingFace Trainer example. (#415)
• Optimize the visualization outputs and change the visualization document to a table layout. (#416)

Dependency

• Update mkdocs dependencies. (#417)

CI

• Add some GitHub actions to automate some processes. (#411, #412, #413)

Contributions

thanks to @​AidinHamedi

pytorch-optimizer v3.7.0

Change Log

Feature

• Implement AdaMuon optimizer. (#394, #395)
  • Adaptive Muon Optimizer
• Implement SPlus optimizer. (#396, #399)
  • A Stable Whitening Optimizer for Efficient Neural Network Training
• Implement EmoNavi, EmoFact, and EmoLynx optimizers. (#393, #400)
  • An emotion-driven optimizer that feels loss and navigates accordingly

CI

• Enable CI for Python 3.8 ~ 3.13. (#402, #404)

... (truncated)

Changelog

Sourced from pytorch-optimizer's changelog.

v3.7.0

Change Log

Feature

• Implement AdaMuon optimizer. (#394, #395)
  • Adaptive Muon Optimizer
• Implement SPlus optimizer. (#396, #399)
  • A Stable Whitening Optimizer for Efficient Neural Network Training
• Implement EmoNavi, EmoFact, and EmoLynx optimizers. (#393, #400)
  • An emotion-driven optimizer that feels loss and navigates accordingly

CI

• Enable CI for Python 3.8 ~ 3.13. (#402, #404)

Fix

• Adjust the value of eps to the fixed value 1e-15 when adding to exp_avg_sq. (#397, #398)
• built-in type-hint in Kron optimizer. (#404)

v3.6.1

Change Log

Feature

• Implement more cooldown types for WSD learning rate scheduler. (#382, #386)
• Implement AdamWSN optimizer. (#387, #389)
  • Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees
• Implement AdamC optimizer. (#388, #390)
  • Why Gradients Rapidly Increase Near the End of Training

Update

• Change the default range of the beta parameter from [0, 1] to [0, 1). (#392)

Fix

• Fix to use momentum buffer instead of the gradient to calculate LMO. (#385)

v3.6.0

Change Log

Feature

• Implement Fira optimizer. (#376)
  • Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?

... (truncated)

Commits

• 483a816 [Release] v3.8.0 (#419)
• 75c7945 [Feature] Implement DistributedMuon optimizer (#418)
• 6071805 [Feature] Support cautious variant for Muon optimizer (#417)
• 6704a22 [Update] Optimizer visualization (#416)
• 0b8c7a5 [Update] lots of stuff (#415)
• 04b3758 [Feature] Implement Refined Schedule-Free AdamW optimizer (#414)
• 4418b8c [CI] Add pr_labeler github action (#413)
• 3fd7147 [CI] Copy README.md to /docs/index.md automatically (#412)
• cff7179 [Feature] Prepend the changelog automatically (#411)
• 49f75d0 [Update] Re-implement Muon and AdaMuon optimizers (#410)
• Additional commits viewable in compare view

You can trigger a rebase of this PR by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

Note
Automatic rebases have been disabled on this pull request as it has been open for over 30 days.

[dependabot]

Superseded by #202.