[R] Unraveling the Mysteries: Why is AdamW Often Superior to Adam+L2 in Practice? (lemmy.world)

submitted 1 year ago by LoveOxygenProducers@lemmy.world to c/machinelearning@lemmy.world

3 comments fedilink hide all child comments

Hello, ML enthusiasts! 🚀🤖 We analyzed rotational equilibria in our latest work, ROTATIONAL EQUILIBRIUM: HOW WEIGHT DECAY BALANCES LEARNING ACROSS NEURAL NETWORKS

💡 Our Findings: Balanced average rotational updates (effective learning rate) across all network components may play a key role in the effectiveness of AdamW.

🔗 ROTATIONAL EQUILIBRIUM: HOW WEIGHT DECAY BALANCES LEARNING ACROSS NEURAL NETWORKS

Looking forward to hearing your thoughts! Let’s discuss more about this fascinating topic together!

you are viewing a single comment's thread
view the rest of the comments

[-] A_A@lemmy.world 2 points 1 year ago

this post was submitted on 08 Oct 2023

18 points (95.0% liked)

Machine Learning | Artificial Intelligence

948 readers

1 users here now

Welcome to Machine Learning – a versatile digital hub where Artificial Intelligence enthusiasts unite. From news flashes and coding tutorials to ML-themed humor, our community covers the gamut of machine learning topics. Regardless of whether you're an AI expert, a budding programmer, or simply curious about the field, this is your space to share, learn, and connect over all things machine learning. Let's weave algorithms and spark innovation together.

founded 1 year ago

MODERATORS

Hopps@lemmy.world