[R] Unraveling the Mysteries: Why is AdamW Often Superior to Adam+L2 in Practice? (lemmy.world)

submitted 1 year ago by LoveOxygenProducers@lemmy.world to c/machinelearning@lemmy.world

3 comments fedilink hide all child comments

Hello, ML enthusiasts! 🚀🤖 We analyzed rotational equilibria in our latest work, ROTATIONAL EQUILIBRIUM: HOW WEIGHT DECAY BALANCES LEARNING ACROSS NEURAL NETWORKS

💡 Our Findings: Balanced average rotational updates (effective learning rate) across all network components may play a key role in the effectiveness of AdamW.

🔗 ROTATIONAL EQUILIBRIUM: HOW WEIGHT DECAY BALANCES LEARNING ACROSS NEURAL NETWORKS

Looking forward to hearing your thoughts! Let’s discuss more about this fascinating topic together!

you are viewing a single comment's thread
view the rest of the comments

[-] wagesj45@kbin.social 1 points 1 year ago

The human brain isn't a blank slate when it comes into existence. There are already structures that are designed to do certain things. These structures come "pre trained" and a lot of the learning humans do is more akin to the fine tuning that we do for foundation models.

this post was submitted on 08 Oct 2023

18 points (95.0% liked)

Machine Learning | Artificial Intelligence

948 readers

1 users here now

Welcome to Machine Learning – a versatile digital hub where Artificial Intelligence enthusiasts unite. From news flashes and coding tutorials to ML-themed humor, our community covers the gamut of machine learning topics. Regardless of whether you're an AI expert, a budding programmer, or simply curious about the field, this is your space to share, learn, and connect over all things machine learning. Let's weave algorithms and spark innovation together.

founded 1 year ago

MODERATORS

Hopps@lemmy.world