Summary
Activation functions are crucial in neural networks, introducing non-linearity and enabling the modeling of complex patterns across varied tasks. This guide delves into the evolution, characteristics, and applications of state-of-the-art activation functions, illustrating their role in enhancing neural network performance. It discusses the transition from classic functions like sigmoid and tanh to advanced ones such as ReLU and its variants, addressing challenges like the vanishing gradient problem and the dying ReLU issue. Concluding with practical heuristics for selecting activation functions, the article emphasizes the importance of considering network architecture and task specifics, highlighting the rich diversity of activation functions available for optimizing neural network designs.
Very frustrating to see this, as large models have shown that scalar activation functions make only a tiny impact when your model is wide enough.
https://arxiv.org/abs/2002.05202v1 shows GLU-based activation functions (2 inputs->1 output) almost universally beat their equivalent scalar functions. IMO there needs to be more work around these kinds of multi-input constructions, as there are much bigger potential gains.
E.g. even for cases where the network only needs static routing (tabular data), transformers sometimes perform magically better than MLPs. This suggests there's something special about self-attention as an "activation function". If that magic can be extracted and made sub-quadratic, it could be a paradigm shift in NN design.
The authors of the blog post seem aware of the limitations of their focus:
Thank you for highlighting this research! At first glance it's interesting that sigmoid functions re-emerge as more useful using the approaches evaluated in that article.