A Review of 10 Most Popular Activation Functions in Neural Networks

In this video, I'll be discussing 10 different activation functions used in machine learning, providing visualizations of their graphs and explaining the behavior of their derivatives. The list of activation functions covered includes: 1. Linear 2. ReLU 3. Leaky ReLU 4. Sigmoid (also known as the logistic sigmoid function) 5. Tanh (also known as the hyperbolic tangent function) 6. Softplus 7. ELU 8. SELU 9. Swish 10. GELU By the end of the video, you'll learn which activation functions have continuous derivatives and which ones have discontinuous derivatives, as well as which ones are monotonic and which ones are non-monotonic functions. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Errata: The output of sigmoid function is between [0, 1].

Join Today
Top Optimizers for Neural Networks
▶︎

Top Optimizers for Neural Networks

Why Do We Need Activation Functions in Neural Networks?
▶︎

Why Do We Need Activation Functions in Neural Networks?

relu
▶︎

relu

The 60-Year Hunt for AI's Most Important Function
▶︎

The 60-Year Hunt for AI's Most Important Function

Inside a Neural Network - Computerphile
▶︎

Inside a Neural Network - Computerphile

Deep Learning(CS7015): Lec 9.3 Better activation functions
▶︎

Deep Learning(CS7015): Lec 9.3 Better activation functions

The Trig Hiding Inside the Factorials (And Harmonic Numbers)
▶︎

The Trig Hiding Inside the Factorials (And Harmonic Numbers)

Back propagation by hand | the math you should know
▶︎

Back propagation by hand | the math you should know

Watching Neural Networks Learn
▶︎

Watching Neural Networks Learn

Introduction to Neural Networks
▶︎

Introduction to Neural Networks

All Machine Learning Models Clearly Explained!
▶︎

All Machine Learning Models Clearly Explained!

But what is a neural network? | Deep learning chapter 1
▶︎

But what is a neural network? | Deep learning chapter 1

Hopfield network: How are memories stored in neural networks? [Nobel Prize in Physics 2024] #SoME2
▶︎

Hopfield network: How are memories stored in neural networks? [Nobel Prize in Physics 2024] #SoME2

Self-Attention Using Scaled Dot-Product Approach
▶︎

Self-Attention Using Scaled Dot-Product Approach

How To Think SO CLEARLY People Assume You're A Genius
▶︎

How To Think SO CLEARLY People Assume You're A Genius

Lecture 10 - Introduction to Neural Networks | Stanford CS229: Machine Learning (Autumn 2018)
▶︎

Lecture 10 - Introduction to Neural Networks | Stanford CS229: Machine Learning (Autumn 2018)

Physics Informed Neural Networks explained for beginners | From scratch implementation and code
▶︎

Physics Informed Neural Networks explained for beginners | From scratch implementation and code

Activation Functions In Neural Networks Explained | Deep Learning Tutorial
▶︎

Activation Functions In Neural Networks Explained | Deep Learning Tutorial

Weights & Biases MADE EASY [2/11]
▶︎

Weights & Biases MADE EASY [2/11]

How convolutional neural networks work, in depth
▶︎

How convolutional neural networks work, in depth