Repository containing the article with examples of custom activation functions for Pytorch and scripts used in the article. See the article on Medium and a kernel on Kaggle.
See also the article about the in-place activations in PyTorch.
Today deep learning is going viral and is applied to a variety of machine learning problems such as image recognition, speech recognition, machine translation, and others. There is a wide range of highly customizable neural network architectures, which can suit almost any problem when given enough data. Each neural network should be elaborated to suit the given problem well enough. You have to fine tune the hyperparameters of the network (the learning rate, dropout coefficients, weight decay, and many others) as well as the number of hidden layers, and the number of units in each layer. Choosing the right activation function for each layer is also crucial and may have a significant impact on metric scores and the training speed of the model.
- My original article on Medium
- My related kernel on Kaggle
- Activation functions wiki
- Tutorial on PyTorch extending.
- My article on Medium explaining in-place operations in PyTorch.