GLR: Gradient-Based Learning Rate Scheduler

Training a neural network is a complex and time-consuming process because of many combinations of hyperparameters that have to be adjusted and tested. One of the most crucial hyperparameters is the learning rate which controls the speed and direction of updates to the weights during training. We proposed an adaptive scheduler called Gradient-based Learning Rate scheduler (GLR) that significantly reduces the tuning effort thanks to a single user-defined parameter. GLR achieves competitive results in a very wide set of experiments compared to the state-of-the-art schedulers and optimizers. The computational cost of our method is trivial and can be used to train different network topologies.