Training a neural network is a complex and time-consuming process because of many combinations of hyperparameters that have to be adjusted and tested. One of the most crucial hyperparameters is the learning rate which controls the speed and direction of updates to the weights during training. We proposed an adaptive scheduler called Gradient-based Learning Rate scheduler (GLR) that significantly reduces the tuning effort thanks to a single user-defined parameter. GLR achieves competitive results in a very wide set of experiments compared to the state-of-the-art schedulers and optimizers. The computational cost of our method is trivial and can be used to train different network topologies.

GLR: Gradient-Based Learning Rate Scheduler

Napoli Spatafora M. A.;Ortis A.;Battiato S.
2023-01-01

Abstract

Training a neural network is a complex and time-consuming process because of many combinations of hyperparameters that have to be adjusted and tested. One of the most crucial hyperparameters is the learning rate which controls the speed and direction of updates to the weights during training. We proposed an adaptive scheduler called Gradient-based Learning Rate scheduler (GLR) that significantly reduces the tuning effort thanks to a single user-defined parameter. GLR achieves competitive results in a very wide set of experiments compared to the state-of-the-art schedulers and optimizers. The computational cost of our method is trivial and can be used to train different network topologies.
2023
978-3-031-43147-0
978-3-031-43148-7
Hyperparameters
Neural network
Optimization
File in questo prodotto:
File Dimensione Formato  
GLR Gradient-Based Learning Rate Scheduler_compressed-1-7.pdf

solo gestori archivio

Tipologia: Versione Editoriale (PDF)
Licenza: NON PUBBLICO - Accesso privato/ristretto
Dimensione 6.46 MB
Formato Adobe PDF
6.46 MB Adobe PDF   Visualizza/Apri
GLR Gradient-Based Learning Rate Scheduler_compressed-8-13.pdf

solo gestori archivio

Tipologia: Versione Editoriale (PDF)
Licenza: NON PUBBLICO - Accesso privato/ristretto
Dimensione 5.01 MB
Formato Adobe PDF
5.01 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/20.500.11769/575492
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? 0
social impact