One of the fundamental drawbacks of learning by gradient descent techniques is the susceptibility to local minima during training. Recently, some authors have independently introduced new learning algorithms that are based on the properties of terminal attractors and repellers. These algorithms were claimed to perform global optimization of the cost in finite time, provided that a null solution exists. In this paper, we prove that, in the case of local minima free error functions, terminal attractor algorithms guarantee that the optimal solution is reached in a number of steps that is independent of the cost function. Moreover, in the case of multimodal functions, we prove that, unfortunately, there are no theoretical guarantees that a global solution can be reached or that the algorithms perform satisfactorily from an operational point of view, unless particular favourable conditions are satisfied. On the other hand, the ideas behind these innovative methods are very interesting and deserve further investigations.
Bianchini, M., S., F., Gori, M., Maggini, M. (1997). Terminal attractor algorithms: A critical analysis. NEUROCOMPUTING, 15(1), 3-13 [10.1016/S0925-2312(96)00045-8].
Terminal attractor algorithms: A critical analysis
BIANCHINI, MONICA;GORI, MARCO;MAGGINI, MARCO
1997-01-01
Abstract
One of the fundamental drawbacks of learning by gradient descent techniques is the susceptibility to local minima during training. Recently, some authors have independently introduced new learning algorithms that are based on the properties of terminal attractors and repellers. These algorithms were claimed to perform global optimization of the cost in finite time, provided that a null solution exists. In this paper, we prove that, in the case of local minima free error functions, terminal attractor algorithms guarantee that the optimal solution is reached in a number of steps that is independent of the cost function. Moreover, in the case of multimodal functions, we prove that, unfortunately, there are no theoretical guarantees that a global solution can be reached or that the algorithms perform satisfactorily from an operational point of view, unless particular favourable conditions are satisfied. On the other hand, the ideas behind these innovative methods are very interesting and deserve further investigations.File | Dimensione | Formato | |
---|---|---|---|
TA_Neucom.pdf
non disponibili
Tipologia:
Post-print
Licenza:
NON PUBBLICO - Accesso privato/ristretto
Dimensione
730 kB
Formato
Adobe PDF
|
730 kB | Adobe PDF | Visualizza/Apri Richiedi una copia |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/11365/21987
Attenzione
Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo