In the context of deep learning models, attention has recently been paid to studying the surface of the loss function in order to better understand training with methods based on gradient descent. This search for an appropriate description, both analytical and topological, has led to numerous efforts in identifying spurious minima and characterize gradient dynamics. Our work aims to contribute to this field by providing a topological measure for evaluating loss complexity in the case of multilayer neural networks. We compare deep and shallow architectures with common sigmoidal activation functions by deriving upper and lower bounds for the complexity of their respective loss functions and revealing how that complexity is influenced by the number of hidden units, training models, and the activation function used. Additionally, we found that certain variations in the loss function or model architecture, such as adding an l_2 regularization term or implementing skip connections in a feedforward network, do not affect loss topology in specific cases.

Bucarelli, M.S., D’Inverno, G.A., Bianchini, M., Scarselli, F., Silvestri, F. (2024). A topological description of loss surfaces based on Betti Numbers. NEURAL NETWORKS, 178 [10.1016/j.neunet.2024.106465].

A topological description of loss surfaces based on Betti Numbers

D’Inverno, Giuseppe Alessio;Bianchini, Monica;Scarselli, Franco;
2024-01-01

Abstract

In the context of deep learning models, attention has recently been paid to studying the surface of the loss function in order to better understand training with methods based on gradient descent. This search for an appropriate description, both analytical and topological, has led to numerous efforts in identifying spurious minima and characterize gradient dynamics. Our work aims to contribute to this field by providing a topological measure for evaluating loss complexity in the case of multilayer neural networks. We compare deep and shallow architectures with common sigmoidal activation functions by deriving upper and lower bounds for the complexity of their respective loss functions and revealing how that complexity is influenced by the number of hidden units, training models, and the activation function used. Additionally, we found that certain variations in the loss function or model architecture, such as adding an l_2 regularization term or implementing skip connections in a feedforward network, do not affect loss topology in specific cases.
2024
Bucarelli, M.S., D’Inverno, G.A., Bianchini, M., Scarselli, F., Silvestri, F. (2024). A topological description of loss surfaces based on Betti Numbers. NEURAL NETWORKS, 178 [10.1016/j.neunet.2024.106465].
File in questo prodotto:
File Dimensione Formato  
1-s2.0-S0893608024003897-main.pdf

accesso aperto

Tipologia: PDF editoriale
Licenza: Creative commons
Dimensione 622.56 kB
Formato Adobe PDF
622.56 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11365/1263175