Supervised and Unsupervised Co-Training of Adaptive Activation Functions in Neural Nets

In spite of the nice theoretical properties of mixtures of logistic activation functions, standard feedforward neural network with limited resources and gradient-descent optimization of the connection weights may practically fail in several, difficult learning tasks. Such tasks would be better faced by relying on a more appropriate, problem-specific basis of activation functions. The paper introduces a connectionist model which features adaptive activation functions. Each hidden unit in the network is associated with a specific pair (f(•), p(•)), where f(•) (the very activation) is modeled via a specialized neural network, and p(•) is a probabilistic measure of the likelihood of the unit itself being relevant to the computation of the output over the current input. While f(•) is optimized in a supervised manner (through a novel backpropagation scheme of the target outputs which do not suffer from the traditional phenomenon of "vanishing gradient" that occurs in standard backpropagation), p(•) is realized via a statistical parametric model learned through unsupervised estimation. The overall machine is implicitly a co-trained coupled model, where the topology chosen for learning each f(•) may vary on a unit-by-unit basis, resulting in a highly non-standard neural architecture. © 2012 Springer-Verlag.