Conditional Activation for Diverse Neurons in Heterogeneous Networks

Albert Lee, Bonnie Lam, Wenyuan Li, Hochul Lee, Wei-Hao Chen, Meng-Fan Chang, Kang. -L. Wang

In this paper, we propose a new scheme for modelling the diverse behavior of neurons. We introduce the conditional activation, in which a neurons activation function is dynamically modified by a control signal. We apply this method to recreate behavior of special neurons existing in the human auditory and visual system. A heterogeneous multilayered perceptron (MLP) incorporating the developed models demonstrates simultaneous improvement in learning speed and performance across a various number of hidden units and layers, compared to a homogeneous network composed of the conventional neuron model. For similar performance, the proposed model lowers the memory for storing network parameters significantly.

Knowledge Graph



Sign up or login to leave a comment