The generalized delta rule has proven to be an important and popular procedure for training ANNs. However, it can be characterized as suffering from two major problems. First, it generally is a very slow learning rule. Second, for difficult problems it often produces a local minimum: a configuration of connection weights which cannot be changed by the learning rule, and which result in the network responding incorrectly to some of the input patterns (e.g., Dawson & Schopflocher, 1992a, Table II).

Recently, we have begun to explore how to take advantage of the local minimum problem in order to speed up learning (Medler & Dawson, 1994). Our approach is to exploit another property of biological networks: redundancy. Instead of training a single network to solve a problem, we train a number of similar networks to solve the same problem, and then pool their responses together. The logic of this approach is that if one uses enough variation in initializing the subnetworks, then the subnetworks will be trained into different local minima (i.e., they will be making mistakes on different input patterns). As a result, the majority of the subnetworks will be responding correctly to an input pattern, and can overrule the incorrectly responding minority of subnetworks when responses are pooled.

Medler & Dawson Used Redundant Nets To Train Churchland's Crab

Our initial research on a redundant architecture that uses five subnetworks has indicated that redundant networks results in a learning speed-up that justifies the computational expense of training larger networks (Medler & Dawson, 1994). We have have examined a function approximation problem (training a simulated two-joint robot arm to "reach" to a target) and a pattern classification problem (training a system to compute whether an input pattern has an even or odd number of "bits" turned on). In both cases, the redundant network learns to solve the problem significantly faster than a single network control, and is more accurate than the control network even when it is given five times the amount of training (to equate the amount of training that has gone into the redundant system). Redundant networks lead to improved performance whether the processors in the units are integration devices or are value units (see results below!).

References