except that the logistic activation function is continuous. in other words, it never jumps suddenly. Therefore, its derivative can be computed, allowing us to perform gradient descent. The second difference between the generalized delta rule and the simple delta rule is that the delta value is multiplied by this logistic activation function. So, to put it simply, the delta for a hidden unit is the logistic activation function for that unit, times the sum of the next weights and deltas.

If all this seems terribly complicated, let me assure you that computer programs for back-propagation can be quite short. In fact, I’d be glad to send a copy of a program for the XOR function problem to any interested readers of EXTROPY.

As I mentioned parenthetically earlier in this article, gradient descent does run into a problem if the steepest path in the error function does not lead to the smallest value of that function. An example is shown in Figure 5.

(FIGURE D HERE)

tion that I am most familiar with is speech recognition, in which researchers attempt to make computers figure what someone said while talking. The implications of achieving this goal are tremendous. Coupled with a natural language understanding system, a speech recognizer would allow us to communicate with computers as easily (or with as much difficulty!) as we talk to one another.

(Note: Virtually every science fiction movie and TV show I’ve ever seen has a computer that the characters talk to, instead of using a keyboard. Interestingly enough, these computers usually have silly, non-human-sounding voices. The technology to give computers human-like voices has been around for quite a while, whereas speech recognition is still in the works. This situation reminds me of the standard response to people who are horrified at the idea of putting only one’s head, and not the whole body, into cryonic suspension: By the time reanimation becomes possible, the technology for cloning a new body will probably have been around for some time.)

The back propagation technique is an example of supervised learning. We tell the network what its output should be, and the network modifies itself by comparing this response with its actual output, via the generalized delta rule. There are also techniques for unsupervised learning, in which we let the network decide what it’s going after, instead of telling it what we want. unsupervised learning is particularly interesting from an Extropian point of view, because it allows us to discover the underlying structure of a system without any explicit assumptions about what that structure is. I’ll discuss unsupervised learning in the next issue of EXTROPY.

FIGURE 5

In Figure 5, we start at a weight value of 0.4. The steepest path is to the left, but that path leads to an error value that is not the smallest. We have reached a local minimum in the error function, and we are stuck, because movement away from that minimum will give us a negative slope.

The problem of local minima is perhaps the most serious problem encountered in using gradient descent techniques. In Chapter 8 of PDP, Rumelhart, Hinton, and Williams report running into this problem only two times in several, hundred training sessions for an XOR neural network. The problem can be more serious in larger networks.

Despite this difficulty, back-propagation has been used successfully in a number of applications. The applica-

Put network picture here

EXTROPY #6

26

Summer 1990

EXTROPY #6

27

Summer 1990