That covers most of what I really want to say about supervised learning. Again, please excuse my glossing over the many, many details; each one of these topics can be discussed in much more detail, and is so discussed in the papers cited.
For the sake of completeness, however, I should say a little about two forms of Hebbian learning which I did not mention above.
Most people who work with Hebbian learning would argue that there are really two different kinds of Hebbian learning system which could be used on supervised learning problems. There are local associative memory systems, which I discussed above. But there are also global systems, which are generally linear, and require that inputs be decorrelated before they enter the supervised learning system. A lot of decorrelating networks have been designed for use with such nets. However, after discussing this matter with Pribram, I am convinced that this latter class of network is not relevant to systems like the human brain. Pribram and others have shown again and again that biological representation systems have a great deal of redundancy (e.g., like wavelets but with a 1.5 amplification factor instead of 2, etc., as in the Simmons talk today). One would expect such redundancy, in any system which also has to have a high degree of fault tolerance. This is inconsistent with the mathematical requirement of orthogonality. In addition, the limitation to the linear case is not encouraging, either.
In 1992, I developed an alternative learning design which appears Hebbian in character, but has radically different properties[4, 18]. It provides a mathematical representation of certain ideas by Pribram about dendritic field processing, which the talk today by Simmons provides strong empirical support for. It is closely linked to Chris Atkeson's experiments with locally-weighted regression, which has performed very well in robotics experiments at MIT. In retrospect, as I reconsider the issue of information flows around dendrites, I suspect that the design still needs to be revised, to account explicitly for the three-dimensional nature of local information flows, at least for biological modeling. In any case, the alternative design is still feedforward in terms of what it accomplishes; therefore, it might possibly be worth considering as a model of the innermost loop of the neocortex, but it does not obviate the need for simultaneous recurrence, and for the unusual kinds of nonHebbian feedback (as in Figure 8) required to adapt key parts of the neocortex and hippocampus -- if those systems are as powerful as I suspect.
In summary, I predict that the human brain contains some very complex circuitry, as required to solve some very complex adaptation problems. At present, most people would find it hard to believe that something that complicated is there, even though it does fit these new results of Freeman and so on. I think we need new experiments, based on living slices, to help get home to people that it's this kind of complexity that's in that system, and that the old models are simply not good enough. So that's the end of supervised learning.
Now let me talk about neurocontrol. This is a subject I've talked about for eight hours at a stretch, so I will have to cut out a lot of important material here today. First, I want to talk about why this is crucial to understanding intelligence. I'll skip over my slides on engineering application areas. I will talk a little bit about the kind of designs that engineers are using today, but only a little. Mainly I will focus on the design concepts which relate directly to understanding the brain.
This is a chart (Figure 9) that people look at and say, "I already know this." But if people could understand the implications of what they already know, this world would be a different place. There are some implications in what we already know that people haven't thought through. Now what I am going to talk about here is the reason why the human brain is a neurocontroller; let me give you the argument in a few stages.