Chauvin Y., Rumelhart D.E. (eds.) Backpropagation. Theory, Architectures, and Applications

Lawrence Erlbaum, 1995. — 575 p.Almost ten years have passed since the publication of the now classic volumes Parallel Distributed Processing: Explorations in the Microstructure of Cognition. These volumes marked a renewal in the study of brain-inspired computations as models of human cognition. Since the publication of these two volumes, thousands of scientists and engineers have joined the study of Artificial Neural Networks (or Parallel Distributed Processing) to attempt to respond to three fundamental questions: (1) how does the brain work? (2) how does the mind work? (3) how could we design machines with equivalent or greater capabilities than biological (including human) brains?
Progress in the last 10 years has given us a better grasp of the complexity of these three problems. Although connectionist neural networks have shed a feeble light on the first question, it has become clear that biological neurons and computations are more complex than their metaphorical connectionist equivalent by several orders of magnitude. Connectionist models of various brain areas, such as the hippocampus, the cerebellum, the olfactory bulb, or the visual and auditory cortices have certainly helped our understanding of their functions and internal mechanisms. But by and large, the biological metaphor has remained a metaphor. And neurons and synapses still remain much more mysterious than hidden units and weights.
Artificial neural networks have inspired not only biologists but also psychologists, perhaps more directly interested in the second question. Although the need for brain-inspired computations as models of the workings of the mind is still controversial, PDP models have been successfully used to model a number of behavioral observations in cognitive, and more rarely, clinical or social psychology. Most of the results are based on models of perception, language, memory, learning, categorization, and control. These results, however, cannot pretend to represent the beginning of a general understanding of the human psyche. First, only a small fraction of the large quantity of data amassed by experimental psychologists has been examined by neural network researchers. Second, some higher levels of human cognition, such as problem solving, judgment, reasoning, or decision making rarely have been addressed by the connectionist community. Third, most models of experimental data remain qualitative and limited in scope: No general connectionist theory has been proposed to link the various aspects of cognitive processes into a general computational framework. Overall, the possibility of an artificial machine that could learn how to function in the world with a reasonable amount of intelligence, communication, or "common sense" remains far away from our current state of knowledge. It is perhaps on the third problem, the design of artificial learning systems, expert in specific tasks, that connectionist approaches have made their best contribution. Such models have had an impact in many different disciplines, most of them represented in this volume. This trend is in part the result of advances in computer, communication, and data acquisition technologies. As databases of information are becoming ubiquitous in many fields, corresponding accurate models of the data-generating process are often unavailable. It is in these areas that machine learning approaches are making their greatest impact. And it is here that connectionist approaches are beneficially interbreeding with several other related disciplines such as statistical mechanics, statistical pattern recognition, signal processing, statistical inference, and information and decision theory.
It may be seen as somewhat of a disappointment to the great excitement of the late 1980s that the idea of "intelligent general learning systems" has to yield to local, specialized, often handcrafted neural networks with limited generalization capabilities. But it is also interesting to realize that prior domain knowledge needs to be introduced to constrain network architectures and statistical performance measures if these networks are to learn and generalize. With hindsight, this realization certainly appears to be a sign of maturity in the field.Backpropagation: The Basic Theory
Phoneme Recognition Using Time-Delay Neural Networks
Automated Aircraft Flare and Touchdown Control Using Neural Networks
Recurrent Backpropagation Networks
A Focused Backpropagation Algorithm for Temporal Pattern Recognition
Nonlinear Control with Neural Networks
Forward Models: Supervised Learning with a Distal Teacher
Backpropagation: Some Comments and Variations
Graded State Machines: The Representation of Temporal Contingencies in Feedback Networks
Spatial Coherence as an Internal Teacher for a Neural Network
Connectionist Modeling and Control of Finite State Systems Given Partial State Information
Backpropagation and Unsupervised Learning in Linear Networks
Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity
When Neural Networks Play Sherlock Holmes
Gradient Descent Learning Algorithms: A Unified Perspective