Integrating new knowledge without catastrophic interference: Computational and theoretical investigations in a hierarchically structured environment
Abstract: According to complementary learning systems theory, integrating new memories into a multi-layer neural network without interfering with what is already known depends on interleaving presentation of the new memories with ongoing presentations of items previously learned. I use deep linear neural networks in hierarchically structured environments previously analyzed by Saxe, McClelland, and Ganguli (SMG) to gain new insights into this process. For the environment I will consider in this talk, its content can be described by the singular value decomposition (SVD) of the environment's input-output covariance matrix, in which each successive dimension corresponds to categorical split in the hierarchical environment. Prior work showed that deep linear networks are sufficient to learn the content of the environment, and they do so in a stage-line way, with each dimension strength rising from near-zero to its maximum strength after a delay inversely proportional to the strength of the dimension, as previously demonstrated by Saxe et al. Several observations are then accessible when we consider learning a new item previously not encountered in the micro-environment. (1) The item can be examined in terms of its projection onto the existing structure, and whether it adds a new categorical split. (2) To the extent the item projects onto existing structure, including it in the training corpus leads to the rapid adjustment of the representation of the categories involved, and effectively no adjustment occurs to categories onto which the new item does not project at all. (3) Learning a new split is slow, and its learning dynamics show the same delayed rise to maximum that depends on the dimension's strength. These observations them motivate the development of a similarity-weighted interleaved learning scheme in which only items similar to the to-be-learned new item need be presented to avoid catastrophic interference.
About the speaker: Jay McClelland received his Ph.D. in Cognitive Psychology from the University of Pennsylvania in 1975. He served on the faculty of the University of California, San Diego, before moving to Carnegie Mellon in 1984, where he became a University Professor and held the Walter Van Dyke Bingham Chair in Psychology and Cognitive Neuroscience. He was a founding Co-Director of the Center for the Neural Basis of Cognition, a joint project of Carnegie Mellon and the University of Pittsburgh. In 2006 McClelland moved to the Department of Psychology at Stanford University, where he served as department chair from fall 2009 through summer 2012. He is currently the Lucie Stern Professor in the Social Sciences, and the founding Director of the Center for Mind, Brain and Computation at Stanford.
Over his career, McClelland has contributed to both the experimental and theoretical literatures in a number of areas, most notably in the application of connectionist/parallel distributed processing models to problems in perception, cognitive development, language learning, and the neurobiology of memory. He was a co-founder with David E. Rumelhart of the Parallel Distributed Processing (PDP) research group, and together with Rumelhart he led the effort leading to the publication in 1986 of the two-volume book, Parallel Distributed Processing, in which the parallel distributed processing framework was laid out and applied to a wide range of topics in cognitive psychology and cognitive neuroscience. McClelland and Rumelhart jointly received the 1993 Howard Crosby Warren Medal from the Society of Experimental Psychologists, the 1996 Distinguished Scientific Contribution Award (see citation) from the American Psychological Association, the 2001 Grawemeyer Prize in Psychology, and the 2002 IEEE Neural Networks Pioneer Award for this work.
McClelland has served as Senior Editor of Cognitive Science, as President of the Cognitive Science Society, as a member of the National Advisory Mental Health Council, and as President of the Federation of Associations in the Behavioral and Brain Sciences (FABBS). He is a member of the National Academy of Sciences, and he has received the APS William James Fellow Award for lifetime contributions to the basic science of psychology, the David E. Rumelhart prize for contributions to the theoretical foundations of Cognitive Science, the NAS Prize in Psychological and Cognitive Sciences, and the Heineken Prize in Cognitive Science.
McClelland currently teaches on the PDP approach to cognition and its neural basis in the Psychology Department and in the Symbolic Systems Program at Stanford and conducts research on learning, memory, conceptual development, decision making, and mathematical cognition.