Dynamical phases of short-term memory: The emergence of slow-point and limit cycle mechanisms in recurrent neural networks

This thesis investigates the computational principles of short-term memory, specifically examining how information is maintained through sequential neural activity in recurrent neural networks (RNNs). While a longstanding theory posits that persistent activity holds information in memory—an idea that has dominated neuroscience for decades and is conceptually aligned with fixed-point attractors in dynamical systems—this work explores the alternative hypothesis that memory can be encoded in the transient, sequential firing patterns of large neural populations. We identify and rigorously characterize two primary mechanisms capable of supporting this dynamic process: slow- point manifolds, which generate direct, non-oscillatory sequences through a saddle-node bifurcation, and limit cycles, which provide a periodic, oscillatory basis for temporal coding. Through the use of simplified, yet analytically tractable, dynamical system models, we derive theoretical scaling laws that precisely relate the critical learning rate— the threshold beyond which training becomes unstable—to the duration of the memory delay. These laws predict that the difficulty of learning increases as a power-law function of the delay, but with different exponents for each mechanism. We provide rigorous empirical validation for these theoretical predictions by training and evaluating a large-scale dataset of over 80,000 RNNs on memory-dependent tasks. The findings reveal a fundamental and robust trade-off among memory duration, learning speed, and network stability, offering a principled computational explanation for the historical difficulty of learning long-term dependencies in RNNs. This framework also offers a new perspective on the biophysical constraints on memory in biological systems, suggesting why certain computational strategies might be favored over others. Furthermore, this work highlights how subtle alterations in task design can fundamentally change the underlying memory mechanisms learned by a network, providing concrete, experimentally testable predictions for systems neuroscience aimed at differentiating these computational strategies in vivo.