19 June 2023
Machine learning is nowadays ubiquitous in our society covering applications from healthcare to chatbots. Major breakthroughs have been made especially on feed-forward neural network architectures leveraging on convolutions, residual connections, and on attention mechanisms. Contrariwise, Recurrent Neural Network (RNN) architectures witnessed a slight disinterest in the last years. The soaring computational power provided by modern hardware accelerators made less appealing RNN architectures, due to their intrinsic difficulty to be implemented in parallel. Nevertheless, RNNs represent more plausibly the way microcircuits in the brain perform computations. Unlike feed-forward neural networks, the key characteristic of RNNs is to spatially encode into the RNN parameters and its hidden state the temporal information present in the input. In theory, the current recurrent hidden state conveys the context of the input signal from the infinite past to present time.
In this work, EMERGE partners from the University of Pisa and Delft University of Technology investigate a physically-inspired recurrent neural network derived from a continuous-time ODE modelling a network of coupled oscillators. Enthralled by the Reservoir Computing paradigm, the authors introduce the Randomly Coupled Oscillators (RCO) model, which leverages an untrained recurrent component with a smart random initialization. They analyse the architectural bias of RCO and its neural dynamics and derive sufficient conditions for the model to have a unique asymptotically uniformly stable input-driven solution. They also derive necessary conditions for stability, that permit to push the system of oscillators slightly beyond the edge of stability. The group empirically assess the effectiveness of RCO in terms of its stability and its long-term memory properties and compare its performance against both fully-trained and randomized recurrent models in a number of time series processing tasks. They find that RCO provides an excellent trade-off between robust long-term memory properties and ability to predict the behavior of non-linear, chaotic systems.
Read the paper in the link below.
More Information
