Publication: Euler State Networks: Non-dissipative Reservoir Computing

24 February 2024

The interest in studying neural network architectures from a dynamical system perspective has recently been attracting increasing research attention. The key insight is that the computation performed by some kinds of neural networks, e.g., residual networks, can be understood as the numerical solution to an ordinary differential equation (ODE) through discretization. This intuitively simple observation brings about the possibility of imposing desirable properties in the behaviour of the neural network by imposing specific conditions on the corresponding ODE. Stability plays a key role in this sense, being related to the propagation of both input signals, during inference, and gradients, during training.

Inspired by the numerical solution of ordinary differential equations, EMERGE partners from the University of Pisa propose a novel Reservoir Computing (RC) model, called the Euler State Network (EuSN). The presented approach makes use of forward Euler discretization and antisymmetric recurrent matrices to design reservoir dynamics that are both stable and non-dissipative by construction.

The authors mathematical analysis shows that the resulting model is biased towards a unitary effective spectral radius and zero local Lyapunov exponents, intrinsically operating near the edge of stability. Experiments on long-term memory tasks show the clear superiority of the proposed approach over standard RC models in problems requiring effective propagation of input information over multiple time steps.

Furthermore, results on time-series classification benchmarks indicate that EuSN can match (or even exceed) the accuracy of trainable Recurrent Neural Networks, while retaining the training efficiency of the RC family, resulting in up to -fold savings in computation time and 464-fold savings in energy consumption. At the same time, our results on time-series modeling tasks show competitive results against standard RC when the architecture is complemented by direct input-readout connections.

Read the paper in the link below.