29 May 2024
Echo state networks (ESNs) are time series processing models working under the echo state property (ESP) principle. The ESP is a notion of stability that imposes an asymptotic fading of the memory of the input. On the other hand, the resulting inherent architectural bias of ESNs may lead to an excessive loss of information, which in turn harms the performance in certain tasks with long short-term memory requirements.
In this work, EMERGE partners from the University of Pisa introduce a new ESN architecture called the Edge of Stability ESN to bring together the fading memory property and the ability to retain as much memory as possible.
The introduced model is based on defining the reservoir layer as a convex combination of a nonlinear reservoir (as in the standard ESN), and a linear reservoir that implements an orthogonal transformation. In virtue of a thorough mathematical analysis, the authors prove that the whole eigenspectrum of the Jacobian of the map can be contained in an annular neighborhood of a complex circle of controllable radius. Their experimental analysis shows that the model can reach the theoretical maximum short-term memory capacity. At the same time, in comparison to conventional reservoir approaches, it is shown to offer an excellent trade-off between memory and nonlinearity, as well as a significant improvement of performance in autoregressive nonlinear modeling and real-world time series modeling.
Read the paper in the link below.
More Information
