Embedding in Neural Networks: a-priori design of hybrid computers for prediction

The prediction of complex signals is among the most important applications of recurrent Neural Networks (RNN). Yet, no theory which completely describes prediction in RNNs exists. As such, these systems remain black boxes. Based on nearest neighbors theory and random nonlinear mapping, we fully describe the mechanisms employed by RNNs solving this essential task. Our approach combines machine learning techniques (Reservoir Computing) and dynamical systems theory. We derive optimization cost functions which are (a) task specific, and (b) go far beyond the simple optimization of the prediction error. Going beyond, we demonstrate the consequences resulting from our theory. Based on our analysis of an RNN stabilizing of an arrhythmic heart, we amend this system by a nonvolatile external FIFO memory. Furthermore, we are able a-priori specify the FIFO's properties, strongly reducing the previous training efforts of such systems.