Human action recognition with a large-scale brain-inspired photonic computer

The recognition of human actions in video streams is a challenging task in computer vision, with cardinal applications in brain-computer interfaces and surveillance, for example. Recently, deep learning has produced remarkable results, but it can be hard to use in practice, as its training requires large datasets and special-purpose and energy-consuming hardware. In this work, we propose a photonic hardware approach. Our experimental set-up comprises off-the-shelf components and implements an easy-to-train recurrent neural network with 16,384 nodes, scalable to hundreds of thousands of nodes. The system, based on the reservoir computing paradigm, is trained to recognize six human actions from the KTH video database using either raw frames as inputs or a set of features extracted with the histograms of an oriented gradients algorithm. We report a classification accuracy of 91.3%, comparable to state-of-the-art digital implementations, while promising a higher processing speed in comparison to the existing hardware approaches. Because of the massively parallel processing capabilities offered by photonic architectures, we anticipate that this work will pave the way towards simply reconfigurable and energy-efficient solutions for real-time video processing. Photonic computing devices have been proposed as a high-speed and energy-efficient approach to implementing neural networks. Using off-the-shelf components, Antonik et al. demonstrate a reservoir computer that recognizes different forms of human action from video streams using photonic neural networks.