A Multi-channel Episodic Memory Model for Human Action Learning and Recognition

Human actions can be realized by observing the trajectories of skeleton joints. In this paper, we propose an unsupervised episodic memory learning model for skeleton based action learning and Recognition. The proposed model, Multi-channel Episodic Memory Adaptive Resonance Theory (McEM-ART), consists of three layers: short term memory, working memory and Episodic memory. The short term memory layer is formed by multiple ART networks to obtain sensory data and cluster them into neurons in working memory layer. Instead of obtaining the whole skeleton as an input, we divide the human skeleton into three parts, upper part body, main body and lower part body. Each of them is then feed to McEM-ART short term memory layer for learning. Episodic memory layer extracts novel events and encodes spatio-temporal connection between them as episodes by generating cognitive neurons incrementally for action Recognition. Comparing with previous works, McEM-ART further integrates a novel memory anticipation functions for encoding crucial events and episodes and recalling them using partial and inexact cues. Experimental results demonstrate that McEM-ART is capable of clustering human skeleton data into event neurons, encoding sequence of activation events as episode neurons for action recalling and Recognition.