Multimodal communication for human-friendly robot partners in informationally structured space

Naoyuki Kubota, Yuichiro Toda

Research output: Contribution to journalArticle

19 Citations (Scopus)

Abstract

This paper proposes a multimodal communication method for human-friendly robot partners based on various types of sensors. First, we explain informationally structured space to extend the cognitive capabilities of robot partners based on environmental systems. Next, we discuss the suitable measurement range for recognition technologies of touch interface, voice recognition, human detection, gesture recognition, and others. Based on the suitable measurement ranges, we propose an integration method to estimate human behaviors based on the human detection using color image and 3-D distance information, and gesture recognition by the multilayered spiking neural network using the time series of human-hand positions. Furthermore, we propose a conversation system to realize the multimodal communication with a person. Finally, we show several experimental results of the proposed method, and discuss the future direction of this research.

Original languageEnglish
Article number6392476
Pages (from-to)1142-1151
Number of pages10
JournalIEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews
Volume42
Issue number6
DOIs
Publication statusPublished - Dec 1 2012
Externally publishedYes

Fingerprint

Gesture recognition
Robots
Communication
Speech recognition
Time series
Color
Neural networks
Sensors

Keywords

  • Computational intelligence
  • human-robot interaction
  • information services
  • intelligent robots
  • ubiquitous computing

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Software
  • Information Systems
  • Human-Computer Interaction
  • Computer Science Applications
  • Electrical and Electronic Engineering

Cite this

@article{1abc79c2068f4662a881cfa439cd976d,
title = "Multimodal communication for human-friendly robot partners in informationally structured space",
abstract = "This paper proposes a multimodal communication method for human-friendly robot partners based on various types of sensors. First, we explain informationally structured space to extend the cognitive capabilities of robot partners based on environmental systems. Next, we discuss the suitable measurement range for recognition technologies of touch interface, voice recognition, human detection, gesture recognition, and others. Based on the suitable measurement ranges, we propose an integration method to estimate human behaviors based on the human detection using color image and 3-D distance information, and gesture recognition by the multilayered spiking neural network using the time series of human-hand positions. Furthermore, we propose a conversation system to realize the multimodal communication with a person. Finally, we show several experimental results of the proposed method, and discuss the future direction of this research.",
keywords = "Computational intelligence, human-robot interaction, information services, intelligent robots, ubiquitous computing",
author = "Naoyuki Kubota and Yuichiro Toda",
year = "2012",
month = "12",
day = "1",
doi = "10.1109/TSMCC.2012.2213810",
language = "English",
volume = "42",
pages = "1142--1151",
journal = "IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews",
issn = "1094-6977",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "6",

}

TY - JOUR

T1 - Multimodal communication for human-friendly robot partners in informationally structured space

AU - Kubota, Naoyuki

AU - Toda, Yuichiro

PY - 2012/12/1

Y1 - 2012/12/1

N2 - This paper proposes a multimodal communication method for human-friendly robot partners based on various types of sensors. First, we explain informationally structured space to extend the cognitive capabilities of robot partners based on environmental systems. Next, we discuss the suitable measurement range for recognition technologies of touch interface, voice recognition, human detection, gesture recognition, and others. Based on the suitable measurement ranges, we propose an integration method to estimate human behaviors based on the human detection using color image and 3-D distance information, and gesture recognition by the multilayered spiking neural network using the time series of human-hand positions. Furthermore, we propose a conversation system to realize the multimodal communication with a person. Finally, we show several experimental results of the proposed method, and discuss the future direction of this research.

AB - This paper proposes a multimodal communication method for human-friendly robot partners based on various types of sensors. First, we explain informationally structured space to extend the cognitive capabilities of robot partners based on environmental systems. Next, we discuss the suitable measurement range for recognition technologies of touch interface, voice recognition, human detection, gesture recognition, and others. Based on the suitable measurement ranges, we propose an integration method to estimate human behaviors based on the human detection using color image and 3-D distance information, and gesture recognition by the multilayered spiking neural network using the time series of human-hand positions. Furthermore, we propose a conversation system to realize the multimodal communication with a person. Finally, we show several experimental results of the proposed method, and discuss the future direction of this research.

KW - Computational intelligence

KW - human-robot interaction

KW - information services

KW - intelligent robots

KW - ubiquitous computing

UR - http://www.scopus.com/inward/record.url?scp=84871766484&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84871766484&partnerID=8YFLogxK

U2 - 10.1109/TSMCC.2012.2213810

DO - 10.1109/TSMCC.2012.2213810

M3 - Article

AN - SCOPUS:84871766484

VL - 42

SP - 1142

EP - 1151

JO - IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews

JF - IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews

SN - 1094-6977

IS - 6

M1 - 6392476

ER -