Skip to main navigation Skip to search Skip to main content

The Conversation is the Command: Interacting with Real-World Autonomous Robot Through Natural Language

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In recent years, autonomous agents have surged in real-world environments such as our homes, offices, and public spaces. However, natural human-robot interaction remains a key challenge. In this paper, we introduce an approach that synergistically exploits the capabilities of large language models (LLMs) and multimodal vision-language models (VLMs) to enable humans to interact naturally with autonomous robots through conversational dialogue. We leveraged the LLMs to decode the high-level natural language instructions from humans and abstract them into precise robot actionable commands or queries. Further, we utilised the VLMs to provide a visual and semantic understanding of the robot's task environment. Our results with 99.13% command recognition accuracy and 97.96% commands execution success show that our approach can enhance human-robot interaction in real-world applications. The video demonstrations of this paper can be found at https://osf.io/wzyf6 and the code is available at our repository.

Original languageEnglish
Title of host publicationHRI 2024 Companion - Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction
PublisherAssociation for Computing Machinery (ACM)
Pages808-812
Number of pages5
ISBN (Electronic)979-8-4007-0323-2
ISBN (Print)979-8-4007-0323-2
DOIs
Publication statusPublished - 11 Mar 2024

Publication series

NameACM/IEEE International Conference on Human-Robot Interaction
ISSN (Electronic)2167-2148

Bibliographical note

Publisher Copyright:
© 2024 Copyright held by the owner/author(s)

Keywords

  • autonomous robots
  • ChatGPT
  • Human-robot interaction
  • LLMs
  • natural language interaction
  • ROS
  • VLMs

Cite this