Abstract
In recent years, autonomous agents have surged in real-world environments such as our homes, offices, and public spaces. However, natural human-robot interaction remains a key challenge. In this paper, we introduce an approach that synergistically exploits the capabilities of large language models (LLMs) and multimodal vision-language models (VLMs) to enable humans to interact naturally with autonomous robots through conversational dialogue. We leveraged the LLMs to decode the high-level natural language instructions from humans and abstract them into precise robot actionable commands or queries. Further, we utilised the VLMs to provide a visual and semantic understanding of the robot's task environment. Our results with 99.13% command recognition accuracy and 97.96% commands execution success show that our approach can enhance human-robot interaction in real-world applications. The video demonstrations of this paper can be found at https://osf.io/wzyf6 and the code is available at our repository.
| Original language | English |
|---|---|
| Title of host publication | HRI 2024 Companion - Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction |
| Publisher | Association for Computing Machinery (ACM) |
| Pages | 808-812 |
| Number of pages | 5 |
| ISBN (Electronic) | 979-8-4007-0323-2 |
| ISBN (Print) | 979-8-4007-0323-2 |
| DOIs | |
| Publication status | Published - 11 Mar 2024 |
Publication series
| Name | ACM/IEEE International Conference on Human-Robot Interaction |
|---|---|
| ISSN (Electronic) | 2167-2148 |
Bibliographical note
Publisher Copyright:© 2024 Copyright held by the owner/author(s)
Keywords
- autonomous robots
- ChatGPT
- Human-robot interaction
- LLMs
- natural language interaction
- ROS
- VLMs
Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver