AIC Seminar Series

Following natural language instructions requires transforming language into situated conditional action; robustly following instructions, despite the director’s natural mistakes and omissions, requires the pragmatic combination of language, action, and domain knowledge. This dissertation demonstrates building a software agent that parses, models and executes human-written natural language instructions to accomplish complex navigation tasks as often as people following the same instructions. By selectively removing various syntactic, semantic, and pragmatic abilities, this work empirically measures how often these abilities are necessary to correctly navigate along extended routes through unknown, large-scale environments to novel destinations.

To study how route instructions are written and followed, we collected a corpus of about 1600 free-form instructions from 30 directors for 252 routes in three virtual environments. About 100 other people followed these instructions and rated them for quality, successfully reaching and identifying the destination only about two-thirds of the trials. Our software agent, Marco, followed the same instructions in the same environments with a success rate approaching human levels. Marco’s performance was a strong predictor of human performance and ratings of individual instructions. By ablation testing, we demonstrate that implicit actions are crucial for following verbal instructions using an approach integrating language, knowledge and action. We also measure the performance impact of a wide range of linguistic, execution, and spatial abilities in successfully following natrual language route instructions.

Matt MacMahon has designed, implemented, tested, and deployed intelligent robotic systems while working at some of the world’s leading AI laboratories at NASA Johnson Space Center, NASA Ames Research Center, and the Navy Center for Applied Research in Artificial Intelligence. Matt’s focus has been on human-robot interaction with adjustable autonomy and reactive execution in the face of unpredictable events. He has published work on these topics and multi-agent systems at AAAI, CogSci, ICRA, FSR, AAMAS, and AI Magazine. Matt is completing his doctorate in Software Engineering at the University of Texas at Austin, under the supervision of Dr. Benjamin Kuipers, Computer Sciences, and Dr. Brian Stankiewicz, Psychology.