Learning Complex Task Structures from Verbal Instruction

This research developed a novel approach to robot task learning from language-based instructions, which focuses on increasing the complexity of task representations that can be taught through verbal instruction. The major proposed contribution is the development of a framework for directly mapping a complex verbal instruction to an executable task representation, from a single training experience. The method can handle the following types of complexities: 1) instructions that use conjunctions to convey complex execution constraints (such as alternative paths of execution, sequential or nonordering constraints, as well as hierarchical representations) and 2) instructions that use prepositions and multiple adjectives to specify action/object parameters relevant for the task.

Role: Main mentor of undergraduate student.

Associated Publications:

• Monica Nicolescu, Natalie Arnold*, Janelle Blankenburg, David Feil-Seifer, Santosh Balajee Banisetty, Mircea Nicolescu, Andrew Palmer, Thor Monteverde. “Learning of Complex-Structured Tasks from Verbal Instruction.” In IEEE-RAS International Conference on Humanoid Robots (Humanoids), Toronto, Canada, Oct 2019. https://jblankenburg.github.io/publication/2019-Humanoids