Object recall from natural-language descriptions for autonomous robotic grasping

Abstract

Humans acquire grasping skills through repeated interaction with the objects; and they internalize the knowledge of various physical attributes of such objects. Even blindfolded, humans can reasonably estimate a suitable grasp pose given only the object's description. Human brain relies on the knowledge of having seen such objects and recalling its physical features such as shape, size, weight to estimate feasible grasps. In such a scenario, knowledge of an object's features is key to executing a successful grasp. This paper aims to provide this 'recall' ability to robots by proposing a methodology to represent human memory of objects with a dataset of objects and their physical features. A joint probability distance metric is derived, which can match the natural language descriptions to reference object features so as to recall and identify a particular object from the reference dataset, thereby facilitating better grasp planning. Experiments were performed to evaluate the accuracy of object recall, and simulation of an anthropomorphic robot hand was conducted for object grasping based on the recalled object features. The experiment results showed the accuracy of the proposed metric and effectiveness in object grasping.