Probabilistic learning of robotic grasping strategy based on natural language object descriptions

Abstract

Humans learn to be dexterous by interacting with a wide variety of objects in different contexts. Given the description of an object’s physical attributes, humans can determine a proper strategy and grasp an object. This paper proposes an approach to determine grasping strategy for a 10 degree-of-freedom anthropomorphic robotic hand simply based on natural-language descriptions of an object. A probabilistic learning-based approach is proposed to help a robotic hand learn suitable grasp poses starting from the natural language description of the object. The solution involves a three-step learning model. In the first step, the information parsed from an object’s natural-language descriptions are used to identify/recognize the object by making use of a novel nearestneighbor distance metric. In the second step, the probability distribution of grasp types for the given object is learned using a deep neural net which takes in object features as input. The labels for this grasp learning model is supplied from human grasping trials. The discrete, two-dimensional grasp type/size vector is mapped back to the ten-dimensional robot joint-angles configuration space using linear inverse-kinematics models. The grasping strategy generated by the proposed approach is evaluated both by simulation study and execution of the grasps on an AR10 robotic hand. Index Terms—robotic grasping, human grasp primitives, natural language processing, object features extraction, neural networks classification.