Vision and manipulation are inextricably intertwined in the primate brain. Tantalizing results from neuroscience are shedding light on the mixed motor and sensory representations used by the brain during reaching, grasping, and object recognition. We now know a great deal about what happens in the brain during these activities, but not necessarily why. Is the integration we see functionally important, or just a reflection of evolution’s lack of enthusiasm for sharp modularity? We instantiate these results in robotic form to probe the technical advantages and to find any lacunae in existing models. We believe it would be missing the point to investigate this on a platform where dexterous manipulation and sophisticated machine vision are already implemented in their mature form, and instead follow a developmental approach from simpler primitives.

People: Francesco Rea, Carlo Ciliberto, Lorenzo Natale

Related project: ITALK, CHRIS, Poeticon

Manipulation can help develop object recognition and the construction of multimodal models of objects. The iCub can grasp and move objects, and use the resulting visual, acoustic, tactile and motor information to learn models of generic objects autonomously.							Manipulating objects and recognizing them visually from the point of view of the robot. Following the approach in [ref 1], we would like to exploit manipulation to learn about objects in a developmental manner.