Appendix

When abatch ofobjects needs tobegrasped with asetof chosen actions, each object is loaded into one of the threads such that the object'scenter is at position [0,0,0.6], The number of PyBullet physics simulation steps required to perform the grasp is only five, andallother computations forthegrasps -e.g. Initially,all touch signals are converted into touch charts using the pre-trainedtouchCNN. In the first part of the deformation process, we pass the image signal through a VGG-likeCNN with networkarchitecture described inTable6toextract image features. In the touch only setting, a collection of76 vision charts (here referred to as touch charts) is formed in the shape of a sphere and the vertices on the borders of these charts share edges with each other to allowcommunicationbetweenthem.