Touch is a crucial sensing modality that provides rich information about object properties and interactions with the physical environment. Humans and robots both benefit from using touch to perceive and interact with the surrounding environment (Johansson and Flanagan, 2009; Li et al., 2020; Calandra et al., 2017). However, no existing systems provide rich, multi-modal digital touch-sensing capabilities through a hemispherical compliant embodiment. Here, we describe several conceptual and technological innovations to improve the digitization of touch. These advances are embodied in an artificial finger-shaped sensor with advanced sensing capabilities. Significantly, this fingertip contains high-resolution sensors (~8.3 million taxels) that respond to omnidirectional touch, capture multi-modal signals, and use on-device artificial intelligence to process the data in real time. Evaluations show that the artificial fingertip can resolve spatial features as small as 7 um, sense normal and shear forces with a resolution of 1.01 mN and 1.27 mN, respectively, perceive vibrations up to 10 kHz, sense heat, and even sense odor. Furthermore, it embeds an on-device AI neural network accelerator that acts as a peripheral nervous system on a robot and mimics the reflex arc found in humans. These results demonstrate the possibility of digitizing touch with superhuman performance. The implications are profound, and we anticipate potential applications in robotics (industrial, medical, agricultural, and consumer-level), virtual reality and telepresence, prosthetics, and e-commerce. Toward digitizing touch at scale, we open-source a modular platform to facilitate future research on the nature of touch.

Despite significant advancements in network-based image harmonization techniques, there still exists a domain disparity between typical training pairs and real-world composites encountered during inference. Most existing methods are trained to reverse global edits made on segmented image regions, which fail to accurately capture the lighting inconsistencies between the foreground and background found in composited images. In this work, we introduce a self-supervised illumination harmonization approach formulated in the intrinsic image domain. First, we estimate a simple global lighting model from mid-level vision representations to generate a rough shading for the foreground region. A network then refines this inferred shading to generate a harmonious re-shading that aligns with the background scene. In order to match the color appearance of the foreground and background, we utilize ideas from prior harmonization approaches to perform parameterized image edits in the albedo domain. To validate the effectiveness of our approach, we present results from challenging real-world composites and conduct a user study to objectively measure the enhanced realism achieved compared to state-of-the-art harmonization methods.

In the near-future, dedicated telescopes observe Earth-like exoplanets in reflected light, allowing their characterization. Because of the huge distances, every exoplanet will be a single pixel, but temporal variations in its spectral flux hold information about the planet's surface and atmosphere. We test convolutional neural networks for retrieving a planet's rotation axis, surface and cloud map from simulated single-pixel flux and polarization observations. We investigate the assumption that the planets reflect Lambertian in the retrieval while their actual reflection is bidirectional, and of including polarization in retrievals. We simulate observations along a planet's orbit using a radiative transfer algorithm that includes polarization and bidirectional reflection by vegetation, desert, oceans, water clouds, and Rayleigh scattering in 6 spectral bands from 400 to 800 nm, at various photon noise levels. The surface-types and cloud patterns of the facets covering a model planet are based on probability distributions. Our networks are trained with simulated observations of millions of planets before retrieving maps of test planets. The neural networks can constrain rotation axes with a mean squared error (MSE) as small as 0.0097, depending on the orbital inclination. On a bidirectionally reflecting planet, 92% of ocean and 85% of vegetation, desert, and cloud facets are correctly retrieved, in the absence of noise. With realistic noise, it should still be possible to retrieve the main map features with a dedicated telescope. Except for face-on orbits, a network trained with Lambertian reflecting planets, yields significant retrieval errors when given observations of bidirectionally reflecting planets, in particular, brightness artefacts around a planet's pole. Including polarization improves retrieving the rotation axis and the accuracy of the retrieval of ocean and cloud facets.