We introduce a novel, training-free method for sampling differentiable representations (diffreps) using pretrained diffusion models. Rather than merely mode-seeking, our method achieves sampling by "pulling back" the dynamics of the reverse-time process--from the image space to the diffrep parameter space--and updating the parameters according to this pulled-back process. We identify an implicit constraint on the samples induced by the diffrep and demonstrate that addressing this constraint significantly improves the consistency and detail of the generated objects. Our method yields diffreps with substantially improved quality and diversity for images, panoramas, and 3D NeRFs compared to existing techniques. Our approach is a general-purpose method for sampling diffreps, expanding the scope of problems that diffusion models can tackle.

Image inpainting is the process of taking an image and generating lost or intentionally occluded portions. Inpainting has countless applications including restoring previously damaged pictures, restoring the quality of images that have been degraded due to compression, and removing unwanted objects/text. Modern inpainting techniques have shown remarkable ability in generating sensible completions for images with mask occlusions. In our paper, an overview of the progress of inpainting techniques will be provided, along with identifying current leading approaches, focusing on their strengths and weaknesses. A critical gap in these existing models will be addressed, focusing on the ability to prompt and control what exactly is generated. We will additionally justify why we think this is the natural next progressive step that inpainting models must take, and provide multiple approaches to implementing this functionality. Finally, we will evaluate the results of our approaches by qualitatively checking whether they generate high-quality images that correctly inpaint regions with the objects that they are instructed to produce.

Denoising Diffusion Probabilistic Models (DDPMs) have recently achieved remarkable results in conditional and unconditional image generation. The pre-trained models can be adapted without further training to different downstream tasks, by guiding their iterative denoising process at inference time to satisfy additional constraints. For the specific task of image inpainting, the current guiding mechanism relies on copying-and-pasting the known regions from the input image at each denoising step. However, diffusion models are strongly conditioned by the initial random noise, and therefore struggle to harmonize predictions inside the inpainting mask with the real parts of the input image, often producing results with unnatural artifacts. Our method, dubbed GradPaint, steers the generation towards a globally coherent image. At each step in the denoising process, we leverage the model's "denoised image estimation" by calculating a custom loss measuring its coherence with the masked input image. Our guiding mechanism uses the gradient obtained from backpropagating this loss through the diffusion model itself. GradPaint generalizes well to diffusion models trained on various datasets, improving upon current state-of-the-art supervised and unsupervised methods.

Accelerating the learning processes for complex tasks by leveraging previously learned tasks has been one of the most challenging problems in reinforcement learning, especially when the similarity between source and target tasks is low or unknown. In this work, we propose a REPresentation-And-INstance Transfer algorithm (REPAINT) for deep actor-critic reinforcement learning paradigm. In representation transfer, we adopt a kickstarted training method using a pre-trained teacher policy by introducing an auxiliary cross-entropy loss. In instance transfer, we develop a sampling approach, i.e., advantage-based experience replay, on transitions collected following the teacher policy, where only the samples with high advantage estimates are retained for policy update. We consider both learning an unseen target task by transferring from previously learned teacher tasks and learning a partially unseen task composed of multiple sub-tasks by transferring from a pre-learned teacher sub-task. In several benchmark experiments, REPAINT significantly reduces the total training time and improves the asymptotic performance compared to training with no prior knowledge and other baselines.

This image shows the original paintings (across the top), as illuminated by different light sources (from left, 6410K, 4291K & 3410K) and below are the same paintings reproduced by the AI in RePaint. This image shows that RePaint works effectively in dramatically different lighting conditions.MIT CSAIL A team from the Massachusetts Institute of Technology (MIT) Computer Science and Artificial Intelligence Laboratory (CSAIL) has designed a system called RePaint that uses artificial intelligence (AI) and 3D printing to reproduce paintings. The system is a workflow for spectral reproduction of paintings can capture the spectral color, regardless of light, and reproduce it. This technology could be used by museums to reproduce artwork that has been stolen or is on loan to another museum. Despite the fact that the reproductions made by the researchers were only the size of a business card, the team noted that RePaint was four times more accurate than state-of-the-art physical models at creating the exact color shades for different artworks.

If you've ever wanted to get your hands on the Mona Lisa, you may just be in luck. For researchers have found a way for art fanatics to create their own version of a priceless masterpiece, through a combination of AI and 3D printing. The replicas have been made by researchers from the Massachusetts Institute of Technology (MIT) using a piece of software called'RePaint'. The replicas have been made by researchers from the Massachusetts Institute of Technology (MIT) using a piece of software called'RePaint' (pictured) Despite the progress so far, the team says they have a few improvements to make before they can whip up a dazzling dupe of'Starry Night.' 'If you just reproduce the colour of a painting as it looks in the gallery, it might look different in your home,' says Changil Kim, one of the researchers from MIT that published a paper on the system, which will be presented in December. 'Our system works under any lighting condition, which shows a far greater colour reproduction capability than almost any other previous work.'