We present a method to create diffusion-based video models from pretrained Text-to-Image (T2I) models. Recently, AnimateDiff proposed freezing the T2I model while only training temporal layers. We advance this method by proposing a unique architecture, incorporating a mapping network and frame-wise tokens, tailored for video generation while maintaining the diversity and creativity of the original T2I model. Key innovations include novel loss functions for temporal smoothness and a mitigating gradient sampling technique, ensuring realistic and temporally consistent video generation despite limited public video data. We have successfully integrated video-specific inductive biases into the architecture and loss functions. Our method, built on the frozen StableDiffusion model, simplifies training processes and allows for seamless integration with off-the-shelf models like ControlNet and DreamBooth.

Image inpainting, the task of reconstructing missing segments in corrupted images using available data, faces challenges in ensuring consistency and fidelity, especially under information-scarce conditions. Traditional evaluation methods, heavily dependent on the existence of unmasked reference images, inherently favor certain inpainting outcomes, introducing biases. Addressing this issue, we introduce an innovative evaluation paradigm that utilizes a self-supervised metric based on multiple re-inpainting passes. This approach, diverging from conventional reliance on direct comparisons in pixel or feature space with original images, emphasizes the principle of self-consistency to enable the exploration of various viable inpainting solutions, effectively reducing biases. Our extensive experiments across numerous benchmarks validate the alignment of our evaluation method with human judgment.

Surgical simulations offer a significant advantage by eliminating the need for patient involvement in skills practice, providing trainees with essential technical lessons before performing procedures on humans [24]. However, current computer-based simulations have lots of drawbacks, such as unrealistic visual appearance, lacking variability, and complex creation procedures taking into account the varying anatomical properties, all of which lead to diminishing the quality of surgical training. Therefore, AI-generated surgical simulations promise significant advancements in medical education since the underlying machine-learning models can learn the anatomical and visual characteristics of surgeries as well as their interactions with surgical tools from real-world data. Similar to recent works on image-guided surgery by Ramalhinho et al. [18] and Schneider et al. [23], our work focuses on laparoscopic surgery. We propose an approach for generating realistic laparoscopic videos conditioned on both text prompts and surgical tool positions. This lays the groundwork for a dynamic and interactive surgical training platform that mimics real-world scenarios. With this approach, we achieve state-of-the-art realism with an FID score of 33.43 and a pixel-wise F1 score of 0.72 for the control of tool positions. Moreover, we successfully generate coherent videos of single surgical actions.

Text-To-Image (TTI) models, such as DALL-E and StableDiffusion, have demonstrated remarkable prompt-based image generation capabilities. Multilingual encoders may have a substantial impact on the cultural agency of these models, as language is a conduit of culture. In this study, we explore the cultural perception embedded in TTI models by characterizing culture across three hierarchical tiers: cultural dimensions, cultural domains, and cultural concepts. Based on this ontology, we derive prompt templates to unlock the cultural knowledge in TTI models, and propose a comprehensive suite of evaluation techniques, including intrinsic evaluations using the CLIP space, extrinsic evaluations with a Visual-Question-Answer (VQA) model and human assessments, to evaluate the cultural content of TTI-generated images. To bolster our research, we introduce the CulText2I dataset, derived from four diverse TTI models and spanning ten languages. Our experiments provide insights regarding Do, What, Which and How research questions about the nature of cultural encoding in TTI models, paving the way for cross-cultural applications of these models.

Copy-Paste is a simple and effective data augmentation strategy for instance segmentation. By randomly pasting object instances onto new background images, it creates new training data for free and significantly boosts the segmentation performance, especially for rare object categories. Although diverse, high-quality object instances used in Copy-Paste result in more performance gain, previous works utilize object instances either from human-annotated instance segmentation datasets or rendered from 3D object models, and both approaches are too expensive to scale up to obtain good diversity. In this paper, we revisit Copy-Paste at scale with the power of newly emerged zero-shot recognition models (e.g., CLIP) and text2image models (e.g., StableDiffusion). We demonstrate for the first time that using a text2image model to generate images or zero-shot recognition model to filter noisily crawled images for different object categories is a feasible way to make Copy-Paste truly scalable. To make such success happen, we design a data acquisition and processing framework, dubbed ``X-Paste", upon which a systematic study is conducted. On the LVIS dataset, X-Paste provides impressive improvements over the strong baseline CenterNet2 with Swin-L as the backbone. Specifically, it archives +2.6 box AP and +2.1 mask AP gains on all classes and even more significant gains with +6.8 box AP, +6.5 mask AP on long-tail classes. Our code and models are available at https://github.com/yoctta/XPaste.

I hate the stance people take with this argument, because it comes off as unnecessarily antagonistic. Preventing anyone from sharing the technology or indeed from using it would be very hard, this is true, but heavy handed regulation could still kill public development. Rather than trying to argue that it's too late to stop the technology, it's much more productive to try and convince people on the positives of the technology, that there's great things that can come from it. You convince no one with an argument that there's nothing they can do about it, because they can certainly try, and they certainly can make the experience worse. No one should want things to be stuck where they are now, with people just trading dated copies of the tech in shadier parts of the internet, and possible consequences for sharing the art.

Recent diffusion-based AI art platforms are able to create impressive images from simple text descriptions. This makes them powerful tools for concept design in any discipline that requires creativity in visual design tasks. This is also true for early stages of architectural design with multiple stages of ideation, sketching and modelling. In this paper, we investigate how applicable diffusion-based models already are to these tasks. We research the applicability of the platforms Midjourney, DALL-E 2 and StableDiffusion to a series of common use cases in architectural design to determine which are already solvable or might soon be. We also analyze how they are already being used by analyzing a data set of 40 million Midjourney queries with NLP methods to extract common usage patterns. With this insights we derived a workflow to interior and exterior design that combines the strengths of the individual platforms.

StableDiffusion is a machine learning model for generating images from text published in August 2022. Service such as DALLE2 and Midjourney exist to generate images from text, but in both cases, the trained models are private and must be accessed via a web service. StableDiffusion allows users to freely generate images on their PCs because the trained models are publicly available. To use Stable Diffusion on Windows, pre-built binaries are provided by GRisk and available at the link below. Since the default parameters do not generate a proper image, set the Steps to 150 and the Resolution to 512.