The Tokyo Metropolitan Government is developing a Generative AI Platform, which will allow government employees to create AI applications to assist with their work. The Tokyo Metropolitan Government and municipal governments throughout the Japanese capital are increasingly using generative artificial intelligence in their administrative operations. To support this trend, the metropolitan government is working with GovTech Tokyo, an affiliated organization that promotes digitalization in local governments, to develop a Generative AI Platform. The system will allow government employees to create generative AI applications tailored to their specific duties. By encouraging active use of the platform, Tokyo authorities aim to boost efficiency in public services and address growing concerns over labor shortages. In a time of both misinformation and too much information, quality journalism is more crucial than ever.

We present an architecture which lets us train deep, directed generative models with many layers of latent variables. We include deterministic paths between all latent variables and the generated output, and provide a richer set of connections between computations for inference and generation, which enables more effective communication of information throughout the model during training. To improve performance on natural images, we incorporate a lightweight autoregressive model in the reconstruction distribution. These techniques permit end-to-end training of models with 10+ layers of latent variables. Experiments show that our approach achieves state-of-the-art performance on standard image modelling benchmarks, can expose latent class structure in the absence of label information, and can provide convincing imputations of occluded regions in natural images.

The core premise of Expert Review needed some expert review. Superhuman has taken its writing assistant Grammarly on quite the merry-go-round ride regarding its approach to AI tools. In August, the company launched a feature called Expert Review that would offer feedback on your writing, offering AI-generated feedback that would appear to come from a famous writer or academic of note. These recreations were based on publicly available information from third-party LLMs, which sounds a lot like web crawlers of dubious legality were involved. The suggested experts would be based on the subject matter and could be anyone from great scientific minds to bestselling fiction authors to your friendly neighborhood tech bloggers.

The problem of model collapse has presented new challenges in iterative training of generative models, where such training with synthetic data leads to an overall degradation of performance. This paper looks at the problem from a statistical viewpoint, illustrating that one can actually hope for improvement when models are trained on data contaminated with synthetic samples, as long as there is some amount of fresh information from the true target distribution. In particular, we consider iterative training on samples sourced from a mixture of the true target and synthetic distributions. We analyze the entire iterative evolution in a next-token prediction language model, capturing how the interplay between the mixture weights and the sample size controls the overall long-term performance. With non-trivial mixture weight of the true distribution, even if it decays over time, simply training the model in a contamination-agnostic manner with appropriate sample sizes can avoid collapse and even recover the true target distribution under certain conditions. Simulation studies support our findings and also show that such behavior is more general for other classes of models.

The rapid progress in generative models has resulted in impressive leaps in generation quality, blurring the lines between synthetic and real data. Web-scale datasets are now prone to the inevitable contamination by synthetic data, directly impacting the training of future generated models. Already, some theoretical results on self-consuming generative models (a.k.a., iterative retraining) have emerged in the literature, showcasing that either model collapse or stability could be possible depending on the fraction of generated data used at each retraining step. However, in practice, synthetic data is often subject to human feedback and curated by users before being used and uploaded online. For instance, many interfaces of popular text-to-image generative models, such as Stable Diffusion or Midjourney, produce several variations of an image for a given query which can eventually be curated by the users. In this paper, we theoretically study the impact of data curation on iterated retraining of generative models and show that it can be seen as an implicit preference optimization mechanism .

This state of the art was recently critiqued by O'Bray et al.

Therefore, when trained on data generated by humans, we may not expect the artificial model to outperform the humans on their original objectives.

Preference data is crucial for aligning generative models with text prompts.

Answers to these questions must begin by formalizing the specification for what a generative algorithm for language should be doing.

Generative models are being increasingly used in science and industry applications.