This work extends the Transformer language model architecture of GPT-2 by scaling it to 170 billion parameters, resulting in a new model called GPT-3. The paper demonstrates that when provided with zero or few labeled examples to condition on, this large model is capable of performing a multitude of language tasks without any further changes to model parameters. While on most tasks the zero-shot/few-shot performance is behind SOTA, the novelty lies in the demonstrated strong zero/few shot performance on diverse tasks. Clarity of exposition is another strength of the paper. One limitation is lack of reproducibility due to the massive compute necessary to train the model.

There is burgeoning interest in designing AI-based systems to assist humans in designing computing systems, including tools that automatically generate computer code. The most notable of these comes in the form of the first self-described "AI pair programmer," GitHub Copilot, a language model trained over open source GitHub code. However, code often contains bugs--and so, given the vast quantity of unvetted code that Copilot has processed, it is certain that the language model will have learned from exploitable, buggy code. This raises concerns on the security of Copilot's code contributions. In this work, we systematically investigate the prevalence and conditions that can cause GitHub Copilot to recommend insecure code.

In recent years, enormous progress has been made in the field of large language models (LLMs). Based on neural network architectures, specifically transformer models, they have proven highly effective in natural language processing (NLP). The models are designed to understand, generate, and work with human language. Trained on large datasets consisting of text from the Internet, books, articles, and many other data sources, the model learns to predict the next word in a sentence based on previous words. LLMs are not only able to generate human language but can also generate source code to support humans in the implementation of software systems.

Weaknesses: * This method is most suitable for variables that have a single parent in the causal DAG -- the class label. This severely restricts the class of attributes that can be modeled and manifests in the paper as experiments with simple attributes (colors in AO-CLEVr, and materials in Zappos). In fact, prior work has noted that attributes (or other compositional modifiers) manifest very differently for different objects ([36] gives the examples from prior work: "fluffy" for towels vs. dogs, "ripe" for one fruit vs. another etc.). For these attributes, and many others, the data generating process is not so straightforward -- there are edges from both attribute labels and object labels to the core features. The authors do acknowledge this limitation in L326, however it is an important weakness to consider given that _difficult_ instances in real world datasets (where both object and attribute are parents of \phi_a for example) are fairly prevalent.

All four reviewers appreciated the neat idea contained in this paper which is also shown to work well in practice. The authors open up the way for studying data generation processes through causal interventions, which is a novel and technically interesting direction. Most importantly, it is a significant direction which is expected to stimulate further research in the field. I am recommending acceptance of this paper, however please consider revising the manuscript to address R4's remarks about clarity and R2's and R3's remarks about deeper discussion of failure cases and limitations.

Writing ---------- The paper is well written and well positioned wrt previous work, overall. However, the main technical core of the paper is only about 1.5 pages long. It would have been better imho to have less detailed results tables and reuse that space for explaining the technique in greater detail. Novelty ---------- While I agree that this paper is the first to address zero-shot semantic segmentation, there have been many papers on zero-shot image classification, abd also some papers on zero-shot object class detection. Importantly, the technique proposed is a direct adaptation of [7], a previous technique for zero-shot image classification.

The submission originally received mixed scores. The reviewers acknowledged that the submission is the first work that addresses zero-shot semantic image segmentation, but they criticized the novelty compared to existing classification methods and identified shortcomings of the experimental setting. The reviewers appreciated the author response and subsequently discussed the work in detail. However, the conclusion was that despite some shortcomings, the community will potentially benefit from the work and that the submission should be accepted. The authors are strongly encouraged to make their results understandable and reproducible, e.g. by releasing code, complete outputs and ideally pretrained models.

Rakuten's chief dismissed skeptics who call the Japanese e-commerce pioneer's mobile foray a mistake and said the telecom arm is central for future growth through artificial intelligence. A decision to enter Japan's cutthroat wireless market has saddled Rakuten with four years of losses, weighing on its cash-churning online shopping mall and finance operations. But that mobile arm and its 8 million-plus users help train an AI poised to expand the conglomerate's business, according to billionaire founder Hiroshi Mikitani. The amount of exclusive data Rakuten gathers from its users is "extremely powerful," Mikitani said in an interview. "We have no intent to compete against OpenAI or Google. But we will actively build a more vertically integrated, specialized AI."

Large Language Models (LLMs) have emerged as pivotal tools in information access and generation, particularly through their capabilities of producing factually accurate texts. As these models become increasingly integrated into various applications, ensuring the accuracy of their responses has become very important. The performance and reliability of LLMs in generating accurate information are significantly influenced by multiple factors, including training data quality, model architecture design, and post-training optimization processes [1], [2], [3]. However, a significant challenge in the deployment of LLMs lies in their propensity to generate nonfactual responses, a phenomenon commonly referred to as hallucination. These hallucinations fundamentally undermine the reliability and faithfulness of LLMs, presenting substantial obstacles to their widespread adoption across various domains [4], [5]. The mitigation of hallucinations has consequently emerged as a critical area of research within the field. While various strategies have been proposed and implemented to reduce hallucinations, showing promising improvements in the faithfulness of LLMs for general-purpose tasks, domain-specific applications remain particularly challenging [6], [7], [8]. In this paper, we present a comprehensive study evaluating the extent of hallucination in LLMs under domain-specific prompting, with a particular focus on scientific literature. We develop and implement a systematic evaluation pipeline to assess fifteen prominent open-source LLMs: Qwen 2.5 [9], Gemma 2 [10], Llama 3 [11], Phi 3 [12], Orca 2 [13], Mistral v-0.3 [14], Deepseek-llm [15], Olmo-2 [16], Mistral-Nemo [17], Eurus-2 [18], and Solar-Pro [19].

The quality of Supervised Fine-Tuning (SFT) data plays a critical role in enhancing the conversational capabilities of Large Language Models (LLMs). However, as LLMs become more advanced, the availability of high-quality human-annotated SFT data has become a significant bottleneck, necessitating a greater reliance on synthetic training data. In this work, we introduce Condor, a novel two-stage synthetic data generation framework that incorporates World Knowledge Tree and Self-Reflection Refinement to produce high-quality SFT data at scale. Our experimental results demonstrate that a base model fine-tuned on only 20K Condor-generated samples achieves superior performance compared to counterparts. The additional refinement stage in Condor further enables iterative self-improvement for LLMs at various scales (up to 72B), validating the effectiveness of our approach. Furthermore, our investigation into the scaling for synthetic data in post-training reveals substantial unexplored potential for performance improvements, opening promising avenues for future research.