Large language models (LLMs) demonstrate remarkable performance on knowledge-intensive tasks, suggesting that real-world knowledge is encoded in their model parameters. However, besides explorations on a few probing tasks in limited knowledge domains, it is not well understood how to evaluate LLMs' knowledge systematically and how well their knowledge abilities generalize, across a spectrum of knowledge domains and progressively complex task formats. To this end, we propose KGQuiz, a knowledge-intensive benchmark to comprehensively investigate the knowledge generalization abilities of LLMs. KGQuiz is a scalable framework constructed from triplet-based knowledge, which covers three knowledge domains and consists of five tasks with increasing complexity: true-or-false, multiple-choice QA, blank filling, factual editing, and open-ended knowledge generation. To gain a better understanding of LLMs' knowledge abilities and their generalization, we evaluate 10 open-source and black-box LLMs on the KGQuiz benchmark across the five knowledge-intensive tasks and knowledge domains. Extensive experiments demonstrate that LLMs achieve impressive performance in straightforward knowledge QA tasks, while settings and contexts requiring more complex reasoning or employing domain-specific facts still present significant challenges. We envision KGQuiz as a testbed to analyze such nuanced variations in performance across domains and task formats, and ultimately to understand, evaluate, and improve LLMs' knowledge abilities across a wide spectrum of knowledge domains and tasks.

The Defense Advanced Research Projects Agency (DARPA) is setting its sights on developing an AI system with a detailed self-understanding of the time dimensions of its learned knowledge. DARPA's Time-Aware Machine Intelligence (TAMI) research program and incubator is looking to develop a new class of neural network architectures that incorporate an explicit time dimension as a fundamental building block for network knowledge representation," according to the TAMI program solicitation. The overall goal is to create an AI system that will be able to "think in and about time" when exercising its learned task knowledge in task performance. Current neural networks do not explicitly model the inherent time characteristics of their encoded knowledge. Consequently, state-of-the-art machine learning does not have the expressive capability to reason with encoded knowledge using time.