Strengths: The main strength of the paper is in the environment, which will certainly be useful for the RL/embodied AI community. The NetHack environment proposed in the paper seems to fill a gap in exiting environments for RL research, which can help develop new RL algorithms, but also new problems related to embodied intelligence. The environment is procedurally generated and stochastic, which avoids having agents memorizing past episodes in order to solve the game, and makes some of the existing exploration methods such as Go-Explore fail. While the observations are symbolic, they contain a large number of symbols corresponding to the different game elements, as well as natural language, creating opportunities for combining NLP and RL. The game entities are compositional, meaning that agents can reason about common attributes to interact with entities of different classes (line 108).

Weaknesses: My first concern is that this model seems far from minimalism. Generating correct answer for RPM is an interesting task. But one of the reasons it is interesting to the current AI community is that humans can somehow generate some results correctly without huge amount of training. Although this work demonstrates the possibility of generator that can show some reasoning capability, I highly speculate that this is a distillation from the subnetworks for context extraction, which is trained with strong supervision. There is still a long distance from this model and human brain. The latter one is believed to be designed by nature following minimalism.

I have read the reviews and the author response and I have also asked an expert AC to also provide a comment in lieu of a 4th reviewer (pasted below for reference). Taken all these together I will recommend acceptance, with a note. NOTE TO AUTHORS: This work is going to be the reference paper for using generation as opposed to discrimination. As such, it is really crucial to set the right path for evaluating model in a fair and rigorous way, so that research that follows on builds on a solid base. The presented evaluation has some issues (see points bellow).

Update after reading the authors' response: The authors' response answered my questions well. One limitation that I missed in my initial review is that the coordinates of the parameter theta (the true signal sequence) are not assumed to be bounded; the only assumption is that theta lies in a total variation ball. This means that the only bound on these coordinates is through the bound C on the total variation of the sequence. Hence, the dependence on the bound B on the sequence (L-infty norm of theta) is implicit and replaced by the worst-case upper bound C, which leads to a dependence on C 2 instead of B*C on the intersection of those balls. I think that this limitation should be addressed, given that in the online learning literature it is more customary to provide the explicit dependence on B. To be specific, this would entail assuming that theta lies in the intersection of a TV ball of radius C and an L-infinity ball of radius B (where both B and C can be assumed to be known, though adaptivity to those can be considered), and providing matching upper and lower bound over this class.

The main contribution of this paper is in solving the finite-sum minimax problem arising from off-line policy evaluation with nonlinear function approximation. The minimax problem is non-convex in the primal variable and strong convexity in the dual subproblem, and a single time-scale algorithm is proposed to find an approximate stationary point. Although it does not address the full stochastic TD learning problem, the progress in the finite-sum off-line version is quite meaningful.

Robots have shown enormous potential to alleviate repetitive, and dangerous tasks from human workers, such as assembly, infrastructure inspection, material handling and heavy rigging [4-6]. Integrating the artificial intelligence (AI) agent with a physical robotic system could further improve the precision, reliability, and consistency of operations with competent training [7, 8]. While AI-enabled robots excel in performing repetitive and predefined tasks, dexterous and complex tasks still pose a significant difficulty such as welding and pipe insertion [9, 10]. Training a robot to perform these dexterous tasks demands delicate manipulation and adaptive force control, which induces diversity and several potential actions leading to a substantial increase in the complexity of the learning process and resulting in slow convergence or lack of convergence [11] To tackle the challenges of learning in high-dimensional action spaces, Imitation Learning (IL) based methods are applied to leverage demonstrations from human experts or proficient use of human demonstrations as a form of instruction and reduce the size of action spaces that need to be explored [12-14]. Generative Adversarial Imitation Learning (GAIL)[15] could further address some key limitations of traditional IL by mitigating distributional shifts, thus enabling better exploration and performance in unseen states and generalizing better to new tasks [15].

The widespread adoption of large language models (LLMs) marks a transformative era in technology, especially within the educational sector. This paper explores the integration of LLMs within learning management systems (LMSs) to develop an adaptive learning management system (ALMS) personalized for individual learners across various educational stages. Traditional LMSs, while facilitating the distribution of educational materials, fall short in addressing the nuanced needs of diverse student populations, particularly in settings with limited instructor availability. Our proposed system leverages the flexibility of AI to provide a customizable learning environment that adjusts to each user's evolving needs. By integrating a suite of general-purpose and domain-specific LLMs, this system aims to minimize common issues such as factual inaccuracies and outdated information, characteristic of general LLMs like OpenAI's ChatGPT. This paper details the development of an ALMS that not only addresses privacy concerns and the limitations of existing educational tools but also enhances the learning experience by maintaining engagement through personalized educational content.

The dream of achieving a student-teacher ratio of 1:1 is closer than ever thanks to the emergence of large language models (LLMs). One potential application of these models in the educational field would be to provide feedback to students in university introductory programming courses, so that a student struggling to solve a basic implementation problem could seek help from an LLM available 24/7. This article focuses on studying three aspects related to such an application. First, the performance of two well-known models, GPT-3.5T and GPT-4T, in providing feedback to students is evaluated. The empirical results showed that GPT-4T performs much better than GPT-3.5T, however, it is not yet ready for use in a real-world scenario. This is due to the possibility of generating incorrect information that potential users may not always be able to detect. Second, the article proposes a carefully designed prompt using in-context learning techniques that allows automating important parts of the evaluation process, as well as providing a lower bound for the fraction of feedbacks containing incorrect information, saving time and effort. This was possible because the resulting feedback has a programmatically analyzable structure that incorporates diagnostic information about the LLM's performance in solving the requested task. Third, the article also suggests a possible strategy for implementing a practical learning tool based on LLMs, which is rooted on the proposed prompting techniques. This strategy opens up a whole range of interesting possibilities from a pedagogical perspective.

One of the most widely used methods to evaluate LLMs are Multiple Choice Question (MCQ) tests. MCQ benchmarks enable the testing of LLM knowledge on almost any topic at scale as the results can be processed automatically. To help the LLM answer, a few examples called few shots can be included in the prompt. Moreover, the LLM can be asked to answer the question directly with the selected option or to first provide the reasoning and then the selected answer, which is known as chain of thought. In addition to checking whether the selected answer is correct, the evaluation can look at the LLM-estimated probability of its response as an indication of the confidence of the LLM in the response. In this paper, we study how the LLM confidence in its answer depends on whether the model has been asked to answer directly or to provide the reasoning before answering. The results of the evaluation of questions on a wide range of topics in seven different models show that LLMs are more confident in their answers when they provide reasoning before the answer. This occurs regardless of whether the selected answer is correct. Our hypothesis is that this behavior is due to the reasoning that modifies the probability of the selected answer, as the LLM predicts the answer based on the input question and the reasoning that supports the selection made. Therefore, LLM estimated probabilities seem to have intrinsic limitations that should be understood in order to use them in evaluation procedures. Interestingly, the same behavior has been observed in humans, for whom explaining an answer increases confidence in its correctness.

We study the problem of interactive decision making in which the underlying environment changes over time subject to given constraints. We propose a framework, which we call \textit{hybrid Decision Making with Structured Observations} (hybrid DMSO), that provides an interpolation between the stochastic and adversarial settings of decision making. Within this framework, we can analyze local differentially private (LDP) decision making, query-based learning (in particular, SQ learning), and robust and smooth decision making under the same umbrella, deriving upper and lower bounds based on variants of the Decision-Estimation Coefficient (DEC). We further establish strong connections between the DEC's behavior, the SQ dimension, local minimax complexity, learnability, and joint differential privacy. To showcase the framework's power, we provide new results for contextual bandits under the LDP constraint.