Selecting an appropriate evaluation metric for classifiers is crucial for model comparison and parameter optimization, yet there is not consensus on a universally accepted metric that serves as a definitive standard. Moreover, there is often a misconception about the perceived need to mitigate imbalance in datasets used to train classification models. Since the final goal in classifier optimization is typically maximizing the return of investment or, equivalently, minimizing the Total Classification Cost (TCC), we define Weighted Accuracy (WA), an evaluation metric for binary classifiers with a straightforward interpretation as a weighted version of the well-known accuracy metric, coherent with the need of minimizing TCC. We clarify the conceptual framework for handling class imbalance in cost-sensitive scenarios, providing an alternative to rebalancing techniques. This framework can be applied to any metric that, like WA, can be expressed as a linear combination of example-dependent quantities and allows for comparing the results obtained in different datasets and for addressing discrepancies between the development dataset, used to train and validate the model, and the target dataset, where the model will be deployed. It also specifies in which scenarios using UCCs-unaware class rebalancing techniques or rebalancing metrics aligns with TCC minimization and when it is instead counterproductive. Finally, we propose a procedure to estimate the WA weight parameter in the absence of fully specified UCCs and demonstrate the robustness of WA by analyzing its correlation with TCC in example-dependent scenarios.

This paper reflects on the process of performing Thematic Analysis with Large Language Models (LLMs). Specifically, the paper deals with the problem of analytical saturation of initial codes, as produced by LLMs. Thematic Analysis is a well-established qualitative analysis method composed of interlinked phases. A key phase is the initial coding, where the analysts assign labels to discrete components of a dataset. Saturation is a way to measure the validity of a qualitative analysis and relates to the recurrence and repetition of initial codes. In the paper we reflect on how well LLMs achieve analytical saturation and propose also a novel technique to measure Inductive Thematic Saturation (ITS). This novel technique leverages a programming framework called DSPy. The proposed novel approach allows a precise measurement of ITS.

Accurate quantification of model uncertainty has long been recognized as a fundamental requirement for trusted AI. In regression tasks, uncertainty is typically quantified using prediction intervals calibrated to a specific operating point, making evaluation and comparison across different studies difficult. Our work leverages: (1) the concept of operating characteristics curves and (2) the notion of a gain over a simple reference, to derive a novel operating point agnostic assessment methodology for prediction intervals. The paper describes the corresponding algorithm, provides a theoretical analysis, and demonstrates its utility in multiple scenarios. We argue that the proposed method addresses the current need for comprehensive assessment of prediction intervals and thus represents a valuable addition to the uncertainty quantification toolbox.

Professor Barry O'Sullivan, Director of Insight at UCC, became the first Irish member of the EurAI board in 2014. Professor Barry O'Sullivan, Director of Insight at UCC, has been elected as Deputy President of the European Association for Artificial Intelligence (EurAI) at its General Assembly held in The Hague. Professor O'Sullivan, who became the first Irish member of the EurAI board after being elected in 2014, said: "It's a huge honour to be the first Irish person to serve on the board of the European Association for Artificial Intelligence, so to be elected as its deputy president is a major thrill." UCC professor named deputy president of world's largest AI body https://t.co/WNjjanZlfz EurAI, formerly ECCAI, was established in July 1982 as a representative body for the European artificial intelligence community.

The paper introduces a novel technique for improving the performance and scalability of best-first progression-based conformant planners. The technique is inspired by different well-known techniques from classical planning, such as landmark and stratification. Its most salient feature is that it is relatively cheap to implement yet quite effective when applicable. The effectiveness of the proposed technique is demonstrated by the development of new conformant planners by integrating the technique in various state-of-the-art conformant planners and an extensive experimental evaluation of the new planners using benchmarks collected from various sources. The result shows that the technique can be applied in several benchmarks and helps improve both performance and scalability of conformant planners.