In statistical classification, machine learning, social and other sciences, a number of measures of association have been developed and used for assessing and comparing individual classifiers, raters, and their groups. Among the measures, we find the weighted kappa, extensively used by psychometricians, and the monotone and supremum correlation coefficients, prominently used by social scientists and statisticians. In this paper, we introduce, justify, and explore several new members of the class of functional correlation coefficients that naturally arise when comparing classifiers. We illustrate the performance of the coefficients by reanalyzing a number of confusion matrices that have appeared in the literature.

A free online course in artificial intelligence (AI) created by the University of Helsinki and technology consultancy Reaktor has drawn 140,000 students from around the world. Launched in spring 2018, the Elements of AI is available in English and Finnish. It was originally envisioned with the ambitious goal of training one percent of the Finnish population -- 55,000 people -- in the fundamentals of AI. Inspired by the Finnish model, Sweden and the Netherlands have created similar courses, with 15 other countries interested in developing comparable course for their citizens. Part of the course's popularity is the fact that it's available online for free and doesn't require any prerequisite technology skills.

Our world is getting smarter by day. Technological innovations such as AI and robotics are disrupting almost every industry, forcing most companies to embrace them or lose out on their benefits. Recent studies have shown that the AI market alone is set to surpass $100 billion within the next five years (by 2025). Even with the huge promises of AI, one sector that is falling behind when it comes to adopting these technologies is the higher education sector! Most universities and higher education institutions are still operating the old-fashioned way, which is noble, but a recipe for disaster.

In recent years, descriptive evaluation has been introduced as a new model for educational evaluation of Iranian students. The current descriptive evaluation method is based on four-valued logic. Assessing all students with only four values is led to a lack of relative justice and the creation of unrealistic equality. Also, the complexity of the evaluation process in the current method increases teacher errors likelihood. As a suitable solution, in this paper, a fuzzy descriptive evaluation system has been proposed. The proposed method is based on fuzzy logic, which is an infinite-valued logic and it can perform approximate reasoning on natural language propositions. By the proposed fuzzy system, student assessment is performed over the school year with infinite values instead of four values. But to eliminate the diversity of assigned values to students, at the end of the school year, the calculated values for each student will be rounded to the nearest value of the four standard values of the current descriptive evaluation system. It can be implemented easily in an appropriate smartphone app, which makes it much easier for the teachers to evaluate the evaluation process. In this paper, the evaluation process of the elementary third-grade mathematics course in Iran during the period from the beginning of the MEHR (The Seventh month of Iran) to the end of BAHMAN (The Eleventh Month of Iran) is examined by the proposed system. To evaluate the validity of this system, the proposed method has been simulated in MATLAB software.

Distance metric learning is successful in discovering intrinsic relations in data. However, most algorithms are computationally demanding when the problem size becomes large. In this paper, we propose a discriminative metric learning algorithm, and develop a distributed scheme learning metrics on moderate-sized subsets of data, and aggregating the results into a global solution. The technique leverages the power of parallel computation. The algorithm of the aggregated distance metric learning (ADML) scales well with the data size and can be controlled by the partition. We theoretically analyse and provide bounds for the error induced by the distributed treatment. We have conducted experimental evaluation of ADML, both on specially designed tests and on practical image annotation tasks. Those tests have shown that ADML achieves the state-of-the-art performance at only a fraction of the cost incurred by most existing methods.

Tree-based machine learning models such as random forests, decision trees, and gradient boosted trees are the most popular non-linear predictive models used in practice today, yet comparatively little attention has been paid to explaining their predictions. Here we significantly improve the interpretability of tree-based models through three main contributions: 1) The first polynomial time algorithm to compute optimal explanations based on game theory. 2) A new type of explanation that directly measures local feature interaction effects. 3) A new set of tools for understanding global model structure based on combining many local explanations of each prediction. We apply these tools to three medical machine learning problems and show how combining many high-quality local explanations allows us to represent global structure while retaining local faithfulness to the original model. These tools enable us to i) identify high magnitude but low frequency non-linear mortality risk factors in the general US population, ii) highlight distinct population sub-groups with shared risk characteristics, iii) identify non-linear interaction effects among risk factors for chronic kidney disease, and iv) monitor a machine learning model deployed in a hospital by identifying which features are degrading the model's performance over time. Given the popularity of tree-based machine learning models, these improvements to their interpretability have implications across a broad set of domains.

Many companies are just starting their machine learning journeys and 37% of organizations have implemented artificial intelligence according to a recent Gartner survey. If you've opened the door to machine learning, you might want to review 10 questions before starting a machine learning proof of concept or the complete guide to AI, machine learning, and deep learning. Machine learning is evolving, with new commercial breakthroughs, scientific advancements, framework improvements, and best practices frequently reported. We have a lot to learn from organizations that have large-scale machine learning programs and view artificial intelligence as core to their business. At the O'Reilly Artificial Intelligence Conference in New York last month I saw several common trends between Facebook's and Twitter's machine learning programs.

In order perform a large variety of tasks and to achieve human-level performance in complex real-world environments, Artificial Intelligence (AI) Agents must be able to learn from their past experiences and gain both knowledge and an accurate representation of their environment from raw sensory inputs. Traditionally, AI agents have suffered from difficulties in using only sensory inputs to obtain a good representation of their environment and then mapping this representation to an efficient control policy. Deep reinforcement learning algorithms have provided a solution to this issue. In this study, the performance of different conventional and novel deep reinforcement learning algorithms was analysed. The proposed method utilises two types of algorithms, one trained with a variant of Q-learning (DQN) and another trained with SARSA learning (DSN) to assess the feasibility of using direct feedback alignment, a novel biologically plausible method for back-propagating the error. These novel agents, alongside two similar agents trained with the conventional backpropagation algorithm, were tested by using the OpenAI Gym toolkit on several classic control theory problems and Atari 2600 video games. The results of this investigation open the way into new, biologically-inspired deep reinforcement learning algorithms, and their implementation on neuromorphic hardware.

Machine learning has evolved into an enabling technology for a wide range of highly successful applications. The potential for this success to continue and accelerate has placed machine learning (ML) at the top of research, economic and political agendas. Such unprecedented interest is fuelled by a vision of ML applicability extending to healthcare, transportation, defence and other domains of great societal importance. Achieving this vision requires the use of ML in safety-critical applications that demand levels of assurance beyond those needed for current ML applications. Our paper provides a comprehensive survey of the state-of-the-art in the assurance of ML, i.e. in the generation of evidence that ML is sufficiently safe for its intended use. The survey covers the methods capable of providing such evidence at different stages of the machine learning lifecycle, i.e. of the complex, iterative process that starts with the collection of the data used to train an ML component for a system, and ends with the deployment of that component within the system. The paper begins with a systematic presentation of the ML lifecycle and its stages. We then define assurance desiderata for each stage, review existing methods that contribute to achieving these desiderata, and identify open challenges that require further research.

Controlled natural languages (CNLs) are effective languages for knowledge representation and reasoning. They are designed based on certain natural languages with restricted lexicon and grammar. CNLs are unambiguous and simple as opposed to their base languages. They preserve the expressiveness and coherence of natural languages. In this report, we focus on a class of CNLs, called machine-oriented CNLs, which have well-defined semantics that can be deterministically translated into formal languages, such as Prolog, to do logical reasoning. Over the past 20 years, a number of machine-oriented CNLs emerged and have been used in many application domains for problem solving and question answering. However, few of them support non-monotonic inference. In our work, we propose non-monotonic extensions of CNL to support defeasible reasoning. In the first part of this report, we survey CNLs and compare three influential systems: Attempto Controlled English (ACE), Processable English (PENG), and Computer-processable English (CPL). We compare their language design, semantic interpretations, and reasoning services. In the second part of this report, we first identify typical non-monotonicity in natural languages, such as defaults, exceptions and conversational implicatures. Then, we propose their representation in CNL and the corresponding formalizations in a form of defeasible reasoning known as Logic Programming with Defaults and Argumentation Theory (LPDA).