Efficient Statistical Methods for Evaluating Trading Agent Performance

AAAI Conferences

Market simulations, like their real-world counterparts, are typically domains of high complexity, high variability, and incomplete information. The performance of autonomous agents in these markets depends both upon the strategies of their opponents and on various market conditions, such as supply and demand. Because the space for possible strategies and market conditions is very large, empirical analysis in these domains becomes exceedingly difficult. Researchers who wish to evaluate their agents must run many test games across multiple opponent sets and market conditions to verify that agent performance has actually improved. Our approach is to improve the statistical power of market simulation experiments by controlling their complexity, thereby creating an environment more conducive to structured agent testing and analysis. We develop a tool that controls variability across games in one such market environment, the Trading Agent Competition for Supply Chain Management (TAC SCM), and demonstrate how it provides an efficient, systematic method for TAC SCM researchers to analyze agent performance.

Application of Deep Neural Networks to assess corporate Credit Rating

arXiv.org Machine Learning

Recent literature implements machine learning techniques to assess corporate credit rating based on financial statement reports. In this work, we analyze the performance of four neural network architectures (MLP, CNN, CNN2D, LSTM) in predicting corporate credit rating as issued by Standard and Poor's. We analyze companies from the energy, financial and healthcare sectors in US. The goal of the analysis is to improve application of machine learning algorithms to credit assessment. To this end, we focus on three questions. First, we investigate if the algorithms perform better when using a selected subset of features, or if it is better to allow the algorithms to select features themselves. Second, is the temporal aspect inherent in financial data important for the results obtained by a machine learning algorithm? Third, is there a particular neural network architecture that consistently outperforms others with respect to input features, sectors and holdout set? We create several case studies to answer these questions and analyze the results using ANOVA and multiple comparison testing procedure.

How AI could lead to improvements in the health care industry


A lot of money goes into artificial intelligence research, and advocates of the technology have praised it as a way to revolutionize health care. The global market for AI in health care is expected to rise from $1.3 billion in 2019 to $10 billion by 2024, according to investment bank Morgan Stanley. Researchers recently published new findings in the Lancet Digital Health Journal that concluded AI is on par with medical professionals in identifying diseases. However, in order to really tap into how AI can improve health care, scientists concluded more research is needed. The research centered around something called deep learning, which uses algorithms, data and computing to emulate human intelligence.

Identifying Real Estate Opportunities using Machine Learning

arXiv.org Machine Learning

The real estate market is exposed to many fluctuations in prices, because of existing correlations with many variables, some of which cannot be controlled or might even be unknown. Housing prices can increase rapidly (or in some cases, also drop very fast), yet the numerous listings available online where houses are sold or rented are not likely to be updated that often. In some cases, individuals interested in selling a house (or apartment) might include it in some online listing, and forget about updating the price. In other cases, some individuals might be interested in deliberately setting a price below the market price in order to sell the home faster, for various reasons. In this paper we aim at developing a machine learning application that identifies opportunities in the real estate market in real time, i.e., houses that are listed with a price substantially below the market price. This program can be useful for investors interested in the housing market. The application is formally implemented as a regression problem, that tries to estimate the market price of a house given features retrieved from public online listings. For building this application, we have performed a feature engineering stage in order to discover relevant features that allows attaining a high predictive performance. Several machine learning algorithms have been tested, including regression trees, k-NN and neural networks, identifying advantages and handicaps of each of them.

A Theory of Dichotomous Valuation with Applications to Variable Selection

arXiv.org Machine Learning

An econometric or statistical model may undergo a marginal gain when a new variable is admitted, and a marginal loss if an existing variable is removed. The value of a variable to the model is quantified by its expected marginal gain and marginal loss. Assuming the equality of opportunity, we derive a few formulas which evaluate the overall performance in potential modeling scenarios. However, the value is not symmetric to marginal gain and marginal loss; thus, we introduce an unbiased solution. Simulation studies show that our new approaches significantly outperform a few practice-used variable selection methods.