Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.

No matter what kind of theoretical concept is incorporated into the heart of such an investment model, we have a few similar issues that have to be properly addressed to increase the probability of generating efficient signals on out-of-sample (OOS) data. Among many others, these include the architecture of testing various models (machine learning, econometric, macroeconomic, or statistical approaches), the structure of the walk-forward procedure (usually consisting of numerous training, validation, and testing periods of different lengths), hyperparameters tuning and parameters optimization, model estimation phase, and finally the appropriate set of time series with possibly diverse characteristics of their distributions. The point is that all of these problems have to be designed optimally in order to avoid potential over-fitting issues and find the best possible variant of the investment model. Majority of papers undertaking the topic of AIS testing do not put proper attention to these problems and focus only on the empirical testing of one or several selected investment models, on a single instrument, over quite short data periods, usually without explaining the details of the whole procedure.