Anomaly detection to recognize unusual events in large scale systems in a time sensitive manner is critical in many industries, eg. bank fraud, enterprise systems, medical alerts, etc. Large-scale systems often grow in size and complexity over time, and anomaly detection algorithms need to adapt to changing structures. A hierarchical approach takes advantage of the implicit relationships in complex systems and localized context. The features in complex systems may vary drastically in data distribution, capturing different aspects from multiple data sources, and when put together provide a more complete view of the system. In this paper, two datasets are considered, the 1st comprising of system metrics from machines running on a cloud service, and the 2nd of application metrics from a large-scale distributed software system with inherent hierarchies and interconnections amongst its system nodes. Comparing algorithms, across the changepoint based PELT algorithm, cognitive learning-based Hierarchical Temporal Memory algorithms, Support Vector Machines and Conditional Random Fields provides a basis for proposing a Hierarchical Global-Local Conditional Random Field approach to accurately capture anomalies in complex systems across various features. Hierarchical algorithms can learn both the intricacies of specific features, and utilize these in a global abstracted representation to detect anomalous patterns robustly across multi-source feature data and distributed systems. A graphical network analysis on complex systems can further fine-tune datasets to mine relationships based on available features, which can benefit hierarchical models. Furthermore, hierarchical solutions can adapt well to changes at a localized level, learning on new data and changing environments when parts of a system are over-hauled, and translate these learnings to a global view of the system over time.

Learning from imbalanced tabular data is a significant challenge in real-world classification tasks. In such cases, neural network performance is substantially impaired due to implicit bias toward the majority class. Existing solutions attempt to eliminate the bias through data re-sampling or re-weighting the loss in the learning process. Still, these methods tend to overfit the minority samples and perform poorly when the structure of the minority class is highly irregular. Here, we propose a novel deep meta-learning technique to augment a given imbalanced dataset with new minority instances. These additional data are incorporated during the classifier's training process, and their contributions are learned explicitly. The augmented samples are modified throughout the training to optimize the classifiers' average-precision score on a validation set. Multiple experiments with synthetic and real-world imbalanced datasets demonstrate the advantage of the proposed method, leading to a significant gap in comparison to many existing baselines.

The identification and control of human factors in climate change is a rapidly growing concern and robust, real-time air-quality monitoring and forecasting plays a critical role in allowing effective policy formulation and implementation. This paper presents DELFI, a novel deep learning-based mixture model to make effective long-term predictions of Particulate Matter (PM) 2.5 concentrations. A key novelty in DELFI is its multi-scale approach to the forecasting problem. The observation that point predictions are more suitable in the short-term and probabilistic predictions in the long-term allows accurate predictions to be made as much as 24 hours in advance. DELFI incorporates meteorological data as well as pollutant-based features to ensure a robust model that is divided into two parts: (i) a stack of three Long Short-Term Memory (LSTM) networks that perform differential modelling of the same window of past data, and (ii) a fully-connected layer enabling attention to each of the components. Experimental evaluation based on deployment of 13 stations in the Delhi National Capital Region (Delhi-NCR) in India establishes that DELFI offers far superior predictions especially in the long-term as compared to even non-parametric baselines. The Delhi-NCR recorded the 3rd highest PM levels amongst 39 mega-cities across the world during 2011-2015 and DELFI's performance establishes it as a potential tool for effective long-term forecasting of PM levels to enable public health management and environment protection.

A significant percentage of COVID-19 survivors experience ongoing multisystemic symptoms that often affect daily living, a condition known as Long Covid or post-acute-sequelae of SARS-CoV-2 infection. However, identifying scientific articles relevant to Long Covid is challenging since there is no standardized or consensus terminology. We developed an iterative human-in-the-loop machine learning framework combining data programming with active learning into a robust ensemble model, demonstrating higher specificity and considerably higher sensitivity than other methods. Analysis of the Long Covid collection shows that (1) most Long Covid articles do not refer to Long Covid by any name (2) when the condition is named, the name used most frequently in the literature is Long Covid, and (3) Long Covid is associated with disorders in a wide variety of body systems.

Probabilistic Regression refers to predicting a full probability density function for the target conditional on the features. We present a nonparametric approach to this problem which combines base classifiers (typically gradient boosted forests) trained on different coarsenings of the target value. By combining such classifiers and averaging the resulting densities, we are able to compute precise conditional densities with minimal assumptions on the shape or form of the density. We combine this approach with a structured cross-entropy loss function which serves to regularize and smooth the resulting densities. Prediction intervals computed from these densities are shown to have high fidelity in practice. Furthermore, examining the properties of these densities on particular observations can provide valuable insight. We demonstrate this approach on a variety of datasets and show competitive performance, particularly on larger datasets.

This section contains case study questions that concern designing machine learning systems to solve practical problems. This section contains portfolio of data science projects completed by me for academic, self learning, and hobby purposes. Movie Recommendation: Designed 4 different models that recommend items on the MovieLens dataset. Trip Optimizer: Used XGBoost and evolutionary algorithms to optimize the travel time for taxi vehicles in New York City. Instacart Market Basket Analysis: Tackled the Instacart Market Basket Analysis challenge to predict which products will be in a user's next order.

Acute myeloid leukemia (AML) is one of the most common hematopoietic malignancies and exhibits a high rate of relapse and unfavorable outcomes. Ferroptosis, a relatively recently described type of cell death, has been reported to be involved in cancer development. However, the prognostic value of ferroptosis related genes (FRGs) in AML remains unclear. In this study, we found 54 differentially expressed ferroptosis related genes (DEFRGs) between AML and normal marrow tissues. Using the least absolute shrinkage and selection operator (LASSO) Cox regression analysis, we selected 10 DEFRGs that were associated with OS to build a prognostic signature.

Machine learning proves immensely helpful in many industries in automating tasks that earlier required human labor one such application of ML is predicting whether a particular trade will be profitable or not. In this article, we will learn how to predict a signal that indicates whether buying a particular stock will be helpful or not by using ML. Let's start by importing some libraries which will be used for various purposes which will be explained later in this article. Python libraries make it very easy for us to handle the data and perform typical and complex tasks with a single line of code. The dataset we will use here to perform the analysis and build a predictive model is Tesla Stock Price data.

This course introduces you to one of the main types of modelling families of supervised Machine Learning: Regression. You will learn how to train regression models to predict continuous outcomes and how to use error metrics to compare across different models. This course also walks you through best practices, including train and test splits, and regularization techniques. By the end of this course you should be able to: Differentiate uses and applications of classification and regression in the context of supervised machine learning Describe and use linear regression models Use a variety of error metrics to compare and select a linear regression model that best suits your data Articulate why regularization may help prevent overfitting Use regularization regressions: Ridge, LASSO, and Elastic net Who should take this course? This course targets aspiring data scientists interested in acquiring hands-on experience with Supervised Machine Learning Regression techniques in a business setting.

Abstract: In this paper, we study the splitting method based on alternating direction method of multipliers for support vector machine in reproducing kernel Hilbert space with lower semi-continuous loss function. If the loss function is lower semi-continuous and subanalytic, we use the Kurdyka-Lojasiewicz inequality to show that the iterative sequence induced by the splitting method globally converges to a stationary point. The numerical experiments also demonstrate the effectiveness of the splitting method. Abstract: Type 2 Diabetes is one of the most major and fatal diseases known to human beings, where thousands of people are subjected to the onset of Type 2 Diabetes every year. However, the diagnosis and prevention of Type 2 Diabetes are relatively costly in today's scenario; hence, the use of machine learning and deep learning techniques is gaining momentum for predicting the onset of Type 2 Diabetes.