This chapter is about data collection and data quality. The chapter starts by introducing key concepts of data. It then describes the most important methods of data collection used in business, economics, and policy analysis, such as web scraping, using administrative sources, and conducting surveys. We introduce aspects of data quality, such as validity and reliability of variables and coverage of observations. We discuss how to assess and link data quality to how the data was collected.

Machine Learning is a step in the direction of artificial intelligence (AI). Machine Learning is a program that analyses data and learns to predict the outcome. The term regression is used when you try to find the relationship between variables. In Machine Learning and statistical modeling, that relationship is used to predict the outcome of future events. Linear Regression uses the relationship between the data points to draw a straight line through all of them.

The threshold of the image is an important part of the image segmentation process. To apply a binary threshold on the image we should make the image in the desire condition where the binary threshold works properly. The meaning of the condition here is that when we do threshold, we want that the image condition should in some intensity level to segment. For example, if have an image A and then this will give a better threshold result. Color-spaces is a process to break a color image into different components of images.

Description Unsupervised Machine Learning involves finding patterns in datasets. After taking this course, students will be able to understand, implement in Python, and apply algorithms of Unsupervised Machine Learning to real-world datasets. The core of this course involves detailed study of the following algorithms: Clustering: Hierarchical, DBSCAN, K Means & Gaussian Mixture Model Dimension Reduction: Principal Component Analysis The course presents the math underlying these algorithms including normal distributions, expectation maximization, and singular value decomposition. The course also presents detailed explanation of code design and implementation in Python, including use of vectorization for speed up, and metrics for measuring quality of clustering and dimension reduction. The course codes are then used to address case studies involving real-world data to perform dimension reduction/clustering for the Iris Flowers Dataset, MNIST Digits Dataset (images), and BBC Text Dataset (articles).

Daouadi et al. [5] used deep learning methods on features based on the amount of interaction to and from each Social Media Platforms (SMPs) like Facebook, Twitter, Instagram Twitter account along with other set of features used previously, etc. have large user base all around the world that generates huge for fake user detection. Abu-El-Rub and Mueen [1] used trending amount of data every second. This includes a lot of posts by fake hashtags to detect bots interested in political trends. Graph based and spam users, typically used by many organisations around the techniques are used to cluster the collected bots and those are fed globe to have competitive edge over others. In this work, we aim to supervised learning to detect user's agreement/disagreement to at detecting such user accounts in Twitter using a novel approach.

Network-structured data becomes ubiquitous in daily life and is growing at a rapid pace. It presents great challenges to feature engineering due to the high non-linearity and sparsity of the data. The local and global structure of the real-world networks can be reflected by dynamical transfer behaviors among nodes. This paper proposes a network embedding framework to capture the transfer behaviors on structured networks via deep prediction models. We first design a degree-weight biased random walk model to capture the transfer behaviors on the network. Then a deep network embedding method is introduced to preserve the transfer possibilities among the nodes. A network structure embedding layer is added into conventional deep prediction models, including Long Short-Term Memory Network and Recurrent Neural Network, to utilize the sequence prediction ability. To keep the local network neighborhood, we further perform a Laplacian supervised space optimization on the embedding feature representations. Experimental studies are conducted on various datasets including social networks, citation networks, biomedical network, collaboration network and language network. The results show that the learned representations can be effectively used as features in a variety of tasks, such as clustering, visualization, classification, reconstruction and link prediction, and achieve promising performance compared with state-of-the-arts.

With machine learning systems increasingly being deployed in real-world settings, there is an urgent need for machine learning approaches that can adapt to--or are robust to--changes in the environment. Despite this, the dominant paradigm for supervised learning [Hastie et al., 2009] remains that of empirical risk minimization (ERM) [Vapnik, 2013], wherein a model is trained by minimizing a loss over a fixed set of training data. A key assumption underlying this approach is that the training data is from the same distribution as the test data--i.e., the distribution of the data does not change between training time and deployment. Such assumptions are well known to rarely hold in practice. Indeed, the distribution may change due to sample selection bias, nonstationarity in the environment [Quionero-Candela et al., 2009], or even adversarial perturbations [Szegedy et al., 2013, Madry et al., 2018], leaving machine learning models trained through ERM particularly susceptible to adversarial attacks [Szegedy et al., 2013, Carlini and Wagner, 2017] or to degraded performance from distribution shifts. Distributionally robust supervised learning (DRSL) seeks to address this issue by explicitly optimizing for solutions that are are robust to adversarial distribution shifts.

With the development of automatic diagnostics based on statistical predictive models, coming from any supervised machine learning (ML) algorithms, important issues about model validation have been raised. For example in the industrial nondestructive testing field (e.g. for aeronautic or nuclear industry), generalized automated inspection (that will allow large gain in terms of efficiency and economy) has to provide high guarantees in terms of performance. In this case, it is necessary to be able to select a validation data basis that will not be used for the training nor the selection of the ML model [3, 7]. This validation data basis (also referred as verification data in the literature) has not to be communicated to the ML developers because it will serve to realize an independent evaluation of the provided ML model (applying a cross validation method is then not possible). This validation sample is typically used to provide prediction residuals (which can be finely analyzed), as well as average ML model quality measures (as the mean square error in a regression problem or the misclassification rate in a classification problem). In this paper, we address the particular question about the way to select a "good" validation basis from a dataset useful to specify a ML model. We use indifferently the term "validation" and "test" for the basis (also called sample) because we restrict our problem to the distinction between a learning sample (which includes the ML fitting and selection phases) and a test sample. An important question is the number and the location of these test points.

Industrial Information Technology (IT) infrastructures are often vulnerable to cyberattacks. To ensure security to the computer systems in an industrial environment, it is required to build effective intrusion detection systems to monitor the cyber-physical systems (e.g., computer networks) in the industry for malicious activities. This paper aims to build such intrusion detection systems to protect the computer networks from cyberattacks. More specifically, we propose a novel unsupervised machine learning approach that combines the K-Means algorithm with the Isolation Forest for anomaly detection in industrial big data scenarios. Since our objective is to build the intrusion detection system for the big data scenario in the industrial domain, we utilize the Apache Spark framework to implement our proposed model which was trained in large network traffic data (about 123 million instances of network traffic) stored in Elasticsearch. Moreover, we evaluate our proposed model on the live streaming data and find that our proposed system can be used for real-time anomaly detection in the industrial setup. In addition, we address different challenges that we face while training our model on large datasets and explicitly describe how these issues were resolved. Based on our empirical evaluation in different use-cases for anomaly detection in real-world network traffic data, we observe that our proposed system is effective to detect anomalies in big data scenarios. Finally, we evaluate our proposed model on several academic datasets to compare with other models and find that it provides comparable performance with other state-of-the-art approaches.

This work proposes to learn fair low-rank tensor decompositions by regularizing the Canonical Polyadic Decomposition factorization with the kernel Hilbert-Schmidt independence criterion (KHSIC). It is shown, theoretically and empirically, that a small KHSIC between a latent factor and the sensitive features guarantees approximate statistical parity. The proposed algorithm surpasses the stateof-the-art algorithm, FATR (Zhu et al., 2018), in controlling the trade-off between fairness and residual fit on synthetic and real data sets. Tensor factorizations are used in many machine learning applications including link prediction (Dunlavy et al., 2011), clustering (Shashua et al., 2006), and recommendation (Kutty et al., 2012), where they are used to find vector representations (embeddings) of entities. With the widespread use of tensor factorization, we hope that decisions made from using tensor data are accurate but fair.