The following problems are taken from the projects / assignments in the edX course Python for Data Science and the coursera course Applied Machine Learning in Python (UMich). The IMDB Movie Dataset (MovieLens 20M) is used for the analysis. The dataset is downloaded from here . This dataset contains 20 million ratings and 465,000 tag applications applied to 27,000 movies by 138,000 users and was released in 4/2015. Understand the trend in average ratings for different movie genres over years (from 1995 to 2015) and Correlation between the trends for different genres (8 different genres are considered: Animation, Comedy, Romance, Thriller, Horror, Sci-Fi and Musical).

Due to increasing urban population and growing number of motor vehicles, traffic congestion is becoming a major problem of the 21st century. One of the main reasons behind traffic congestion is accidents which can not only result in casualties and losses for the participants, but also in wasted and lost time for the others that are stuck behind the wheels. Early detection of an accident can save lives, provides quicker road openings, hence decreases wasted time and resources, and increases efficiency. In this study, we propose a preliminary real-time autonomous accident-detection system based on computational intelligence techniques. Istanbul City traffic-flow data for the year 2015 from various sensor locations are populated using big data processing methodologies. The extracted features are then fed into a nearest neighbor model, a regression tree, and a feed-forward neural network model. For the output, the possibility of an occurrence of an accident is predicted. The results indicate that even though the number of false alarms dominates the real accident cases, the system can still provide useful information that can be used for status verification and early reaction to possible accidents.

Turning on machine-learning based cloud security tools like Amazon Web Service's (AWS) new GuardDuty and Macie offerings might be a no-brainer for AWS customers. It raises the bar for attackers, but will not protect you from sophisticated adversaries, experts say. The AWS Macie service, announced in August, trains on the content of users' Amazon S3 buckets and alerts customers when it detects suspicious activity, with a focus on PCI, HIPAA, and GDPR compliance. AWS GuardDuty, a complementary offering announced at the end of November, uses machine learning to analyze AWS CloudTrail, VPC Flow Logs, and AWS DNS logs. Like Macie, GuardDuty focuses on anomaly detection to alert customers to suspicious activity.

Genomics has revolutionized biology, enabling the interrogation of whole transcriptomes, genome-wide binding sites for proteins, and many other molecular processes. However, individual genomic assays measure elements that interact in vivo as components of larger molecular machines. Understanding how these high-order interactions drive gene expression presents a substantial statistical challenge. Building on Random Forests (RF), Random Intersection Trees (RITs), and through extensive, biologically inspired simulations, we developed the iterative Random Forest algorithm (iRF). iRF trains a feature-weighted ensemble of decision trees to detect stable, high-order interactions with same order of computational cost as RF. We demonstrate the utility of iRF for high-order interaction discovery in two prediction problems: enhancer activity in the early Drosophila embryo and alternative splicing of primary transcripts in human derived cell lines. In Drosophila, among the 20 pairwise transcription factor interactions iRF identifies as stable (returned in more than half of bootstrap replicates), 80% have been previously reported as physical interactions. Moreover, novel third-order interactions, e.g. between Zelda (Zld), Giant (Gt), and Twist (Twi), suggest high-order relationships that are candidates for follow-up experiments. In human-derived cells, iRF re-discovered a central role of H3K36me3 in chromatin-mediated splicing regulation, and identified novel 5th and 6th order interactions, indicative of multi-valent nucleosomes with specific roles in splicing regulation. By decoupling the order of interactions from the computational cost of identification, iRF opens new avenues of inquiry into the molecular mechanisms underlying genome biology.

To build our first antivirus, we need to know about the virus first. Every antivirus program needs to constantly update their database to defend against new virus thread. Day by day not only our security systems are getting smarter but the viruses as well. Let's see some example of viruses - Polymorphic virus - This type of virus is hard to detect because this virus changes its signature every time it creates a replica. Antivirus software takes more than days to detect this virus type.

Cross-validation is one of the most popular model selection methods in statistics and machine learning. Despite its wide applicability, traditional cross validation methods tend to select overfitting models, due to the ignorance of the uncertainty in the testing sample. We develop a new, statistically principled inference tool based on cross-validation that takes into account the uncertainty in the testing sample. This new method outputs a set of highly competitive candidate models containing the best one with guaranteed probability. As a consequence, our method can achieve consistent variable selection in a classical linear regression setting, for which existing cross-validation methods require unconventional split ratios. When used for regularizing tuning parameter selection, the method can provide a further trade-off between prediction accuracy and model interpretability. We demonstrate the performance of the proposed method in several simulated and real data examples.

Abstract--The diversification (generating slightly varying separating discriminators) of Support V ector Machines (SVMs) for boosting has proven to be a challenge due to the strong learning nature of SVMs. Based on the insight that perturbing the SVM kernel may help in diversifying SVMs, we propose two kernel perturbation based boosting schemes where the kernel is modified in each round so as to increase the resolution of the kernel-induced Reimannian metric in the vicinity of the datapoints misclassified in the previous round. We propose a method for identifying the disjuncts in a dataset, dispelling the dependence on rule-based learning methods for identifying the disjuncts. We also present a new performance measure called Geometric Small Disjunct Index (GSDI) to quantify the performance on small disjuncts for balanced as well as class imbalanced datasets. Experimental comparison with a variety of state-of-the-art algorithms is carried out using the best classifiers of each type selected by a new approach inspired by multi-criteria decision making. The proposed method is found to outperform the contending state-of-the-art methods on different datasets (ranging from mildly imbalanced to highly imbalanced and characterized by varying number of disjuncts) in terms of three different performance indices (including the proposed GSDI). UPPORT V ector Machines (SVMs) [1] are a family of popular classifiers having elegant mathematical basis that can be used to model both linear and nonlinear (using the kernel trick) decision boundaries. The kernel trick is used to map the data to a higher dimensional feature space in order to facilitate linear separability between classes not linearly separable in the native input space. Shounak Datta, Sankha Subhra Mullick, and Swagatam Das are with the Electronics and Communication Sciences Unit, Indian Statistical Institute, Kolkata, India. Sayak Nag is with the Department of Instrumentation and Electronics Engineering, Jadavpur University, Kolkata, India. While being highly effective for non-overlapping classes, the performance of SVMs suffers in case of overlapping classes, due to the presence of data irregularities such as class imbalance (under-represented classes) [2]-[4] and small disjuncts (under-represented sub-concepts within classes) [5]-[7]. Class imbalanced often results in greater misclassification from the minority class.

A classification algorithm, called the Linear Centralization Classifier (LCC), is introduced. The algorithm seeks to find a transformation that best maps instances from the feature space to a space where they concentrate towards the center of their own classes, while maximimizing the distance between class centers. We formulate the classifier as a quadratic program with quadratic constraints. We then simplify this formulation to a linear program that can be solved effectively using a linear programming solver (e.g., simplex-dual). We extend the formulation for LCC to enable the use of kernel functions for non-linear classification applications. We compare our method with two standard classification methods (support vector machine and linear discriminant analysis) and four state-of-the-art classification methods when they are applied to eight standard classification datasets. Our experimental results show that LCC is able to classify instances more accurately (based on the area under the receiver operating characteristic) in comparison to other tested methods on the chosen datasets. We also report the results for LCC with a particular kernel to solve for synthetic non-linear classification problems.

Customer retention campaigns increasingly rely on predictive models to detect potential churners in a vast customer base. From the perspective of machine learning, the task of predicting customer churn can be presented as a binary classification problem. Using data on historic behavior, classification algorithms are built with the purpose of accurately predicting the probability of a customer defecting. The predictive churn models are then commonly selected based on accuracy related performance measures such as the area under the ROC curve (AUC). However, these models are often not well aligned with the core business requirement of profit maximization, in the sense that, the models fail to take into account not only misclassification costs, but also the benefits originating from a correct classification. Therefore, the aim is to construct churn prediction models that are profitable and preferably interpretable too. The recently developed expected maximum profit measure for customer churn (EMPC) has been proposed in order to select the most profitable churn model. We present a new classifier that integrates the EMPC metric directly into the model construction. Our technique, called ProfTree, uses an evolutionary algorithm for learning profit driven decision trees. In a benchmark study with real-life data sets from various telecommunication service providers, we show that ProfTree achieves significant profit improvements compared to classic accuracy driven tree-based methods.

The goal of the present study is to identify autism using machine learning techniques and resting-state brain imaging data, leveraging the temporal variability of the functional connections (FC) as the only information. We estimated and compared the FC variability across brain regions between typical, healthy subjects and autistic population by analyzing brain imaging data from a world-wide multi-site database known as ABIDE (Autism Brain Imaging Data Exchange). Our analysis revealed that patients diagnosed with autism spectrum disorder (ASD) show increased FC variability in several brain regions that are associated with low FC variability in the typical brain. We then used the enhanced FC variability of brain regions as features for training machine learning models for ASD classification and achieved 65% accuracy in identification of ASD versus control subjects within the dataset. We also used node strength estimated from number of functional connections per node averaged over the whole scan as features for ASD classification.The results reveal that the dynamic FC measures outperform or are comparable with the static FC measures in predicting ASD.