Performance Analysis
A View Independent Classification Framework for Yoga Postures
Chasmai, Mustafa, Das, Nirjhar, Bhardwaj, Aman, Garg, Rahul
Yoga is a globally acclaimed and widely recommended practice for a healthy living. Maintaining correct posture while performing a Yogasana is of utmost importance. In this work, we employ transfer learning from Human Pose Estimation models for extracting 136 key-points spread all over the body to train a Random Forest classifier which is used for estimation of the Yogasanas. The results are evaluated on an in-house collected extensive yoga video database of 51 subjects recorded from 4 different camera angles. We propose a 3 step scheme for evaluating the generalizability of a Yoga classifier by testing it on 1) unseen frames, 2) unseen subjects, and 3) unseen camera angles. We argue that for most of the applications, validation accuracies on unseen subjects and unseen camera angles would be most important. We empirically analyze over three public datasets, the advantage of transfer learning and the possibilities of target leakage. We further demonstrate that the classification accuracies critically depend on the cross validation method employed and can often be misleading. To promote further research, we have made key-points dataset and code publicly available.
Cross-Validation in Machine Learning
The model performance is based on dividing the known data into two parts, one to train the model and the other to test the prediction performance, thus obtaining the model accuracy and adjusting it according to the results. However, accuracy depends on how we slip the data, which can lead to possible biases in the model that prevent accuracy from generalizing to unseen data. Cross-validation is used to combat the random split of the data. This is a method that allows testing the performance of a predictive machine learning model, based on the same principle of the Train-Test split technique but with the difference that it must be performed k times and obtain the accuracy of each attempt. This technique is known as k-folds, where each fold is a specific division of the data different from the rest.
Machine Learning Based Radiomics for Glial Tumor Classification and Comparison with Volumetric Analysis
Turk, Sevcan, Oguz, Kaya, Orman, Mehmet, Caliskan, Emre, Ertan, Yesim, Ozgiray, Erkin, Akalin, Taner, Srinivasan, Ashok, Kitis, Omer
Purpose; The purpose of this study is to classify glial tumors into grade II, III and IV categories noninvasively by application of machine learning to multi-modal MRI features in comparison with volumetric analysis. Methods; We retrospectively studied 57 glioma patients with pre and postcontrast T1 weighted, T2 weighted, FLAIR images, and ADC maps acquired on a 3T MRI. The tumors were segmented into enhancing and nonenhancing portions, tumor necrosis, cyst and edema using semiautomated segmentation of ITK-SNAP open source tool. We measured total tumor volume, enhancing-nonenhancing tumor, edema, necrosis volume and the ratios to the total tumor volume. Training of a support vector machine (SVM) classifier and artificial neural network (ANN) was performed with labeled data designed to answer the question of interest. Specificity, sensitivity, and AUC of the predictions were computed by means of ROC analysis. Differences in continuous measures between groups were assessed by using Kruskall Wallis, with post hoc Dunn correction for multiple comparisons. Results; When we compared the volume ratios between groups, there was statistically significant difference between grade IV and grade II-III glial tumors. Edema and tumor necrosis volume ratios for grade IV glial tumors were higher than that of grade II and III. Volumetric ratio analysis could not distinguish grade II and III tumors successfully. However, SVM and ANN correctly classified each group with accuracies up to 98% and 96%. Conclusion; Application of machine learning methods to MRI features can be used to classify brain tumors noninvasively and more readily in clinical settings.
Covert Message Passing over Public Internet Platforms Using Model-Based Format-Transforming Encryption
Bauer, Luke A., Howes, James K. IV, Markelon, Sam A., Bindschaedler, Vincent, Shrimpton, Thomas
We introduce a new type of format-transforming encryption where the format of ciphertexts is implicitly encoded within a machine-learned generative model. Around this primitive, we build a system for covert messaging over large, public internet platforms (e.g., Twitter). Loosely, our system composes an authenticated encryption scheme, with a method for encoding random ciphertext bits into samples from the generative model's family of seed-indexed token-distributions. By fixing a deployment scenario, we are forced to consider system-level and algorithmic solutions to real challenges -- ~such as receiver-side parsing ambiguities, and the low information-carrying capacity of actual token-distributions~ -- that were elided in prior work. We use GPT-2 as our generative model so that our system cryptographically transforms plaintext bitstrings into natural-language covertexts suitable for posting to public platforms. We consider adversaries with full view of the internet platform's content, whose goal is to surface posts that are using our system for covert messaging. We carry out a suite of experiments to provide heuristic evidence of security and to explore tradeoffs between operational efficiency and detectability.
UAV-CROWD: Violent and non-violent crowd activity simulator from the perspective of UAV
Rahmun, Mahieyin, Deb, Tonmoay, Bijoy, Shahriar Ali, Raha, Mayamin Hamid
Unmanned Aerial Vehicle (UAV) has gained significant traction in the recent years, particularly the context of surveillance. However, video datasets that capture violent and non-violent human activity from aerial point-of-view is scarce. To address this issue, we propose a novel, baseline simulator which is capable of generating sequences of photo-realistic synthetic images of crowds engaging in various activities that can be categorized as violent or non-violent. The crowd groups are annotated with bounding boxes that are automatically computed using semantic segmentation. Our simulator is capable of generating large, randomized urban environments and is able to maintain an average of 25 frames per second on a mid-range computer with 150 concurrent crowd agents interacting with each other. We also show that when synthetic data from the proposed simulator is augmented with real world data, binary video classification accuracy is improved by 5% on average across two different models.
On the Limitations of Continual Learning for Malware Classification
Rahman, Mohammad Saidur, Coull, Scott E., Wright, Matthew
Malicious software (malware) classification offers a unique challenge for continual learning (CL) regimes due to the volume of new samples received on a daily basis and the evolution of malware to exploit new vulnerabilities. On a typical day, antivirus vendors receive hundreds of thousands of unique pieces of software, both malicious and benign, and over the course of the lifetime of a malware classifier, more than a billion samples can easily accumulate. Given the scale of the problem, sequential training using continual learning techniques could provide substantial benefits in reducing training and storage overhead. To date, however, there has been no exploration of CL applied to malware classification tasks. In this paper, we study 11 CL techniques applied to three malware tasks covering common incremental learning scenarios, including task, class, and domain incremental learning (IL). Specifically, using two realistic, large-scale malware datasets, we evaluate the performance of the CL methods on both binary malware classification (Domain-IL) and multi-class malware family classification (Task-IL and Class-IL) tasks. To our surprise, continual learning methods significantly underperformed naive Joint replay of the training data in nearly all settings -- in some cases reducing accuracy by more than 70 percentage points. A simple approach of selectively replaying 20% of the stored data achieves better performance, with 50% of the training time compared to Joint replay. Finally, we discuss potential reasons for the unexpectedly poor performance of the CL techniques, with the hope that it spurs further research on developing techniques that are more effective in the malware classification domain.
NURD: Negative-Unlabeled Learning for Online Datacenter Straggler Prediction
Ding, Yi, Rao, Avinash, Song, Hyebin, Willett, Rebecca, Hoffmann, Henry
Datacenters execute large computational jobs, which are composed of smaller tasks. A job completes when all its tasks finish, so stragglers -- rare, yet extremely slow tasks -- are a major impediment to datacenter performance. Accurately predicting stragglers would enable proactive intervention, allowing datacenter operators to mitigate stragglers before they delay a job. While much prior work applies machine learning to predict computer system performance, these approaches rely on complete labels -- i.e., sufficient examples of all possible behaviors, including straggling and non-straggling -- or strong assumptions about the underlying latency distributions -- e.g., whether Gaussian or not. Within a running job, however, none of this information is available until stragglers have revealed themselves when they have already delayed the job. To predict stragglers accurately and early without labeled positive examples or assumptions on latency distributions, this paper presents NURD, a novel Negative-Unlabeled learning approach with Reweighting and Distribution-compensation that only trains on negative and unlabeled streaming data. The key idea is to train a predictor using finished tasks of non-stragglers to predict latency for unlabeled running tasks, and then reweight each unlabeled task's prediction based on a weighting function of its feature space. We evaluate NURD on two production traces from Google and Alibaba, and find that compared to the best baseline approach, NURD produces 2--11 percentage point increases in the F1 score in terms of prediction accuracy, and 2.0--8.8 percentage point improvements in job completion time.
Succinct Differentiation of Disparate Boosting Ensemble Learning Methods for Prognostication of Polycystic Ovary Syndrome Diagnosis
Gupta, Abhishek, Shetty, Sannidhi, Joshi, Raunak, Laban, Ronald Melwin
The most common gynecological disorder affecting women globally is known as polycystic ovary syndrome (PCOS). The symptoms of PCOS include irregular periods, hirsutism, thinning hair and hair loss over head, oily skin or acne and weight gain. PCOS can lead to risk in later life with a lifelong situation that causes a person's blood sugar levels to promote type-II diabetes. High blood pressure and high cholesterol which can lead to heart stroke, overweight ladies may expand sleep apnoea, a situation that causes interrupted breathing at some stage in sleep. Around 10 - 15% of reproductive age (15 to 49 years) of women suffer from this. The monetary expenses of this disease and its comorbidities need the development of instruments and techniques so one can permit for early and precise identification. To cope with this problem this paper proposes a system for the early detection and prediction of PCOS from the most reliable and minimal and promising scientific and metabolic parameters, which is early detection for these diseases. Machine Learning[Shinde and Shah, 2018] can be leveraged to perform prognostication of PCOS that exigently extracts factual records from the given statistics considering the fact that machine learning is better known as glorified statistics. A specific type of machine learning algorithm that seeks to improve the overall performance by combining the predictions from more than one model which is a trendy meta method is known as an Ensemble Learning Approach.
SNGuess: A method for the selection of young extragalactic transients
Miranda, N., Freytag, J. C., Nordin, J., Biswas, R., Brinnel, V., Fremling, C., Kowalski, M., Mahabal, A., Reusch, S., van Santen, J.
With a rapidly rising number of transients detected in astronomy, classification methods based on machine learning are increasingly being employed. Their goals are typically to obtain a definitive classification of transients, and for good performance they usually require the presence of a large set of observations. However, well-designed, targeted models can reach their classification goals with fewer computing resources. This paper presents SNGuess, a model designed to find young extragalactic nearby transients with high purity. SNGuess works with a set of features that can be efficiently calculated from astronomical alert data. Some of these features are static and associated with the alert metadata, while others must be calculated from the photometric observations contained in the alert. Most of the features are simple enough to be obtained or to be calculated already at the early stages in the lifetime of a transient after its detection. We calculate these features for a set of labeled public alert data obtained over a time span of 15 months from the Zwicky Transient Facility (ZTF). The core model of SNGuess consists of an ensemble of decision trees, which are trained via gradient boosting. Approximately 88% of the candidates suggested by SNGuess from a set of alerts from ZTF spanning from April 2020 to August 2021 were found to be true relevant supernovae (SNe). For alerts with bright detections, this number ranges between 92% and 98%. Since April 2020, transients identified by SNGuess as potential young SNe in the ZTF alert stream are being published to the Transient Name Server (TNS) under the AMPEL_ZTF_NEW group identifier. SNGuess scores for any transient observed by ZTF can be accessed via a web service. The source code of SNGuess is publicly available.
Gradient Estimation for Binary Latent Variables via Gradient Variance Clipping
Kunes, Russell Z., Yin, Mingzhang, Land, Max, Haviv, Doron, Pe'er, Dana, Tavaré, Simon
Gradient estimation is often necessary for fitting generative models with discrete latent variables, in contexts such as reinforcement learning and variational autoencoder (VAE) training. The DisARM estimator (Yin et al. 2020; Dong, Mnih, and Tucker 2020) achieves state of the art gradient variance for Bernoulli latent variable models in many contexts. However, DisARM and other estimators have potentially exploding variance near the boundary of the parameter space, where solutions tend to lie. To ameliorate this issue, we propose a new gradient estimator \textit{bitflip}-1 that has lower variance at the boundaries of the parameter space. As bitflip-1 has complementary properties to existing estimators, we introduce an aggregated estimator, \textit{unbiased gradient variance clipping} (UGC) that uses either a bitflip-1 or a DisARM gradient update for each coordinate. We theoretically prove that UGC has uniformly lower variance than DisARM. Empirically, we observe that UGC achieves the optimal value of the optimization objectives in toy experiments, discrete VAE training, and in a best subset selection problem.