Stochastic Gradient Descent (SGD) methods are prominent for training machine learning and deep learning models. The performance of these techniques depends on their hyperparameter tuning over time and varies for different models and problems. Manual adjustment of hyperparameters is very costly and time-consuming, and even if done correctly, it lacks theoretical justification which inevitably leads to "rule of thumb" settings. In this paper, we propose a generic approach that utilizes the statistics of an unbiased gradient estimator to automatically and simultaneously adjust two paramount hyperparameters: the learning rate and momentum. We deploy the proposed general technique for various SGD methods to train Convolutional Neural Networks (CNN's). The results match the performance of the best settings obtained through an exhaustive search and therefore, removes the need for a tedious manual tuning.

In-bed pose estimation has shown value in fields such as hospital patient monitoring, sleep studies, and smart homes. In this paper, we present a novel in-bed pressure-based pose estimation approach capable of accurately detecting body parts from highly ambiguous pressure data. We exploit the idea of using a learnable pre-processing step, which transforms the vague pressure maps to a representation close to the expected input space of common purpose pose identification modules, which fail if solely used on the pressure data. To this end, a fully convolutional network with multiple scales is used as the learnable pre-processing step to provide the pose-specific characteristics of the pressure maps to the pre-trained pose identification module. A combination of loss functions is used to model the constraints, ensuring that unclear body parts are reconstructed correctly while preventing the pre-processing block from generating arbitrary images. The evaluation results show high visual fidelity in the generated pre-processed images as well as high detection rates in pose estimation. Furthermore, we show that the trained pre-processing block can be effective for pose identification models for which it has not been trained as well.

Improving a semi-supervised image segmentation task has the option of adding more unlabelled images, labelling the unlabelled images or combining both, as neither image acquisition nor expert labelling can be considered trivial in most clinical applications. With a laparoscopic liver image segmentation application, we investigate the performance impact by altering the quantities of labelled and unlabelled training data, using a semi-supervised segmentation algorithm based on the mean teacher learning paradigm. We first report a significantly higher segmentation accuracy, compared with supervised learning. Interestingly, this comparison reveals that the training strategy adopted in the semi-supervised algorithm is also responsible for this observed improvement, in addition to the added unlabelled data. We then compare different combinations of labelled and unlabelled data set sizes for training semi-supervised segmentation networks, to provide a quantitative example of the practically useful trade-off between the two data planning strategies in this surgical guidance application.

--Deep neural networks (DNNs) have demonstrated impressive performance on many challenging machine learning tasks. However, DNNs are vulnerable to adversarial inputs generated by adding maliciously crafted perturbations to the benign inputs. As a growing number of attacks have been reported to generate adversarial inputs of varying sophistication, the defense-attack arms race has been accelerated. MODEF intelligently combines unsupervised model denoising ensemble with supervised model verification ensemble by quantifying model diversity, aiming to boost the robustness of the target model against adversarial examples. Evaluated using eleven representative attacks on popular benchmark datasets, we show that MODEF achieves remarkable defense success rates, compared with existing defense methods, and provides a superior capability of repairing adversarial inputs and making correct predictions with high accuracy in the presence of black-box attacks. The recent advances in deep neural networks (DNNs) have powered numerous applications in different domains due to their outstanding performance compared to traditional machine learning techniques. However, it has been shown that DNNs can be easily fooled by adversarial inputs [1], making them become a double-edged sword as the vulnerability of DNNs to adversarial attacks has posed serious threats to many security-critical applications, such as biometric authentication and autonomous driving. As a number of defenses are being proposed, more attacks of varying sophistication have been put forward, accelerating the defense-attack arms race. Some even argue that designing new attacks requires much less efforts than developing effective defenses. Thus, improving the robustness and defensibility against adversarial attacks is crucial. Adversarial examples are generated by maliciously perturbing benign examples sent to the target DNN model through querying its prediction API, aiming to fool and mislead the target model to misclassify by producing incorrect predictions randomly (untargeted attack) or purposefully (targeted attack).

Although deep convolutional neural networks achieve state-of-the-art performance across nearly all image classification tasks, they are often regarded as black boxes. Because they compute a nonlinear function of the entire input image, their decisions are difficult to interpret. One approach that offers some level of interpretability by design is \textit{hard attention}, which selects only relevant portions of the image. However, training hard attention models with only class label supervision is challenging, and hard attention has proved difficult to scale to complex datasets. Here, we propose a novel hard attention model, which we term Saccader, as well as a self-supervised pretraining procedure for this model that does not suffer from optimization challenges. Through pretraining and policy gradient optimization, the Saccader model estimates the relevance of different image patches to the downstream task, and uses a novel cell to select patches to classify at different times. Our approach achieves high accuracy on ImageNet while providing more interpretable predictions.

Privacy preserving machine learning algorithms are crucial for learning models over user data to protect sensitive information. Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed. At each step, these algorithms modify the gradients and add noise proportional to the sensitivity of the modified gradients. Under this framework, we propose AdaCliP, a theoretically motivated differentially private SGD algorithm that provably adds less noise compared to the previous methods, by using coordinate-wise adaptive clipping of the gradient. We empirically demonstrate that AdaCliP reduces the amount of added noise and produces models with better accuracy.

Graph neural networks (GNNs) are widely used in many applications. However, their robustness against adversarial attacks is criticized. Prior studies show that using unnoticeable modifications on graph topology or nodal features can significantly reduce the performances of GNNs. It is very challenging to design robust graph neural networks against poisoning attack and several efforts have been taken. Existing work aims at reducing the negative impact from adversarial edges only with the poisoned graph, which is sub-optimal since they fail to discriminate adversarial edges from normal ones. On the other hand, clean graphs from similar domains as the target poisoned graph are usually available in the real world. By perturbing these clean graphs, we create supervised knowledge to train the ability to detect adversarial edges so that the robustness of GNNs is elevated. However, such potential for clean graphs is neglected by existing work. To this end, we investigate a novel problem of improving the robustness of GNNs against poisoning attacks by exploring clean graphs. Specifically, we propose PA-GNN, which relies on a penalized aggregation mechanism that directly restrict the negative impact of adversarial edges by assigning them lower attention coefficients. To optimize PA-GNN for a poisoned graph, we design a meta-optimization algorithm that trains PA-GNN to penalize perturbations using clean graphs and their adversarial counterparts, and transfers such ability to improve the robustness of PA-GNN on the poisoned graph. Experimental results on four real-world datasets demonstrate the robustness of PA-GNN against poisoning attacks on graphs.

Circadian rhythms govern most essential biological processes in the human body; they influence multiple biological activities including sleep, performance, mood, skin temperature, hormone production, and nutrient absorption. The dim light melatonin onset (DLMO) is the current gold standard for measuring human circadian phase (or timing). The collection of DLMO is expensive and time consuming since multiple saliva or blood samples are required from nighttime studies in specialized environmental conditions. In the past few years, several non-invasive approaches have been designed for estimating DLMO values. These methods collect daily sampled data (e.g., sleep onset/offset times) or frequently sampled data (e.g., light exposure, skin temperature, physical activity collected every minute) to train learning models for estimating DLMO, therefore previous studies only leveraged one time scale. In this paper, we propose a two-step framework for estimating DLMO using the data of both time scales. The first step summarizes the data prior to the current day, while the second step combines this summary with frequently sampled data of the current day. We evaluate several variants of moving average model which input sleep timing data as the first step and recurrent neural network models as the second step for estimating DLMO. The experimental results show that our two-step model with two-time-scale features has statistically significantly lower root-mean-square errors than the models that use either daily sampled data or frequently sampled data alone.

We propose a novel high-performance interpretable deep tabular data learning network, TabNet. TabNet utilizes a sequential attention mechanism to choose which features to reason from at each decision step and then aggregates the processed information towards the final decision. Explicit selection of sparse features enables more efficient learning as the model capacity at each decision step is fully utilized for the most relevant features, and also more interpretable decision making via visualization of selection masks. We demonstrate that TabNet outperforms other neural network and decision tree variants on a wide range of tabular data learning datasets while yielding interpretable feature attributions and insights into the global model behavior.

-- As deep learning applications are becoming more and more pervasive in robotics, the question of evaluating the reliability of inferences becomes a central question in the robotics community. This domain, known as predictive uncertainty, has come under the scrutiny of research groups developing Bayesian approaches adapted to deep learning such as Monte Carlo Dropout. Unfortunately, for the time being, the real goal of predictive uncertainty has been swept under the rug. Indeed, these approaches are solely evaluated in terms of raw performance of the network prediction, while the quality of their estimated uncertainty is not assessed. Evaluating such uncertainty prediction quality is especially important in robotics, as actions shall depend on the confidence in perceived information. In this context, the main contribution of this article is to propose a novel metric that is adapted to the evaluation of relative uncertainty assessment and directly applicable to regression with deep neural networks. T o experimentally validate this metric, we evaluate it on a toy dataset and then apply it to the task of monocular depth estimation. The outcome of deep learning these past few years - capable of reaching and even sometimes surpass human performances, e.g. for vision tasks [1], [2] - offers novel avenues in robotics, but also raises some novel questions: as pointed out by [3], how much trust can we put in the predictions of a deep learning system?