We explore the limitations of and best practices for using black-box variational inference to estimate posterior summaries of the model parameters. By taking an importance sampling perspective, we are able to explain and empirically demonstrate: 1) why the intuitions about the behavior of approximate families and divergences for low-dimensional posteriors fail for higher-dimensional posteriors, 2) how we can diagnose the pre-asymptotic reliability of variational inference in practice by examining the behavior of the density ratios (i.e., importance weights), 3) why the choice of variational objective is not as relevant for higher-dimensional posteriors, and 4) why, although flexible variational families can provide some benefits in higher dimensions, they also introduce additional optimization challenges. Based on these findings, for high-dimensional posteriors we recommend using the exclusive KL divergence that is most stable and easiest to optimize, and then focusing on improving the variational family or using model parameter transformations to make the posterior more similar to the approximating family. Our results also show that in low to moderate dimensions, heavy-tailed variational families and mass-covering divergences can increase the chances that the approximation can be improved by importance sampling.

Existing deterministic variational inference approaches for diffusion processes use simple proposals and target the marginal density of the posterior. We construct the variational process as a controlled version of the prior process and approximate the posterior by a set of moment functions. In combination with moment closure, the smoothing problem is reduced to a deterministic optimal control problem. Exploiting the path-wise Fisher information, we propose an optimization procedure that corresponds to a natural gradient descent in the variational parameters. Our approach allows for richer variational approximations that extend to state-dependent diffusion terms. The classical Gaussian process approximation is recovered as a special case.

The rapid increase of the data scale in Internet of Vehicles (IoV) system paradigm, hews out new possibilities in boosting the service quality for the emerging applications through data sharing. Nevertheless, privacy concerns are major bottlenecks for data providers to share private data in traditional IoV networks. To this end, federated learning (FL) as an emerging learning paradigm, where data providers only send local model updates trained on their local raw data rather than upload any raw data, has been recently proposed to build a privacy-preserving data sharing models. Unfortunately, by analyzing on the differences of uploaded local model updates from data providers, private information can still be divulged, and performance of the system cannot be guaranteed when partial federated nodes executes malicious behavior. Additionally, traditional cloud-based FL poses challenges to the communication overhead with the rapid increase of terminal equipment in IoV system. All these issues inspire us to propose an autonomous blockchain empowered privacy-preserving FL framework in this paper, where the mobile edge computing (MEC) technology was naturally integrated in IoV system.

The question of whether to use one classifier or a combination of classifiers is a central topic in Machine Learning. We propose here a method for finding an optimal linear combination of classifiers derived from a bias-variance framework for the classification task.

One of the increasingly important technologies dealing with the growing complexity of the digitalization of almost all human activities is Artificial intelligence, more precisely machine learning Despite the fact, that we live in a Big data world where almost everything is digitally stored, there are many real-world situations, where researchers are faced with small data samples. The present study aim is to answer the following research question namely What is the small data problem in machine learning and how it is solved?. Our bibliometric study showed a positive trend in the number of research publications concerning the use of small datasets and substantial growth of the research community dealing with the small dataset problem, indicating that the research field is moving toward higher maturity levels. Despite notable international cooperation, the regional concentration of research literature production in economically more developed countries was observed.

We present a novel counterfactual framework for both Zero-Shot Learning (ZSL) and Open-Set Recognition (OSR), whose common challenge is generalizing to the unseen-classes by only training on the seen-classes. Our idea stems from the observation that the generated samples for unseen-classes are often out of the true distribution, which causes severe recognition rate imbalance between the seen-class (high) and unseen-class (low). We show that the key reason is that the generation is not Counterfactual Faithful, and thus we propose a faithful one, whose generation is from the sample-specific counterfactual question: What would the sample look like, if we set its class attribute to a certain class, while keeping its sample attribute unchanged? Thanks to the faithfulness, we can apply the Consistency Rule to perform unseen/seen binary classification, by asking: Would its counterfactual still look like itself? If ``yes'', the sample is from a certain class, and ``no'' otherwise. Through extensive experiments on ZSL and OSR, we demonstrate that our framework effectively mitigates the seen/unseen imbalance and hence significantly improves the overall performance. Note that this framework is orthogonal to existing methods, thus, it can serve as a new baseline to evaluate how ZSL/OSR models generalize. Codes are available at https://github.com/yue-zhongqi/gcm-cf.

Despite many proposed algorithms to provide robustness to deep learning (DL) models, DL models remain susceptible to adversarial attacks. We hypothesize that the adversarial vulnerability of DL models stems from two factors. The first factor is data sparsity which is that in the high dimensional data space, there are large regions outside the support of the data distribution. The second factor is the existence of many redundant parameters in the DL models. Owing to these factors, different models are able to come up with different decision boundaries with comparably high prediction accuracy. The appearance of the decision boundaries in the space outside the support of the data distribution does not affect the prediction accuracy of the model. However, they make an important difference in the adversarial robustness of the model. We propose that the ideal decision boundary should be as far as possible from the support of the data distribution.\par In this paper, we develop a training framework for DL models to learn such decision boundaries spanning the space around the class distributions further from the data points themselves. Semi-supervised learning was deployed to achieve this objective by leveraging unlabeled data generated in the space outside the support of the data distribution. We measure adversarial robustness of the models trained using this training framework against well-known adversarial attacks We found that our results, other regularization methods and adversarial training also support our hypothesis of data sparcity. We show that the unlabeled data generated by noise using our framework is almost as effective as unlabeled data, sourced from existing data sets or generated by synthesis algorithms, on adversarial robustness. Our code is available at https://github.com/MahsaPaknezhad/AdversariallyRobustTraining.

Obtaining historical data on the stocks that we want to observe is a two-step process. The library get-all-tickers allows us to compile a list of stock tickers by filtering companies on aspects like market cap or exchange. For this example, I am looking at companies that have a market cap between $150,000 and $10,000,000 (in millions). You will notice that I also included a line of code to print the number of tickers we are using. You will need to be sure that you are not targeting more than 2,000 tickers, because the Yfinance API has a 2,000 API calls per hour limit.

There are so many machine learning algorithms out there, how do you choose the best one for your problem? This question is going to have a different response based on the application and the data. Is it classification, regression, supervised, unsupervised, natural language processing, time series? There are so many avenues to take but in this article I am going to focus on on algorithm that I particularly find very interesting, XGBoost. XGBoost stands for extreme gradient boosting and is an open source library that provides an efficient and effective implementation of gradient boosting.

Ordinary Least Squares is one of the easiest and most widely used ML algorithms. But it suffers from a fatal flaw -- it is super easy for the algorithm to overfit the training data. But as the number of predictor variables (or dimensions) increases, the coefficients β_i also tend to get very large. With large coefficients, it is easy to predict nearly everything -- you just take the relevant combination of individual slopes (βs) and you get the answer. That's why it is common for linear regression models to overfit the training data.