Using information-theoretic principles, we consider the generalization error (gen-error) of iterative semi-supervised learning (SSL) algorithms that iteratively generate pseudo-labels for a large amount of unlabelled data to progressively refine the model parameters. In contrast to most previous works that {\em bound} the gen-error, we provide an {\em exact} expression for the gen-error and particularize it to the binary Gaussian mixture model. Our theoretical results suggest that when the class conditional variances are not too large, the gen-error decreases with the number of iterations, but quickly saturates. On the flip side, if the class conditional variances (and so amount of overlap between the classes) are large, the gen-error increases with the number of iterations. To mitigate this undesirable effect, we show that regularization can reduce the gen-error. The theoretical results are corroborated by extensive experiments on the MNIST and CIFAR datasets in which we notice that for easy-to-distinguish classes, the gen-error improves after several pseudo-labelling iterations, but saturates afterwards, and for more difficult-to-distinguish classes, regularization improves the generalization performance.

Predictions and forecasts of machine learning models should take the form of probability distributions, aiming to increase the quantity of information communicated to end users. Although applications of probabilistic prediction and forecasting with machine learning models in academia and industry are becoming more frequent, related concepts and methods have not been formalized and structured under a holistic view of the entire field. Here, we review the topic of predictive uncertainty estimation with machine learning algorithms, as well as the related metrics (consistent scoring functions and proper scoring rules) for assessing probabilistic predictions. The review covers a time period spanning from the introduction of early statistical (linear regression and time series models, based on Bayesian statistics or quantile regression) to recent machine learning algorithms (including generalized additive models for location, scale and shape, random forests, boosting and deep learning algorithms) that are more flexible by nature. The review of the progress in the field, expedites our understanding on how to develop new algorithms tailored to users' needs, since the latest advancements are based on some fundamental concepts applied to more complex algorithms. We conclude by classifying the material and discussing challenges that are becoming a hot topic of research.

The random Fourier features (RFFs) method is a powerful and popular technique in kernel approximation for scalability of kernel methods. The theoretical foundation of RFFs is based on the Bochner theorem that relates symmetric, positive definite (PD) functions to probability measures. This condition naturally excludes asymmetric functions with a wide range applications in practice, e.g., directed graphs, conditional probability, and asymmetric kernels. Nevertheless, understanding asymmetric functions (kernels) and its scalability via RFFs is unclear both theoretically and empirically. In this paper, we introduce a complex measure with the real and imaginary parts corresponding to four finite positive measures, which expands the application scope of the Bochner theorem. By doing so, this framework allows for handling classical symmetric, PD kernels via one positive measure; symmetric, non-positive definite kernels via signed measures; and asymmetric kernels via complex measures, thereby unifying them into a general framework by RFFs, named AsK-RFFs. Such approximation scheme via complex measures enjoys theoretical guarantees in the perspective of the uniform convergence. In algorithmic implementation, to speed up the kernel approximation process, which is expensive due to the calculation of total mass, we employ a subset-based fast estimation method that optimizes total masses on a sub-training set, which enjoys computational efficiency in high dimensions. Our AsK-RFFs method is empirically validated on several typical large-scale datasets and achieves promising kernel approximation performance, which demonstrate the effectiveness of AsK-RFFs.

With the wide spread of sensors and smart devices in recent years, the data generation speed of the Internet of Things (IoT) systems has increased dramatically. In IoT systems, massive volumes of data must be processed, transformed, and analyzed on a frequent basis to enable various IoT services and functionalities. Machine Learning (ML) approaches have shown their capacity for IoT data analytics. However, applying ML models to IoT data analytics tasks still faces many difficulties and challenges, specifically, effective model selection, design/tuning, and updating, which have brought massive demand for experienced data scientists. Additionally, the dynamic nature of IoT data may introduce concept drift issues, causing model performance degradation. To reduce human efforts, Automated Machine Learning (AutoML) has become a popular field that aims to automatically select, construct, tune, and update machine learning models to achieve the best performance on specified tasks. In this paper, we conduct a review of existing methods in the model selection, tuning, and updating procedures in the area of AutoML in order to identify and summarize the optimal solutions for every step of applying ML algorithms to IoT data analytics. To justify our findings and help industrial users and researchers better implement AutoML approaches, a case study of applying AutoML to IoT anomaly detection problems is conducted in this work. Lastly, we discuss and classify the challenges and research directions for this domain.

Research on human reading has long documented that reading behavior shows task-specific effects, but it has been challenging to build general models predicting what reading behavior humans will show in a given task. We introduce NEAT, a computational model of the allocation of attention in human reading, based on the hypothesis that human reading optimizes a tradeoff between economy of attention and success at a task. Our model is implemented using contemporary neural network modeling techniques, and makes explicit and testable predictions about how the allocation of attention varies across different tasks. We test this in an eyetracking study comparing two versions of a reading comprehension task, finding that our model successfully accounts for reading behavior across the tasks. Our work thus provides evidence that task effects can be modeled as optimal adaptation to task demands.

Since the development of writing 5000 years ago, human-generated data gets produced at an ever-increasing pace. Classical archival methods aimed at easing information retrieval. Nowadays, archiving is not enough anymore. The amount of data that gets generated daily is beyond human comprehension, and appeals for new information retrieval strategies. Instead of referencing every single data piece as in traditional archival techniques, a more relevant approach consists in understanding the overall ideas conveyed in data flows. To spot such general tendencies, a precise comprehension of the underlying data generation mechanisms is required. In the rich literature tackling this problem, the question of information interaction remains nearly unexplored. First, we investigate the frequency of such interactions. Building on recent advances made in Stochastic Block Modelling, we explore the role of interactions in several social networks. We find that interactions are rare in these datasets. Then, we wonder how interactions evolve over time. Earlier data pieces should not have an everlasting influence on ulterior data generation mechanisms. We model this using dynamic network inference advances. We conclude that interactions are brief. Finally, we design a framework that jointly models rare and brief interactions based on Dirichlet-Hawkes Processes. We argue that this new class of models fits brief and sparse interaction modelling. We conduct a large-scale application on Reddit and find that interactions play a minor role in this dataset. From a broader perspective, our work results in a collection of highly flexible models and in a rethinking of core concepts of machine learning. Consequently, we open a range of novel perspectives both in terms of real-world applications and in terms of technical contributions to machine learning.

Factorizable joint shift (FJS) was recently proposed as a type of dataset shift for which the complete characteristics can be estimated from feature data observations on the test dataset by a method called Joint Importance Aligning. For the multinomial (multiclass) classification setting, we derive a representation of factorizable joint shift in terms of the source (training) distribution, the target (test) prior class probabilities and the target marginal distribution of the features. On the basis of this result, we propose alternatives to joint importance aligning and, at the same time, point out that factorizable joint shift is not fully identifiable if no class label information on the test dataset is available and no additional assumptions are made. Other results of the paper include correction formulae for the posterior class probabilities both under general dataset shift and factorizable joint shift. In addition, we investigate the consequences of assuming factorizable joint shift for the bias caused by sample selection.

This is part one of our three part series Part 1: API as a package: Structure (this post) Part 2: API as a package: Logging (to be published) Part 3: API as a package: Testing (to be published) Introduction At Jumping Rivers we were recently tasked with taking a prototype application built in {shiny} to a public facing production environment for a public sector organisation. During the scoping exercise it was determined that a more appropriate solution to fit the requirements was to build the application with a {plumber} API providing the interface to the Bayesian network model and other application tools written in R. When building applications in {shiny} we have for some time been using the “app as a package” approach which has been popularised by tools like {golem} and {leprechaun}, in large part due to the convenience that comes with leveraging the testing and dependency structure that our R developers are comfortable with in authoring packages, and the ease with which one can install and run an application in a new environment as a result. For this project we looked to take some of these ideas to a {plumber} application. This blog post discusses some of the thoughts and resultant structure that came as a result of that process. As I began to flesh out this blog post I realised that it was becoming very long, and there were a number of different aspects that I wanted to discuss: structure, logging and testing to name a few. To try to keep this a bit more palatable I will instead do a mini-series of blog posts around the API as a package idea and focus predominantly on the structure elements here. Do you use RStudio Pro? If so, checkout out our managed RStudio services API as a package There are a few things I really like about the {shiny} app as a package approach that I wanted to reflect in the design and build of a {plumber} application as package. It encourages a regular structure and organisation for an application. All modules have a consistent naming pattern and structure. It encourages leveraging the {testthat} package and including some common tests across a series of applications, see golem::use_reccommended_tests() for example. An instance of the app can be created via a single function call which does all the necessary set up, say my_package::run_app() Primarily I wanted these features, which could be reused across {plumber} applications that we create both internally and for our clients. As far as I know there isn’t a similar package that provides an opinionated way of laying out a {plumber} application as a package, and it is my intention to create one as a follow up to this work. Regular structure When developing the solution for this particular project I did have in the back of my mind that I wanted to create as much reusable structure for any future projects of this sort as possible. I really wanted to have an easy way to, from a package structure, be able to build out an API with nested routes, using code that could easily transfer to another package. I opted for a structure that utilised the inst/extdata/api/routes directory of a package as a basis with the idea that the following file structure | inst/extdata/api/routes/ | | - model.R | - reports/ - | | - pdf.R with example route definitions inside # model.R #* @post /prediction exported_function_from_my_package # pdf.R #* @post /weekly exported_function_from_my_package would translate to an API with the following endpoints /model/prediction /reports/pdf/weekly A few simple function definitions would allow us to do this for any given package that uses this file structure. The first function here just grabs the directory from the current package where I will define the endpoints that make up my API. get_internal_routes = function(path = ".") { system.file("extdata", "api", "routes", path, package = utils::packageName(), mustWork = TRUE) } create_routes will recursively list out all of the .R files within the chosen directory and name them according to the name of the file, this will make it easy to build out a a number of “nested” routers that will all be mounted into the same API, achieving the compartmentalisation that we desire. For example the two files at /inst/extdata/api/routes/model.R and /inst/extdata/api/routes/reports/pdf.R will take on the names "model" and "reports/pdf" respectively. add_default_route_names = function(routes, dir) { names = stringr::str_remove(routes, pattern = dir) names = stringr::str_remove(names, pattern = "\.R$") names(routes) = names routes } create_routes = function(dir) { routes = list.files( dir, recursive = TRUE, full.names = TRUE, pattern = "*\.R$" ) add_default_route_names(routes, dir) } The final few pieces to the puzzle ensure that we have / at the beginning of a string (ensure_slash()), for the purpose of mounting components to my router. add_plumber_definition() just calls the necessary functions from {plumber} to process a new route file, i.e from the decorated functions in the file create the routes, and then mount them at a given path to an existing router object. For example given a file “test.R” that has a #* @get /identity decorator against a function definition and endpoint = "test" we would add /test/identity to the existing router. generate_api() takes a full named vector/list of file paths, ensures they all have an appropriate name and mounts them all to a new Plumber router object. ensure_slash = function(string) { has_slash = grepl("^/", string) if (has_slash) string else paste0("/", string) } add_plumber_definition = function(pr, endpoint, file, ...) { router = plumber::pr(file = file, ...) plumber::pr_mount(pr = pr, path = endpoint, router = router ) } generate_api = function(routes, ...) { endpoints = purrr::map_chr(names(routes), ensure_slash) purrr::reduce2( .x = endpoints, .y = routes, .f = add_plumber_definition, ..., .init = plumber::pr(NULL) ) } With these defined I can then, as I develop my package, add new routes by defining functions and adding {plumber} tag annotations to files in /inst/ and rebuild the new API with get_internal_routes() %>% create_routes() %>% generate_api() and nothing about this code is specific to my current package so is transferable. As a concrete, but very much simplified example, I might have the following collection of files/annotations under /inst/extdata/api/routes ## File: /example.R # Taken from plumber quickstart documentation # https://www.rplumber.io/articles/quickstart.html #* @get /echo function(msg="") { list(msg = paste0("The message is: '", msg, "'")) } ## File: /test.R #* @get /is_alive function() { list(alive = TRUE) } ## File: /nested/example.R # Taken from plumber quickstart documentation # https://www.rplumber.io/articles/quickstart.html #* @get /echo function(msg="") { list(msg = paste0("The message is: '", msg, "'")) } which would give me get_internal_routes() %>% create_routes() %>% generate_api() # # Plumber router with 0 endpoints, 4 filters, and 3 sub-routers. # # Use `pr_run()` on this object to start the API. # ├──[queryString] # ├──[body] # ├──[cookieParser] # ├──[sharedSecret] # ├──/example # │ │ # Plumber router with 1 endpoint, 4 filters, and 0 sub-routers. # │ ├──[queryString] # │ ├──[body] # │ ├──[cookieParser] # │ ├──[sharedSecret] # │ └──/echo (GET) # ├──/nested # │ ├──/example # │ │ │ # Plumber router with 1 endpoint, 4 filters, and 0 sub-routers. # │ │ ├──[queryString] # │ │ ├──[body] # │ │ ├──[cookieParser] # │ │ ├──[sharedSecret] # │ │ └──/echo (GET) # ├──/test # │ │ # Plumber router with 1 endpoint, 4 filters, and 0 sub-routers. # │ ├──[queryString] # │ ├──[body] # │ ├──[cookieParser] # │ ├──[sharedSecret] # │ └──/is_alive (GET) This {cookieCutter} example is available to view at our Github blog repo. Basic testing In my real project I refrained from having any actual function definitions being made in inst/. Instead each function that was part of the exposed API was a proper exported function from my package (additionally filenames for said functions followed a regular structure too of api_.R). This allows for leveraging {testthat} against the logic of each of the functions as well as using other tools like {lintr} and ensuring that dependencies, documentation etc are all dealt with appropriately. Testing individual functions that will be exposed as routes can be a little different to other R functions in that the objects passed as arguments come from a request. As alluded to in the introduction I will prepare another blog post detailing some elements of testing for API as a package but a short snippet that I found particularly helpful for testing that a running API is functioning as I expect is included here. The following code could be used to set up (and subsequently tear down) a running API that is expecting requests for a package cookieCutter # tests/testthat/setup.R ## run before any tests # pick a random available port to serve your app locally port = httpuv::randomPort() # start a background R process that launches an instance of the API # serving on that random port running_api = callr::r_bg( function(port) { dir = cookieCutter::get_internal_routes() routes = cookieCutter::create_routes(dir) api = cookieCutter::generate_api(routes) api$run(port = port, host = "0.0.0.0") }, list(port = port) ) # Small wait for the background process to ensure it # starts properly Sys.sleep(1) ## run after all tests withr::defer(running_api$kill(), testthat::teardown_env()) A simple test to ensure that our is_alive endpoint works then might look like test_that("is alive", { res = httr::GET(glue::glue("http://0.0.0.0:{port}/test/is_alive")) expect_equal(res$status_code, 200) }) Logging {shiny} has some useful packages for adding logging, in particular {shinylogger} is very helpful at giving you plenty of logging for little effort on my part as the user. As far as I could find nothing similar exists for {plumber} so I set up a bunch of hooks, using the {logger} package to write information to both file and terminal. Since that could form it’s own blogpost I will save that discussion for the future. For updates and revisions to this article, see the original post

Provably efficient Model-Based Reinforcement Learning (MBRL) based on optimism or posterior sampling (PSRL) is ensured to attain the global optimality asymptotically by introducing the complexity measure of the model. However, the complexity might grow exponentially for the simplest nonlinear models, where global convergence is impossible within finite iterations. When the model suffers a large generalization error, which is quantitatively measured by the model complexity, the uncertainty can be large. The sampled model that current policy is greedily optimized upon will thus be unsettled, resulting in aggressive policy updates and over-exploration. In this work, we propose Conservative Dual Policy Optimization (CDPO) that involves a Referential Update and a Conservative Update. The policy is first optimized under a reference model, which imitates the mechanism of PSRL while offering more stability. A conservative range of randomness is guaranteed by maximizing the expectation of model value. Without harmful sampling procedures, CDPO can still achieve the same regret as PSRL. More importantly, CDPO enjoys monotonic policy improvement and global optimality simultaneously.

Data Science and Analytics have Basketball, Soccer, Tennis, and Cricket. In the modern world, optimized almost every domain that exists in the market. In Sports Analytics is found to be used in almost every our survey we tend to focus mainly how the field of organized sport that is played. Today, we have Sports Analytics has been adopted in the field of sports, how it has Analytics put into use in all primary sports right from Team-contributed to the transformation of the game right from the Selection and On-ground Decision making to business assessment of on-field players and their selection to aspects of the sport. The development of this domain had its prediction of winning team and to the marketing of tickets roots primarily from Statistics, Game Theory, and Decision and business aspects of big sports tournaments. We will Theory, and today, the field also uses Machine Learning and present the analytical tools, algorithms and methodologies Modern Analytical Approaches to decisions on the team and adopted in the field of Sports Analytics for different sports the game itself.