Most real-world problems that machine learning algorithms are expected to solve face the situation with 1) unknown data distribution; 2) little domain-specific knowledge; and 3) datasets with limited annotation. We propose Non-Parametric learning by Compression with Latent Variables (NPC-LV), a learning framework for any dataset with abundant unlabeled data but very few labeled ones. By only training a generative model in an unsupervised way, the framework utilizes the data distribution to build a compressor. Using a compressor-based distance metric derived from Kolmogorov complexity, together with few labeled data, NPC-LV classifies without further training. We show that NPC-LV outperforms supervised methods on all three datasets on image classification in low data regime and even outperform semi-supervised learning methods on CIFAR-10. We demonstrate how and when negative evidence lowerbound (nELBO) can be used as an approximate compressed length for classification. By revealing the correlation between compression rate and classification accuracy, we illustrate that under NPC-LV, the improvement of generative models can enhance downstream classification accuracy.

Privacy has become a major concern in machine learning. In fact, the federated learning is motivated by the privacy concern as it does not allow to transmit the private data but only intermediate updates. However, federated learning does not always guarantee privacy-preservation as the intermediate updates may also reveal sensitive information. In this paper, we give an explicit information-theoretical analysis of a federated expectation maximization algorithm for Gaussian mixture model and prove that the intermediate updates can cause severe privacy leakage. To address the privacy issue, we propose a fully decentralized privacy-preserving solution, which is able to securely compute the updates in each maximization step. Additionally, we consider two different types of security attacks: the honest-but-curious and eavesdropping adversary models. Numerical validation shows that the proposed approach has superior performance compared to the existing approach in terms of both the accuracy and privacy level.

Research on human reading has long documented that reading behavior shows task-specific effects, but it has been challenging to build general models predicting what reading behavior humans will show in a given task. We introduce NEAT, a computational model of the allocation of attention in human reading, based on the hypothesis that human reading optimizes a tradeoff between economy of attention and success at a task. Our model is implemented using contemporary neural network modeling techniques, and makes explicit and testable predictions about how the allocation of attention varies across different tasks. We test this in an eyetracking study comparing two versions of a reading comprehension task, finding that our model successfully accounts for reading behavior across the tasks. Our work thus provides evidence that task effects can be modeled as optimal adaptation to task demands.

Since the development of writing 5000 years ago, human-generated data gets produced at an ever-increasing pace. Classical archival methods aimed at easing information retrieval. Nowadays, archiving is not enough anymore. The amount of data that gets generated daily is beyond human comprehension, and appeals for new information retrieval strategies. Instead of referencing every single data piece as in traditional archival techniques, a more relevant approach consists in understanding the overall ideas conveyed in data flows. To spot such general tendencies, a precise comprehension of the underlying data generation mechanisms is required. In the rich literature tackling this problem, the question of information interaction remains nearly unexplored. First, we investigate the frequency of such interactions. Building on recent advances made in Stochastic Block Modelling, we explore the role of interactions in several social networks. We find that interactions are rare in these datasets. Then, we wonder how interactions evolve over time. Earlier data pieces should not have an everlasting influence on ulterior data generation mechanisms. We model this using dynamic network inference advances. We conclude that interactions are brief. Finally, we design a framework that jointly models rare and brief interactions based on Dirichlet-Hawkes Processes. We argue that this new class of models fits brief and sparse interaction modelling. We conduct a large-scale application on Reddit and find that interactions play a minor role in this dataset. From a broader perspective, our work results in a collection of highly flexible models and in a rethinking of core concepts of machine learning. Consequently, we open a range of novel perspectives both in terms of real-world applications and in terms of technical contributions to machine learning.

Error correction code (ECC) is an integral part of the physical communication layer, ensuring reliable data transfer over noisy channels. Recently, neural decoders have demonstrated their advantage over classical decoding techniques. However, recent state-of-the-art neural decoders suffer from high complexity and lack the important iterative scheme characteristic of many legacy decoders. In this work, we propose to employ denoising diffusion models for the soft decoding of linear codes at arbitrary block lengths. Our framework models the forward channel corruption as a series of diffusion steps that can be reversed iteratively. Three contributions are made: (i) a diffusion process suitable for the decoding setting is introduced, (ii) the neural diffusion decoder is conditioned on the number of parity errors, which indicates the level of corruption at a given step, (iii) a line search procedure based on the code's syndrome obtains the optimal reverse diffusion step size. The proposed approach demonstrates the power of diffusion models for ECC and is able to achieve state of the art accuracy, outperforming the other neural decoders by sizable margins, even for a single reverse diffusion step.

Factorizable joint shift (FJS) was recently proposed as a type of dataset shift for which the complete characteristics can be estimated from feature data observations on the test dataset by a method called Joint Importance Aligning. For the multinomial (multiclass) classification setting, we derive a representation of factorizable joint shift in terms of the source (training) distribution, the target (test) prior class probabilities and the target marginal distribution of the features. On the basis of this result, we propose alternatives to joint importance aligning and, at the same time, point out that factorizable joint shift is not fully identifiable if no class label information on the test dataset is available and no additional assumptions are made. Other results of the paper include correction formulae for the posterior class probabilities both under general dataset shift and factorizable joint shift. In addition, we investigate the consequences of assuming factorizable joint shift for the bias caused by sample selection.

Recently there is a large amount of work devoted to the study of Markov chain stochastic gradient methods (MC-SGMs) which mainly focus on their convergence analysis for solving minimization problems. In this paper, we provide a comprehensive generalization analysis of MC-SGMs for both minimization and minimax problems through the lens of algorithmic stability in the framework of statistical learning theory. For empirical risk minimization (ERM) problems, we establish the optimal excess population risk bounds for both smooth and non-smooth cases by introducing on-average argument stability. For minimax problems, we develop a quantitative connection between on-average argument stability and generalization error which extends the existing results for uniform stability \cite{lei2021stability}. We further develop the first nearly optimal convergence rates for convex-concave problems both in expectation and with high probability, which, combined with our stability results, show that the optimal generalization bounds can be attained for both smooth and non-smooth cases. To the best of our knowledge, this is the first generalization analysis of SGMs when the gradients are sampled from a Markov process.

Active inference is a first principles approach for understanding the brain in particular, and sentient agents in general, with the single imperative of minimizing free energy. As such, it provides a computational account for modelling artificial intelligent agents, by defining the agent's generative model and inferring the model parameters, actions and hidden state beliefs. However, the exact specification of the generative model and the hidden state space structure is left to the experimenter, whose design choices influence the resulting behaviour of the agent. Recently, deep learning methods have been proposed to learn a hidden state space structure purely from data, alleviating the experimenter from this tedious design task, but resulting in an entangled, non-interpreteable state space. In this paper, we hypothesize that such a learnt, entangled state space does not necessarily yield the best model in terms of free energy, and that enforcing different factors in the state space can yield a lower model complexity. In particular, we consider the problem of 3D object representation, and focus on different instances of the ShapeNet dataset. We propose a model that factorizes object shape, pose and category, while still learning a representation for each factor using a deep neural network. We show that models, with best disentanglement properties, perform best when adopted by an active agent in reaching preferred observations.

This is part one of our three part series Part 1: API as a package: Structure (this post) Part 2: API as a package: Logging (to be published) Part 3: API as a package: Testing (to be published) Introduction At Jumping Rivers we were recently tasked with taking a prototype application built in {shiny} to a public facing production environment for a public sector organisation. During the scoping exercise it was determined that a more appropriate solution to fit the requirements was to build the application with a {plumber} API providing the interface to the Bayesian network model and other application tools written in R. When building applications in {shiny} we have for some time been using the “app as a package” approach which has been popularised by tools like {golem} and {leprechaun}, in large part due to the convenience that comes with leveraging the testing and dependency structure that our R developers are comfortable with in authoring packages, and the ease with which one can install and run an application in a new environment as a result. For this project we looked to take some of these ideas to a {plumber} application. This blog post discusses some of the thoughts and resultant structure that came as a result of that process. As I began to flesh out this blog post I realised that it was becoming very long, and there were a number of different aspects that I wanted to discuss: structure, logging and testing to name a few. To try to keep this a bit more palatable I will instead do a mini-series of blog posts around the API as a package idea and focus predominantly on the structure elements here. Do you use RStudio Pro? If so, checkout out our managed RStudio services API as a package There are a few things I really like about the {shiny} app as a package approach that I wanted to reflect in the design and build of a {plumber} application as package. It encourages a regular structure and organisation for an application. All modules have a consistent naming pattern and structure. It encourages leveraging the {testthat} package and including some common tests across a series of applications, see golem::use_reccommended_tests() for example. An instance of the app can be created via a single function call which does all the necessary set up, say my_package::run_app() Primarily I wanted these features, which could be reused across {plumber} applications that we create both internally and for our clients. As far as I know there isn’t a similar package that provides an opinionated way of laying out a {plumber} application as a package, and it is my intention to create one as a follow up to this work. Regular structure When developing the solution for this particular project I did have in the back of my mind that I wanted to create as much reusable structure for any future projects of this sort as possible. I really wanted to have an easy way to, from a package structure, be able to build out an API with nested routes, using code that could easily transfer to another package. I opted for a structure that utilised the inst/extdata/api/routes directory of a package as a basis with the idea that the following file structure | inst/extdata/api/routes/ | | - model.R | - reports/ - | | - pdf.R with example route definitions inside # model.R #* @post /prediction exported_function_from_my_package # pdf.R #* @post /weekly exported_function_from_my_package would translate to an API with the following endpoints /model/prediction /reports/pdf/weekly A few simple function definitions would allow us to do this for any given package that uses this file structure. The first function here just grabs the directory from the current package where I will define the endpoints that make up my API. get_internal_routes = function(path = ".") { system.file("extdata", "api", "routes", path, package = utils::packageName(), mustWork = TRUE) } create_routes will recursively list out all of the .R files within the chosen directory and name them according to the name of the file, this will make it easy to build out a a number of “nested” routers that will all be mounted into the same API, achieving the compartmentalisation that we desire. For example the two files at /inst/extdata/api/routes/model.R and /inst/extdata/api/routes/reports/pdf.R will take on the names "model" and "reports/pdf" respectively. add_default_route_names = function(routes, dir) { names = stringr::str_remove(routes, pattern = dir) names = stringr::str_remove(names, pattern = "\.R$") names(routes) = names routes } create_routes = function(dir) { routes = list.files( dir, recursive = TRUE, full.names = TRUE, pattern = "*\.R$" ) add_default_route_names(routes, dir) } The final few pieces to the puzzle ensure that we have / at the beginning of a string (ensure_slash()), for the purpose of mounting components to my router. add_plumber_definition() just calls the necessary functions from {plumber} to process a new route file, i.e from the decorated functions in the file create the routes, and then mount them at a given path to an existing router object. For example given a file “test.R” that has a #* @get /identity decorator against a function definition and endpoint = "test" we would add /test/identity to the existing router. generate_api() takes a full named vector/list of file paths, ensures they all have an appropriate name and mounts them all to a new Plumber router object. ensure_slash = function(string) { has_slash = grepl("^/", string) if (has_slash) string else paste0("/", string) } add_plumber_definition = function(pr, endpoint, file, ...) { router = plumber::pr(file = file, ...) plumber::pr_mount(pr = pr, path = endpoint, router = router ) } generate_api = function(routes, ...) { endpoints = purrr::map_chr(names(routes), ensure_slash) purrr::reduce2( .x = endpoints, .y = routes, .f = add_plumber_definition, ..., .init = plumber::pr(NULL) ) } With these defined I can then, as I develop my package, add new routes by defining functions and adding {plumber} tag annotations to files in /inst/ and rebuild the new API with get_internal_routes() %>% create_routes() %>% generate_api() and nothing about this code is specific to my current package so is transferable. As a concrete, but very much simplified example, I might have the following collection of files/annotations under /inst/extdata/api/routes ## File: /example.R # Taken from plumber quickstart documentation # https://www.rplumber.io/articles/quickstart.html #* @get /echo function(msg="") { list(msg = paste0("The message is: '", msg, "'")) } ## File: /test.R #* @get /is_alive function() { list(alive = TRUE) } ## File: /nested/example.R # Taken from plumber quickstart documentation # https://www.rplumber.io/articles/quickstart.html #* @get /echo function(msg="") { list(msg = paste0("The message is: '", msg, "'")) } which would give me get_internal_routes() %>% create_routes() %>% generate_api() # # Plumber router with 0 endpoints, 4 filters, and 3 sub-routers. # # Use `pr_run()` on this object to start the API. # ├──[queryString] # ├──[body] # ├──[cookieParser] # ├──[sharedSecret] # ├──/example # │ │ # Plumber router with 1 endpoint, 4 filters, and 0 sub-routers. # │ ├──[queryString] # │ ├──[body] # │ ├──[cookieParser] # │ ├──[sharedSecret] # │ └──/echo (GET) # ├──/nested # │ ├──/example # │ │ │ # Plumber router with 1 endpoint, 4 filters, and 0 sub-routers. # │ │ ├──[queryString] # │ │ ├──[body] # │ │ ├──[cookieParser] # │ │ ├──[sharedSecret] # │ │ └──/echo (GET) # ├──/test # │ │ # Plumber router with 1 endpoint, 4 filters, and 0 sub-routers. # │ ├──[queryString] # │ ├──[body] # │ ├──[cookieParser] # │ ├──[sharedSecret] # │ └──/is_alive (GET) This {cookieCutter} example is available to view at our Github blog repo. Basic testing In my real project I refrained from having any actual function definitions being made in inst/. Instead each function that was part of the exposed API was a proper exported function from my package (additionally filenames for said functions followed a regular structure too of api_.R). This allows for leveraging {testthat} against the logic of each of the functions as well as using other tools like {lintr} and ensuring that dependencies, documentation etc are all dealt with appropriately. Testing individual functions that will be exposed as routes can be a little different to other R functions in that the objects passed as arguments come from a request. As alluded to in the introduction I will prepare another blog post detailing some elements of testing for API as a package but a short snippet that I found particularly helpful for testing that a running API is functioning as I expect is included here. The following code could be used to set up (and subsequently tear down) a running API that is expecting requests for a package cookieCutter # tests/testthat/setup.R ## run before any tests # pick a random available port to serve your app locally port = httpuv::randomPort() # start a background R process that launches an instance of the API # serving on that random port running_api = callr::r_bg( function(port) { dir = cookieCutter::get_internal_routes() routes = cookieCutter::create_routes(dir) api = cookieCutter::generate_api(routes) api$run(port = port, host = "0.0.0.0") }, list(port = port) ) # Small wait for the background process to ensure it # starts properly Sys.sleep(1) ## run after all tests withr::defer(running_api$kill(), testthat::teardown_env()) A simple test to ensure that our is_alive endpoint works then might look like test_that("is alive", { res = httr::GET(glue::glue("http://0.0.0.0:{port}/test/is_alive")) expect_equal(res$status_code, 200) }) Logging {shiny} has some useful packages for adding logging, in particular {shinylogger} is very helpful at giving you plenty of logging for little effort on my part as the user. As far as I could find nothing similar exists for {plumber} so I set up a bunch of hooks, using the {logger} package to write information to both file and terminal. Since that could form it’s own blogpost I will save that discussion for the future. For updates and revisions to this article, see the original post

Machine learning is a set of tools that, broadly speaking, allow us to "teach" computers how to perform tasks by providing examples of how they should be done. For example, suppose we wish to write a program to distinguish between valid email messages and unwanted spam. We could try to write a set of simple rules, for example, flagging messages that contain certain features (such as the word "viagra" or obviously-fake headers). However, writing rules to accurately distinguish which text is valid can actually be quite difficult to do well, resulting either in many missed spam messages, or, worse, many lost emails. Worse, the spammers will actively adjust the way they send spam in order to trick these strategies (e.g., writing "vi@gr@"). Writing effective rules -- and keeping them up to date -- quickly becomes an insurmountable task. Fortunately, machine learning has provided a solution. Modern spam filters are "learned" from Examples: we provide the learning algorithm with example emails which we have manually labelled as "ham" (valid email) or "spam" (unwanted email), and the algorithms learn to distinguish between them automatically.