Collaborating Authors

Lossy source encoding via message-passing and decimation over generalized codewords of LDGM codes Artificial Intelligence

We describe message-passing and decimation approaches for lossy source coding using low-density generator matrix (LDGM) codes. In particular, this paper addresses the problem of encoding a Bernoulli(0.5) source: for randomly generated LDGM codes with suitably irregular degree distributions, our methods yield performance very close to the rate distortion limit over a range of rates. Our approach is inspired by the survey propagation (SP) algorithm, originally developed by Mezard et al. for solving random satisfiability problems. Previous work by Maneva et al. shows how SP can be understood as belief propagation (BP) for an alternative representation of satisfiability problems. In analogy to this connection, our approach is to define a family of Markov random fields over generalized codewords, from which local message-passing rules can be derived in the standard way. The overall source encoding method is based on message-passing, setting a subset of bits to their preferred values (decimation), and reducing the code.

Visual Information Theory -- colah's blog


I love the feeling of having a new way to think about the world. I especially love when there's some vague idea that gets formalized into a concrete concept. Information theory is a prime example of this. Information theory gives us precise language for describing a lot of things. How uncertain am I? How much does knowing the answer to question A tell me about the answer to question B? How similar is one set of beliefs to another? I've had informal versions of these ideas since I was a young child, but information theory crystallizes them into precise, powerful ideas. These ideas have an enormous variety of applications, from the compression of data, to quantum physics, to machine learning, and vast fields in between. Unfortunately, information theory can seem kind of intimidating. I don't think there's any reason it should be. In fact, many core ideas can be explained completely visually! Before we dive into information theory, let's think about how we can visualize simple probability distributions. We'll need this later on, and it's convenient to address now. As a bonus, these tricks for visualizing probability are pretty useful in and of themselves! Sometimes it rains, but mostly there's sun! Let's say it's sunny 75% of the time. It's easy to make a picture of that: Most days, I wear a t-shirt, but some days I wear a coat. Let's say I wear a coat 38% of the time. It's also easy to make a picture for that! What if I want to visualize both at the same time?

Deep Compressive Autoencoder for Action Potential Compression in Large-Scale Neural Recording Artificial Intelligence

Understanding the coordinated activity underlying brain computations requires large-scale, simultaneous recordings from distributed neuronal structures at a cellular-level resolution. One major hurdle to design high-bandwidth, high-precision, large-scale neural interfaces lies in the formidable data streams that are generated by the recorder chip and need to be online transferred to a remote computer. The data rates can require hundreds to thousands of I/O pads on the recorder chip and power consumption on the order of Watts for data streaming alone. We developed a deep learning-based compression model to reduce the data rate of multichannel action potentials. The proposed model is built upon a deep compressive autoencoder (CAE) with discrete latent embeddings. The encoder is equipped with residual transformations to extract representative features from spikes, which are mapped into the latent embedding space and updated via vector quantization (VQ). The decoder network reconstructs spike waveforms from the quantized latent embeddings. Experimental results show that the proposed model consistently outperforms conventional methods by achieving much higher compression ratios (20-500x) and better or comparable reconstruction accuracies. Testing results also indicate that CAE is robust against a diverse range of imperfections, such as waveform variation and spike misalignment, and has minor influence on spike sorting accuracy. Furthermore, we have estimated the hardware cost and real-time performance of CAE and shown that it could support thousands of recording channels simultaneously without excessive power/heat dissipation. The proposed model can reduce the required data transmission bandwidth in large-scale recording experiments and maintain good signal qualities. The code of this work has been made available at

Solar energy and moonshine politics Brief letters

The Guardian > Energy

Did I invent the solar panels scheme which paid a generous feed-in tariff to install panels on your roof? I think I may also have imagined a green deal which was so advantageous that nobody much took it up. I fear this new initiative (UK'on verge of clean energy revolution', 25 July) is going to place a similar strain on my mental faculties when it vanishes without trace under the label "green crap". As such many see them as superior rather than inferior to marriage. I mind how irritating they are.

A strong converse bound for multiple hypothesis testing, with applications to high-dimensional estimation Machine Learning

In statistical inference problems, we wish to obtain lower bounds on the minimax risk, that is to bound the performance of any possible estimator. A standard technique to obtain risk lower bounds involves the use of Fano's inequality. In an information-theoretic setting, it is known that Fano's inequality typically does not give a sharp converse result (error lower bound) for channel coding problems. Moreover, recent work has shown that an argument based on binary hypothesis testing gives tighter results. We adapt this technique to the statistical setting, and argue that Fano's inequality can always be replaced by this approach to obtain tighter lower bounds that can be easily computed and are asymptotically sharp. We illustrate our technique in three applications: density estimation, active learning of a binary classifier, and compressed sensing, obtaining tighter risk lower bounds in each case.