We present Theseus, an efficient application-agnostic open source library for differentiable nonlinear least squares (DNLS) optimization built on PyTorch, providing a common framework for end-to-end structured learning in robotics and vision. Existing DNLS implementations are application specific and do not always incorporate many ingredients important for efficiency. Theseus is application-agnostic, as we illustrate with several example applications that are built using the same underlying differentiable components, such as second-order optimizers, standard costs functions, and Lie groups. For efficiency, Theseus incorporates support for sparse solvers, automatic vectorization, batching, GPU acceleration, and gradient computation with implicit differentiation and direct loss minimization. We do extensive performance evaluation in a set of applications, demonstrating significant efficiency gains and better scalability when these features are incorporated.

However, safe RL often suffers from sample inefficiency, requiring extensive interactions with the environment to learn a safe policy.

Neurons in early sensory areas rapidly adapt to changing sensory statistics, both by normalizing the variance of their individual responses and by reducing correlations between their responses. Together, these transformations may be viewed as an adaptive form of statistical whitening. Existing mechanistic models of adaptive whitening exclusively use either synaptic plasticity or gain modulation as the biological substrate for adaptation; however, on their own, each of these models has significant limitations. In this work, we unify these approaches in a normative multi-timescale mechanistic model that adaptively whitens its responses with complementary computational roles for synaptic plasticity and gain modulation. Gains are modified on a fast timescale to adapt to the current statistical context, whereas synapses are modified on a slow timescale to match structural properties of the input statistics that are invariant across contexts. Our model is derived from a novel multi-timescale whitening objective that factorizes the inverse whitening matrix into basis vectors, which correspond to synaptic weights, and a diagonal matrix, which corresponds to neuronal gains. We test our model on synthetic and natural datasets and find that the synapses learn optimal configurations over long timescales that enable adaptive whitening on short timescales using gain modulation.

To evaluate the convexity of the self-supervised loss function for event-based optical flow estimation from [56] and the adaptation that we propose in this work, we conducted an experiment with two partitions of 40k events from the ECD dataset [31]. In this experiment, for the selected partitions, we computed the value of Eq. 4 (with and without the scaling) for four sets of optical flow vectors given by: u Figure 1 highlights the main difference between the original and our adapted formulation. On the contrary, the scaling that we propose in Section 3.2 fixes this issue, and results in a convex loss function for any value of d. Numbers on top indicate the maximum per-axis pixel displacement for each column. Scaled L Original L (see Eq. 6) but without the reset mechanism.

The field of neuromorphic computing promises extremely low-power and lowlatency sensing and processing. Challenges in transferring learning algorithms from traditional artificial neural networks (ANNs) to spiking neural networks (SNNs) have so far prevented their application to large-scale, complex regression tasks. Furthermore, realizing a truly asynchronous and fully neuromorphic pipeline that maximally attains the abovementioned benefits involves rethinking the way in which this pipeline takes in and accumulates information. In the case of perception, spikes would be passed as-is and one-by-one between an event camera and an SNN, meaning all temporal integration of information must happen inside the network. In this article, we tackle these two problems. We focus on the complex task of learning to estimate optical flow from event-based camera inputs in a self-supervised manner, and modify the state-of-the-art ANN training pipeline to encode minimal temporal information in its inputs.

A How does the sketch complexity influence communication? 2 B How important is the rasteriser? 2 C What is the impact of L in the computation of the perceptual loss on the emergent sketches? A How does the sketch complexity influence communication? An interesting question one might ask about a model that learns to communicate by drawing is how complex the sketch image needs to be so that its meaning can be conveyed successfully and communication can be established. We attempt to answer this question by varying the number of lines that our model is allowed to draw to represent the input photograph. In Table I, we show results for experiments run with 5, 10 and 20 lines allowed for sketching.

In this section we provide full descriptions of the diffusion processes introduced in Sec. 3. A.1 Continuous noise diffusion The forward process is defined by noise variances. We use the cosine variance schedule from Nichol and Dhariwal [55].