This paper describes an extension to the PILCO algorithm (Probabilistic Inference and Learning for COntrol, a data-efficient reinforcement algorithm). The proposed algorithm applies a measurement filtering algorithm during the actual experiment and explicitly takes this measurement filtering algorithm into account during the policy learning step, which uses data from the experiment. This is an important practical extension addressing the fact that measurements are often very noisy. My intuitive explanation for this approach is that the proposed approach makes the overall feedback system more "repeatable" (noise is mostly filtered out) and therefore learning is faster (given that the filtering is effective, see last sentence of the conclusion). The paper presents detailed mathematical derivations and strong simulation results that highlight the properties of the proposed algorithm.

I am happy with all of your responses, though slightly confused over Q2 (rev2). One can't draw samples from improper priors in the first place, and other techniques (such as Rodrigues et al) won't save you there. You simply need to draw your samples from a distribution that is not the prior. I am still positively inclined towards this paper, and following the response and comparison to EP-ABC I will increase my score to 7 (from 6). Of course when the prior is improper or merely diffuse with respect to the posterior this will be impossible or at best highly inefficient.

The paper proposes TSDE, a posterior sampling algorithm for RL in the average reward infinite horizon setting. This algorithm uses dynamic episodes but unlike Lazy-PSRL avoids technical issues by not only terminating an episode when an observation count doubled but also terminating episodes when they become too long. This ensures that the episode length cannot grow faster than linear and ultimately a Bayesian regret bound of O(HS(AT) .5) is shown. Posterior sampling methods typically outperform UCB-type algorithms and therefore a posterior sampling algorithm for non-episodic RL with rigorous regret bounds is desirable. This paper proposes such an algorithm, which is of high interest.

The paper proposes an entropy estimate for Markov chains by reduction to optimal entropy estimation for i.i.d samples. Sample complexity analysis is provided for different mixing scenarios with a minimax rate established for a particular rate. The estimator is used to assess the capacity of language models. This is a very clear and well-written paper. I appreciate the efforts done by the authors to summarize the results.

The presented method learns a structure of a deep ANN by first learning a BN and then constructing the ANN from this BN. The authors state that they "propose a new interpretation for depth and inter-layer connectivity in deep neural networks". Neurons in deep layers represent low-order conditional independencies (ie small conditioning set) and those in'early' (non-deep) layers represent high-order CI relationships. These are all CI relations in the "X" ie the input vector of (observed) random variables. Perhaps I am missing something here but I could not find an argument as to why this is a principled way to build deep ANNs with good performance.

It seems that this could perhaps be expressed more concisely using the output of the discriminator (and the true label) as functions, rather than introducing new random variables. Further, it seems the algorithm is described in sufficient detail to be re-implemented. The experiments are missing some detail to be reproduced or interpreted (e.g.

This paper studies the problem of computing probability bounds, more specifically bounds over probability of atoms of the joint space and conditional probabilities of the class, under the assumption that only some pairwise marginal as well as some univariate marginal values are known. The idea is that such marginals may be easier to obtain than fully specified probabilities, and that cautious inferences can then be used to produce predictions. It is shown that when the marginals follow a tree structure (results are extended to a few other structures), then the problem can actually be solved in closed, analytical form, relating it to cover set and maximum flow problems. Some experiments performed on neural networks show that this simple method is actually competitive with other more complex approaches (Ladder, VAE), while outperforming methods of comparable complexity. The paper is elegantly written, with quite understandable and significant results.

This paper presents H4MF model (HMM MF for dynamic Missingness) for implicit feedback data. With implicit data, we only observe positive feedback and the missing entries (zeros) in the data can indicate either negative feedback or users are not exposed of the items. H4MF is based on the previous work on modeling user latent exposure (ExpoMF, Liang et al., Modeling user exposure in recommendation, 2016) -- the basic idea is that for each user-item pair, there is a latent binary variable to represent exposure; if it's 1, it means this user is exposed to the item thus 0 feedback mean true negative, while if it's 0, it means this user have not yet been exposed to this item yet. The difference in H4MF is that H4MF uses a hidden Markov model to capture the temporal dynamics in the user exposure (user intent in this paper). The basic idea is that whether or not a user is exposed to something can be dependent on some other items he/she has been exposed before.

Overview: this paper introduces the Truncated Gaussian (TruG) unit (as per eq 1, and its expectation in eq 2). By adjusting the cutoffs \xi_1 and \xi_2 this can mimic ReLU and sigmoid/tanh units. It can be used as a stochastic unit in RBMs (sec 3), in temporal RBMs (sec 4), and in the TGGM (truncated Gaussian graphical model, sec 5). An old but relevant reference is "Continuous sigmoidal belief networks trained using slice sampling" B. J. Frey, in M. C. Mozer, M. I. Jordan and T. Petsche (eds), Advances in Neural Information Processing Systems 9, 452-459, January 1997. One might criticise this paper by saying that once one has come up with the TruG unit, one simply has to "turn the handle" on the usual derivations to get TruG-RBMs, temporal TruG-RBMs and TruG-TGGMs.

Summary: This paper presents a new transfer learning approach using Bayesian Neural Network in MDPs. They are building on the existing framework of Hidden Parameter MDPs, and replace the Gaussian process with BNNs, thereby also modeling the joint uncertainty in the latent weights and the state space. Overall, this proposed approach is sound, well developed and seems to help scale the inference. The authors have also shown that it works well by applying it to multiple domains. The paper is extremely well written.