Where the original RHN work primarily provides theoretical treatment of the subject, we demonstrate experimentally that RHNs benefit from far better gradient flow than LSTMs in addition to their improved task accuracy. The original hypernetworks work presents detailed experimental results but leaves several theoretical issues unresolved--we consider these in depth and frame several feasible solutions that we believe will yield further gains in the future. We demonstrate that these approaches are complementary: by combining RHNs and hypernetworks, we make a significant improvement over current state-of-the-art character-level language modeling performance on Penn Treebank while relying on much simpler regularization. Finally, we argue for RHNs as a drop-in replacement for LSTMs (analogous to LSTMs for vanilla RNNs) and for hypernetworks as a de-facto augmentation (analogous to attention) for recurrent architectures.

Bayesian deep learning has always garnered substantial interest in the machine learning community.

Deep reinforcement learning (RL) is notoriously impractical to deploy due to sample inefficiency.

Structural pruning has emerged as a promising solution to reduce the costs of LLMs without requiring post-processing steps.

Neural Information Processing Systems http://nips.cc/

Our method comes close to the original models' performance in cross-lingual and

Cooperative multi-agent reinforcement learning (MARL) has been proposed for multi-agent collaborations toaccomplish manychallenging tasks[1,2,3,4].