Neuro-symbolic neural networks have been extensively studied to integrate symbolic operations with neural networks, thereby improving systematic generalization. Specifically, Tensor Product Representation (TPR) framework enables neural networks to perform differentiable symbolic operations by encoding the symbolic structure of data within vector spaces. However, TPR-based neural networks often struggle to decompose unseen data into structured TPR representations, undermining their symbolic operations. To address this decomposition problem, we propose a Discrete Dictionary-based Decomposition (D3) layer designed to enhance the decomposition capabilities of TPR-based models. D3 employs discrete, learnable key-value dictionaries trained to capture symbolic features essential for decomposition operations. It leverages the prior knowledge acquired during training to generate structured TPR representations by mapping input data to pre-learned discrete features within these dictionaries. D3 is a straightforward drop-in layer that can be seamlessly integrated into any TPR-based model without modifications. Our experimental results demonstrate that D3 significantly improves the systematic generalization of various TPR-based models while requiring fewer additional parameters. Notably, D3 outperforms baseline models on the synthetic task that demands the systematic decomposition of unseen combinatorial data.

We provide additional analytical and experimental results to support the content presented in the main manuscript. Section 1 of this document presents a detailed discussion on the Rumi-LSGAN. In Section 2, we impose the Rumi formulation on f-GANs [1], and in Section 3, we generalize it to include integral probability metric (IPM) based GANs such as the Wasserstein GAN (WGAN) [2]. In Section 4, we compare the performance of Rumi-SGAN, Rumi-LSGAN, and Rumi-WGAN on the MNIST dataset. Finally, in Section 5, we provide additional results and comparisons on CelebA and CIFAR-10 datasets.

Generative adversarial networks (GANs) were originally envisioned as unsupervised generative models that learn to follow a target distribution.

This supplementary material contains additional details of Section 2.4, 3 and 4.4 and a discussion about the broader impacts of this paper. The details include: Searching in the searched space. We provide the details of the two steps for vision transformer search: (1) Supernet training without resource constraints; (2) Evolution search under resource constraint. Q-K-V dimension could be smaller than the embedding dimension. We suppose the underlying reasons might be that the feature maps of the different heads are similar in deeper layers.