The code for our experiments is available at https://github.com/AndyShih12/HyperSPN. To examine the merits of HyperSPNs as discussed in Section 3, we construct a hand-crafted dataset to test the three types of models described in Figure 4: SPN-Large, SPN-Small, and HyperSPN. The hand-crafted dataset is procedurally generated with 256 binary variables and 10000 instances, broken into train/valid/test splits at 70/10/20%. The generation procedure is designed such that the correlation between variable i and j is dependent on the path length between leaves i and j of a complete binary tree over the 256 variables. The exact details can be found in our code.

HR-based orthogonal fine-tuning is equivalent to an adaptive low-rank adaptation.

The feed-forward hidden dimension of Transformers is set to 1024. AC-to-all Transformer decoders, have 2 layers. The same setup is used for both the WOMD dataset and the A V2 dataset. In the following, we report the configuration of ablation models. VRAM (RTX 3090 in our case) because they require more GPU memory at training time.

Current PEFT methods for LLMs can achieve high quality, efficient training, or scalable serving, but not all three simultaneously.

Neural Information Processing Systems http://nips.cc/