Reward allocation, also known as the credit assignment problem, has been an important topic ineconomics, engineering, andmachine learning.

In this paper, we present DSA, the first automated framework for discovering sparsity allocation schemes for layer-wise pruning in Large Language Models (LLMs). LLMs have become increasingly powerful, but their large parameter counts make them computationally expensive. Existing pruning methods for compressing LLMs primarily focus on evaluating redundancies and removing element-wise weights. However, these methods fail to allocate adaptive layerwise sparsities, leading to performance degradation in challenging tasks.

Continuous allocation tasks are a class of problems where an agent needs to distribute a limited amount of resources over a set of entities at each time step.

Research on scaling large language models (LLMs) has primarily focused on model parameters and training data size, overlooking the role of vocabulary size.

Large Language Models (LLMs) have revolutionized the field of natural language processing with their impressive capabilities.

Our analysis highlights sufficient properties for a regret minimization algorithm to be used as leader.

However, an emerging issue in allocation problems is data privacy.