This paper challenges the well-established paradigm for building any-to-any networks for training Large Language Models (LLMs). We show that LLMs exhibit a unique communication pattern where only small groups of GPUs require high-bandwidth communication to achieve near-optimal training performance. Across these groups of GPUs, the communication is insignificant and homogeneous. We propose a new network architecture that resembles the communication requirement of LLMs. Our architecture partitions the cluster into sets of GPUs interconnected with non-blocking any-to-any high-bandwidth interconnects that we call HB domains. Across the HB domains, the network only connects GPUs with non-zero communication demands. We develop an analytical formulation of the training iteration time to evaluate our proposal. Our formulation closely estimates the hardware floating-point utilization within 0.15\% from the ground truth established in prior studies for larger models. We show that our proposed architecture reduces the network cost by 37% to 75% compared to the state-of-the-art any-to-any Clos networks without compromising the performance of LLM training.

Researchers are proposing a framework that would allow users to understand the rationale behind artificial intelligence (AI) decisions. The work is significant, given the push to move away from "black box" AI systems – particularly in sectors, such as military and law enforcement, where there is a need to justify decisions. "One thing that sets our framework apart is that we make these interpretability elements part of the AI training process," says Tianfu Wu, first author of the paper and an assistant professor of computer engineering at North Carolina State University. "For example, under our framework, when an AI program is learning how to identify objects in images, it is also learning to localize the target object within an image, and to parse what it is about that locality that meets the target object criteria. This information is then presented alongside the result." In a proof-of-concept experiment, researchers incorporated the framework into the widely-used R-CNN AI object identification system.