Enhancing Cluster Scheduling in HPC: A Continuous Transfer Learning for Real-Time Optimization

This is the accepted version of the paper publis hed in 2025 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) . Given Name Surname line 2: dept. Given Name Surname line 2: dept. Abstract -- This study presents a machine learning - assisted approach to optimize task scheduling in cluster systems, focusing on node - affinity constraints. Traditional schedulers like Kubernetes struggle with real - time adaptability, whereas the proposed continuous transfer learning model evolves dynamically during operations, minimizing retraining needs. Evaluated on Google Cluster Data, the model achieves over 99% accuracy, reducing computational overhead and improving scheduling latency for constrained tasks. This scalable solution enables real - time optimization, advancing ma chine learning integration in cluster management and paving the way for future adaptive scheduling strategies. In the rapidly evolving landscape of cloud computing and distributed high - performance environments, the efficient management of architectural and software resources became apparently paramount for ensuring suitable performance and minimizing latency. As long as the industry organizations increasingly rely on cluster - based architectures to orchestrate their broad areas of possible applications, the importance of effective task scheduling has come to the forefront . Over the last few years, traditional schedulers, such as Kubernetes and some more, have laid the groundwork for managing containerized workloads; however, it was found that it poses a challenge for them to adapt to the dynamic nature of real - time workloads and node - affinity constraints [ 35 ] . These limitations result in inefficient resource utilization and longer scheduling delays, which ultimately affect overall system performance, especially in high - performance systems [9][18] . In mission - critical environments, these issues can escalate, disrupting vital systems like power networks, healthcare, defen s e systems, and others.