Controlling the Injection of Best-Effort Tasks to Harvest Idle Computing Grid Resources

Quentin Guilloteau (1) , Olivier Richard (2) , Bogdan Robu (3) , Eric Rutten (1)

1 CTRL-A - Control for Autonomic computing systems
2 DATAMOVE - Data Aware Large Scale Computing
3 GIPSA-MODUS - GIPSA - Modelling and Optimal Decision for Uncertain Systems

High Performance Computing systems are facing more and more variability in their performance, related to e.g., Input/Output (I/O) behavior and power consumption : they are less predictable, which requires more run-time management to meet the requirements. This can be addressed following feedback approach, where a management feedback loop, in response to monitored information in the systems, based on analysis of this data, decides to activate system-level or application-level adaptation mechanisms. One such regulation problem is found in the context of CiGri, a lightweight computing grid system which exploits the unused resources of a set of computing clusters. The computing power left over by the execution of premium cluster users' HPC applications, is used to execute smaller jobs, which are injected as much as the global system allows. The feedback loop which we design has to regulate this injection of jobs in such a way as to avoid overloading of the distributed file system (or file-server), which would be detrimental to the main performance, while self-adapting to variations in load in order to make the best use of available resources. We put in place a mechanism for feedback control in system software by controlling the number of jobs to be sent to the cluster in response to system information about the current number of processed jobs and the file-server load, which has a significant impact of the performance of the priority users jobs. We perform experimental validation by comparing several control solutions.

Mots clés: Dynamic resource management; High performance computing; autonomic computing; instrumentation; self-adaptive systems

Domaines: Calcul parallèle, distribué et partagé [cs.DC]; Automatique; Génie logiciel [cs.SE]

Quentin Guilloteau, Olivier Richard, Bogdan Robu, Eric Rutten. Controlling the Injection of Best-Effort Tasks to Harvest Idle Computing Grid Resources. ICSTCC 2021 - 25th International Conference on System Theory, Control and Computing, Oct 2021, Iași, Romania. pp.1-6, ⟨10.1109/ICSTCC52150.2021.9607292⟩. ⟨hal-03363709⟩