Running Ray at Scale on AKS

"GPU scarcity is one of the most significant operational challenges in large-scale ML. High-demand accelerators, such as NVIDIA GPUs, often have quota and availability issues in Azure regions. This can delay cluster setup and job scheduling. Microsoft's proposed solution uses a multi-cluster, multi-region setup. Distributing Ray clusters across different AKS instances in various Azure regions allows teams to aggregate GPU quota beyond regional limits, automatically reroute workloads during outages or capacity issues and extend the compute pool to on-premises systems or other cloud providers using Azure Arc with AKS."

"Ray is a Python-native distributed compute framework designed to scale AI and ML workloads from a single laptop to clusters spanning thousands of nodes. Anyscale's managed platform enhances Ray with features for production use. The new guidance shows a partnership between Microsoft and Anyscale to improve Azure integration."

"Anyscale's improved runtime, previously known as RayTurbo. This runtime offers smart autoscaling, improved monitoring, and fault-tolerant training features. They are all based on the open-source Ray framework."

Microsoft's Azure Kubernetes Service team partnered with Anyscale to provide guidance for deploying Anyscale's managed Ray service at scale. The guidance addresses three critical challenges: GPU capacity constraints, scattered ML storage, and credential expiration problems. Anyscale's improved runtime, built on the open-source Ray framework, offers smart autoscaling, enhanced monitoring, and fault-tolerant training capabilities. Ray is a Python-native distributed compute framework that scales AI and ML workloads from single machines to multi-thousand node clusters. To overcome GPU scarcity in Azure regions, Microsoft recommends multi-cluster, multi-region setups that aggregate GPU quota, automatically reroute workloads during outages, and extend compute pools across on-premises systems and other cloud providers using Azure Arc. The Anyscale console provides unified cluster management, while Anyscale Workspaces handles workload scheduling based on available capacity.

Read at InfoQ