Introduction to Azure HPC
Most modern technologies such as artificial intelligence, machine learning, and deep learning consume mountains of data every day to produce the desired results. But harnessing the full power of the acquired data requires the use of HPC.
HPC or High-Performance Computing is a high-speed system that can perform quadrillions of calculations per second. This is miles ahead when you compare it with regular laptops and desktops which perform at most 3 billion calculations per second.
Now that you know what HPC can achieve, your next question is how do I implement it in my organization. Unfortunately, employing on-premise HPC systems can cost an arm and a leg. This is where virtualization can help. Thanks to virtualization, using HPC servers on the cloud is becoming increasingly preferred in large-scale enterprises. A really popular name in the world of HPC virtualization is Azure HPC. In this post, let us understand everything about it right from its benefits to working.
Table of Contents
- How is HPC different on the cloud?
- What is Azure HPC?
- How to connect your on-premise HPC systems to Azure?
- Azure HPC components
- HPC platform services
- Verdict
How is HPC different on the cloud?
Autoscaling is the primary difference between on-premies and virtual HPC servers. This allows dynamic scaling of the HPC infrastructure. This helps improve the compute capacity by eliminating performance issues that remain as a bottleneck, allowing customers to use the infrastructure that exactly meets the requirements of their jobs.
But in the case of on-premises HPC servers, you can only scale by bringing in more HPC servers to increase the compute capacity, thereby leading to more storage and upfront costs.
This is where Azure HPC comes in with its fully managed platform services and a robust architecture to run HPC workloads and applications.
What is Azure HPC?
Azure high-performance computing is a set of computing, storage, and networking resources integrated with the required workload orchestration service for HPC systems to run. With a solid HPC infrastructure, solutions, and services to scale systems dynamically, Azure has a competitive edge in comparison to the on-premises HPC options. In addition to the high performance, Azure comes with next-gen machine learning tools to drive smarter simulations and make intelligent decisions.
How to connect your on-premise HPC systems to Azure?
If you have an on-premise HPC server and want to connect to Azure, there are several ways to achieve the HPC virtualization. Below are some ways to get started -
- You can extend your on-premises network to Azure cloud and achieve cloud bursting through the Azure stack VPN gateway. This redirects all traffic from the on-premises network to the Azure virtual network.
- Through Azure ExpressRoute that uses a private, dedicated connection through a third-party connectivity provider. This private connection extends to your private on-premises network.
- Implement a secure and available site-to-site network architecture that spans across your Azure virtual network and your on-premises network using ExpressRoute with VPN failover. VPN failover provides an automatic backup connection for VPN providing all-time connectivity for IPSec connections. IPSec is a group of protocols used together to set up encrypted connections between devices.
Having explored the ways of connecting your on-premises network to Azure virtual network, next let us try to understand the different components present in Azure HPC.
Azure HPC components
While HPC deployments in Azure can vary according to your budget and workload needs, there are some standard components present in any deployment. They are -
Azure Resource Manager - The ARM allows you to deploy your applications to your HPC clusters via script files and templates. An HPC cluster is nothing but a collection of servers or computers called nodes connected via a fast interconnect in a data center.
HPC head node - Used to assign workloads and schedule jobs to the HPC worker nodes. This is a virtual network used to manage HPC clusters.
Virtual networks - Used to create isolated networks for your HPC clusters and storage through secure connections using IPSec VPN and ExpressRoute which are the platform services provided by Azure for networking.
Virtual machines scale sets - Used for provisioning multiple VMs for your clusters with features like auto-scaling, load balancing, and multi-zone deployments. You can use these scale sets to run different databases such as Cassandra, MongoDB, and Hadoop.
Storage - Enables you to mount storage to your clusters in the form of disk, blob, file, data lake storage, or virtual block stores.
HPC platform services
Compute
Azure offers robust compute services that provide unlimited scalability options for your HPC applications. You can use H-series virtual machines for memory-bound applications, N-series virtual machines for graphic-intensive applications, or Cray which provides a fully customized supercomputer as a managed service.
Storage
Burst your HPC applications into Azure using data stored in on-premises HPC servers using HPC cache. Accessing large amounts of I/O has been made easy (latency rates hovering around sub milliseconds) with Azure NetApp Files which is an Azure data service within the Azure datacenters. In case of a high throughput storage solution, you can use Cray ClusterSor, a baremetal HPC storage solution fully integrated with Azure. Baremetal servers are physical computers required for running dedicated workloads for extended periods. They are highly reliable, durable, and stable.
Networking
You can establish private, secure tunnels for hybrid cloud connectivity to your HPC workloads through Azure ExpressRoute. Also take advantage of Linux Remote Directory Memory Access(RDMA) using InfiniBand, for message passing interface workloads within your HPC datacenter.
Application services
Azure application services for HPC allow you to create, manage and schedule jobs in applications using Azure Batch. Azure Batch is a platform service that allows the running of large-scale parallel and high-performance computing applications seamlessly on the cloud. Also, you can use Azure CycleCloud to provision HPC clusters dynamically on the cloud.
Containers can also be used to manage HPC workloads. Azure Kubernetes Service is one such container orchestration service that can run and manage deployed HPC clusters in Azure.
Intelligent services
Azure offers data lake analytics which enables you to build the next generation of applications using predictive analysis. You can build and run machine learning models on top of your HPC data to gain insights that ultimately lead to better decisions.
Verdict
Azure is constantly expanding its HPC services across different domains. Currently Azure offers HPC services in domains like automotive, energy, manufacturing, financial services, health research, and IoT. Also used with transformative technologies such as AI and machine learning, Azure HPC can be instrumental to reach the desired outcomes at faster rates for your data-intensive applications.