Leveraging Singularity to Unleash the Power of Hybrid HPC Clouds

Today’s HPC centers demand the flexibility of hybrid computing environments.  CFOs and CIOs understand the potential cost benefits of cloud computing and do not wish to be limited by on-premise compute investments.  A challenge administrators face when extending their computing capacity outside of their datacenter is delivering a cloud resource in a way that is easy for users to adopt. They need a straightforward extension of computing power into on-demand cloud services

HPC users have a myriad of operating system and application choices available to them in the open source community. Whether due to personal preference or scientific need, it is impossible to fit all application workflows into a single bucket.  This requires that HPC centers provide users with the tools needed to build custom application stacks in their environment of choice.

While virtualization technologies provide the capability to compartmentalize software into portable virtual machines, HPC applications face unacceptable performance and scalability limitations when running in a virtual environment.

The rise of Linux containers and the rapid success of frameworks like Docker quickly became of interest to the HPC community as containers enable the encapsulation of custom applications and their runtime environments without the performance penalty of virtualization.  Containers gave hope for the long awaited, bare-metal answer to software portability which could streamline the adaptability of hybrid cloud computing. HPC administrators quickly realized that although Linux containers could facilitate hybrid computing models, the available tools were designed for enterprise microservices and could not be easily, or securely be integrated into HPC clusters.

Singularity, founded by Gregory M. Kurtzer and supported by Lawrence Berkeley National Lab was launched in April of 2016 as the definitive framework for HPC Linux containers.  The Singularity team addressed all of the HPC challenges presented by enterprise container solutions including native support for MPI, a single flat-file image for hosting containers on parallel filesystems, restricted privilege escalation inside the container for security, and a user-space execution architecture that allowed HPC schedulers to retain control of resource management and job scheduling.

Additionally, Singularity is compatible with popular container repositories such as DockerHub.  This provides users the option to leverage the growing public depots of pre-packaged, containerized applications.  With two commands, a user can create a ready to run Singularity container from a Docker registry.

“I spoke with numerous scientists, and the problem to solve became clear. We need to support application mobility via an easily verifiable reproducible software stack that the scientists control and a runtime that fits a usage paradigm that fits a traditional HPC system architecture. This was the birth of the Singularity project.”  — Gregory M. Kurtzer.

Penguin Computing is a leader in open computing solutions, and delivers various hybrid computing options through Penguin Computing On-Demand (POD), a bare-metal HPC cloud service.  Coupled with Singularity, POD users are able to compartmentalize their software, and migrate workloads to and from the cloud without any changes to their operating system or application binaries.  Their HPC application stack is consistent and reproducible whether on-premise or in the cloud, and is not restrained by the underlying operating system HPC administrators choose to provision their clusters.

Administrators are able to drop-in Singularity to their on-premise clusters without any changes to their provisioning or scheduling environment.  The simple installation of Singularity encourages and empowers users to adopt the hybrid computing options their organization wishes to embrace. An HPC cluster can easily be enabled to allow users to create Linux containers that extend into POD for seamless integration into the cloud.

Penguin Computing’s POD is proud to be part of the community of HPC centers embracing Singularity, such as the National Institute of Health, the Texas Advanced Computing Center, and Lawrence Berkeley National Lab.

Register for Penguin Computing’s upcoming whitepaper on enabling hybrid HPC computing in the cloud through Singularity.