If you’ve read our earlier post you’ll understand how to create an AWS ParallelCluster and adapt the head and compute nodes with custom scripts that you place in an AWS S3 bucket.

Here we discus using the AWS ParallelCluster [ImageBuilder] utility to create a custom AMI image that you can use to create your cluster, useful if you find you’re installing a lot of custom packages that slow down the formation of new compute nodes.

Rather than installing packages and tools every time a cluster node is created (costing valuable minutes) the [ImageBuilder] lets you construct a customised AMI.

Here we’re using the AWS ParallelCluster, an AWS supported open source cluster management tool that helps you to deploy and manage high performance computing clusters in the AWS Cloud. For a background refer to the AWS introduction.

In our earlier post we demonstrated the use of AWS ParallelCluster (v2) for Nextflow. Here we briefly discuss switching from ParallelCluster v2 to v3.

You’ve probably created a machine image at some point. A base image for AWS that builds upon someone’s work by adding a particular version of Java or Python or a new utility. Did you create the image on AWS using an EC2 instance, login, run some yum or apt-get and then save it? Great, and if someone wants the source code for that image or you want to build a similar image on a different region or provider? Well, [Packer] is an IaC tool for automating the construction of machine images.

Here we’re going to be looking at the the idea of applying automation tools to the wider product development process. Tools that help you do this are part of a collection known as “Infrastructure as Code”, which refers to the the provisioning of compute instances (physical machines and their operating systems) and software applications using revision-controlled machine-readable text files.