Supercomputers – As fancy as they sound are difficult to manage and operate. Being a systems manager for a supercomputing facility is inherently tasking. They are constantly planning, evaluating, deploying, fixing some part of the tech and most of the times simultaneously. Feels like running a marathon? Yes, they run a few everyday! They should be rightfully called Super Admins. One of the deliverables in the list of all things to be managed is application delivery and keeping the software stack up to date – the user facing applications, the required runtime libraries and the tool chains involved.
It is a general practice in most HPC Centers to build the applications from source to get the best performance on a given hardware. Building HPC applications from source comes with a long list of runtime libraries. They also need latest tool chains to take advantage of the instruction set support. It takes a lot of effort and time to acquire the know-hows to build optimal and performant HPC application softwares from source, especially when supporting users from various domains which is not uncommon. To keep up with the software releases sets the difficulty level to legend.
As the super admins always do, they turn to automation to solve this problem. Continuous Integration and Continuous Delivery (CICD) tools like Jenkins look to be a natural choice and many HPC Centers are using them in some form. However, tools like EasyBuild and Spack are becoming a popular choice for application delivery in HPC. This post explores the possibilities with EasyBuild
Prerequisites
If you are spinning up a new virtual machine to try Easybuild like me, vagrant could save you some time. The steps provided below works with CentOS 7.4. Make sure you have python and Environment Modules packages installed.
yum install -y python \ environment-modules Lmod-6.5.1-2.el7.x86_64 easy_install patch . /etc/profile.d/modules.sh # Bootstrap Module Environment
Installation
wget https://raw.githubusercontent.com/easybuilders/easybuild-framework/develop/easybuild/scripts/bootstrap_eb.py export EASYBUILD_PREFIX=/share/apps python bootstrap_eb.py $EASYBUILD_PREFIX
Note: It is recommended to install easybuild as an unprivileged user
Deploying GROMACS with easybuild
GROMACS is an open source molecular dynamics simulation software. Building GROMACS from source is considered fairly straight-forward and is documented well. Yet you will see at the end of the installation the sheer number of libraries and tool chain dependencies that would be required to install one version of GROMACS.
Make sure you have all the defaults properly set in $HOME/.bashrc . I had the following two lines added to my setup
. /etc/profile.d/modules.sh export EASYBUILD_PREFIX=/share/apps
Add the location of module files to be used
module use /usr/share/lmod/lmod/modulefiles/Core module use /share/apps/modules/all
Check if modules environment is initialised properly
[vagrant@easybuild ~]$ module avail ------------------- /usr/share/lmod/lmod/modulefiles/Core ---------------------- lmod/6.5.1 (D) settarg/6.5.1 (D) ---------------- /share/apps/modules/all --------------------- EasyBuild/3.6.2
Load EasyBuild and Lmod to the environment
[vagrant@easybuild ~]$ module load EasyBuild/3.6.2 lmod/6.5.1 [vagrant@easybuild ~]$ module list Currently Loaded Modules: 1) EasyBuild/3.6.2 2) lmod/6.5.1
Search and select the EasyBlock recipe to be used to build GROMACS. I am filtering the output for easyblocks that use GCC compilers
[vagrant@easybuild ~]$ eb -S GROMACS | grep -i foss * $CFGS1/g/GROMACS/GROMACS-2016-foss-2016b-hybrid.eb * $CFGS1/g/GROMACS/GROMACS-2016-foss-2016b-mt.eb * $CFGS1/g/GROMACS/GROMACS-2016.1-foss-2017a-PLUMED.eb * $CFGS1/g/GROMACS/GROMACS-2016.2-foss-2017a.eb * $CFGS1/g/GROMACS/GROMACS-2016.3-foss-2016b-GPU-enabled.eb * $CFGS1/g/GROMACS/GROMACS-2016.3-foss-2017a.eb * $CFGS1/g/GROMACS/GROMACS-2018-foss-2018a.eb * $CFGS1/g/GROMACS/GROMACS-2018.2-foss-2018b.eb * $CFGS1/g/GROMACS/GROMACS-2018.2-fosscuda-2018b.eb * $CFGS1/g/GROMACS/GROMACS-5.1.2-foss-2015b-hybrid.eb * $CFGS1/g/GROMACS/GROMACS-5.1.2-foss-2016a-hybrid.eb * $CFGS1/g/GROMACS/GROMACS-5.1.2-foss-2016a-mt.eb * $CFGS1/g/GROMACS/GROMACS-5.1.4-foss-2016b-hybrid.eb * $CFGS1/g/GROMACS/GROMACS-5.1.4-foss-2016b-mt.eb
I have selected GROMACS-2018.2-foss-2018b.eb. It will install GROMACS version 2018.2 and the build generates binaries with MPI and thread support, uses open source compilers and libraries. Start the build process with the following command
eb GROMACS-2018.2-foss-2018b.eb -r
It would have been a fitting finish to the title if I had to conclude the build and install process here. But, there were a few hiccups that I suppose are very specific to my test environment.
1. The VM had only two CPUs that were not enough to run the parallel tests and sanity checks. I had to edit the following files to turn off the tests
$EASYBUILD_PREFIX/software/EasyBuild/3.6.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.6.2-py2.7.egg/easybuild/easyconfigs/g/GROMACS/GROMACS-2018.2-foss-2018b.eb
$EASYBUILD_PREFIX/software/EasyBuild/3.6.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.6.2-py2.7.egg/easybuild/easyconfigs/f/FFTW/FFTW-3.3.8-gompi-2018b.eb
and insert the following code
skipsteps=['test']
2. The highest compiler optimisation levels imposed by EasyBuild were not compatible on my VMs. I could quickly fallback to a safer optimisation level by passing the –optarch=GCC:GENERIC option to the build command
eb GROMACS-2018.2-foss-2018b.eb -r --optarch=GCC:GENERIC
3. Automake required a newer version of PERL? I used easyblock to install and load the latest PERL. I made sure I am using the same version of GCC build PERL
eb Perl-5.28.0-GCCcore-7.3.0.eb -r
It might looks like a lot of work. But at the end, the build was successful and relatively was faster and took little effort.
[vagrant@easybuild ~]$ module purge # Purge all loaded modules from environment [vagrant@easybuild ~]$ module list # List loaded modules No modules loaded [vagrant@easybuild ~]$ module load GROMACS/2018.2-foss-2018b [vagrant@easybuild ~]$ module list Currently Loaded Modules: 1) GCCcore/7.3.0 2) binutils/2.30-GCCcore-7.3.0 3) GCC/7.3.0-2.30 4) zlib/1.2.11-GCCcore-7.3.0 5) numactl/2.0.11-GCCcore-7.3.0 6) hwloc/1.11.10-GCCcore-7.3.0 7) OpenMPI/3.1.1-GCC-7.3.0-2.30 8) OpenBLAS/0.3.1-GCC-7.3.0-2.30 9) gompi/2018b 10) FFTW/3.3.8-gompi-2018b 11) ScaLAPACK/2.0.2-gompi-2018b-OpenBLAS-0.3.1 12) foss/2018b 13) GROMACS/2018.2-foss-2018b
And that is a whole lot of dependencies automagically installed, configured with appropriate module files in place and ready to use. EasyBuild has easyblocks for more than a 1000 applications in the HPC space now. If all it takes is a line of code, end users have the power to experience the ultimate application performance – self built, built easy. And perhaps let the super admins save the world!
More on Application Delivery as the exploration continues.