Apptainer Containers & Spack Packages

HPC

Containers

Published

December 16, 2022

Modified

December 22, 2022

Base Image

Apptainer is a widely use container-engine in (HPC) high-performance computing. Spack is a package manager primarily developed to accommodate the installation of applications for high-performance compute-clusters. Using Spack in a container image is therefore a very commen use-case.

Following examples builds an Apptainer Spack container and illustrate how to use such a container for staged builds. Typically users would add a specific software package using following container as base layer.

Container definition file apptainer.def:

# vim:ft=sh
BootStrap: docker
From: quay.io/fedora/fedora:37

%labels
Version Spack 0.19

%environment
export SPACK_ROOT=/srv/spack

%post
export SPACK_ROOT=/srv/spack
export SPACK_VERSION=v0.19

# Spack dependencies...
dnf install -y @development-tools \
      bat curl exa fd-find fzf findutils \
      gcc-c++ gcc gcc-gfortran git git-delta gnupg2 \
      hostname iproute netcat neovim make patch \
      python3 python3-pip python3-setuptools \
      python3-boto3 psmisc rsync tmux tree wget unzip

# clone the Spack source code repository
git clone -c feature.manyFiles=true https://github.com/spack/spack.git $SPACK_ROOT
(cd $SPACK_ROOT ; git checkout releases/$SPACK_VERSION)
rm -rf $SPACK_ROOT/.git

# add Spack to the shell environment
source $SPACK_ROOT/share/spack/setup-env.sh

# Setup spack bootstrap...
spack bootstrap root /srv/spack-bootstrap
spack bootstrap now

# find the compiles from the host container
spack compiler find --scope system $(which gcc)
spack compiler find --scope system $(which g++)
spack compiler find --scope system $(which gfortran)

# configure the Spack installation location
spack config add config:install_tree:/srv/spack-store

# make sure Spack is loaded when initializing Bash
ln -s $SPACK_ROOT/share/spack/setup-env.sh /etc/profile.d/spack.sh

%runscript
if ! [ $# -gt 0 ]
then
      /bin/bash --rcfile /etc/profile -l
else
      /bin/bash --rcfile /etc/profile -l -c "$@"
fi

Spack directories in the container image…

/srv/spack …$SPACK_ROOT directory
/srv/spack-bootstrap…Spack bootstrap software
/srv/spack-store …install tree

Build a container image using the apptainer build sub-command…

# container build command
apptainer build apptainer.sif apptainer.def

Customization

The container image build in the previous section is meant to be used as base-image for derivative containers utilizing Spack to build and install additional software package.

Specify a container base image from the local file-system in your container definition file header:

Bootstrap: localimage
From: ../../spack/apptainer.sif
#...

Note that %post section does not load Spack into the environment by default.

Make sure to source the Spack environment setup script:

#...
%post

source /srv/spack/share/spack/setup-env.sh
#...

Spack Environments

Spack Environments are used to build, install and load a selection of Spack packages. This mechanism uses a configuration file called spack.yaml. This example creates a sub-directory spack-environment/ to store this file:

# for example...
mkdir spack-environment
cat > spack-environment/spack.yaml <<EOF
spack:
  specs:
    - neovim
EOF

Modify the container definition to…

…copy the spack-environment/ directory into the container.
Use spack env ... in the %post section for software build and installation.

%files
spack-environment/ /opt/spack-environment
#...
%post
# ...
# ...additional Spack environments
source /srv/spack/share/spack/setup-env.sh
spack env activate -d /opt/spack-environment/
spack install -j $(nproc) --fail-fast
spack gc
spack env activate --sh -d /opt/spack-environment  >> /etc/profile.d/spack_environment.sh

The last line above generates shell-profile that loads the software provided by the Spack environment when stating a Bash shell.

Spack Packages

Similar to the previous section it is possible to add a custom Spack package repository to a container. Create a sub-directory spack-packages/ to store the package repository:

mkdir -p spack-packages/packages
cat > spack-packages/repo.yaml <<EOF
repo:
  namespace: local
EOF
# ...add packages to the spack-packages/packages sub-directory

Modify the container definition to…

…copy the spack-packages/ directory into the container
…use the spack repo ... command in %post section to add the package repository to the Spack configuration

%files
spack-packages/ /opt/spack-packages
#...
%post
#...
source /srv/spack/share/spack/setup-env.sh
spack repo add /opt/spack-packages/

Note that this needs to be done before a Spack environment builds and installs packages defined in this repository.

Usage

Test spack using a temporary overlay include commands like spack concretize:

apptainer run --fakeroot --overlay $(mktemp -d) apptainer.sif

However it is not possible to run spack install. This would requires to use a Apptainer sandbox as described in the next section or to utilize an definition file with the apptainer build command.

Sandbox

Sandboxes are very useful to work interactively. However it is recommend to uses a sandbox primarily for testing the configuration of Spack in order to implement an Apptainer definition file.

# ...desposable path
cd $(mktemp -d /var/tmp/$USER-apptainer-XXXXXX)
export APPTAINER_TMPDIR=$PWD && export APPTAINER_CACHEDIR=$PWD

#...create a container within a writable directory
apptainer build --fix-perms --sandbox rootfs/ $HPC_CONTAINER_PATH/spack/0.19/apptainer.sif

#...make changes within the container
apptainer shell --writable --fakeroot --home $PWD rootfs

#...build a new container from the sandbox
apptainer build apptainer.sif rootfs/

Persistent Overlay

Following example decouples the file-system layer storing Spack from the base-image layer. This enables a writable overlay within the container image.

Build a basic container image…

…including Spack dependencies installed from the platform
…create an overlay file-system
…embed the overlay into the SIF file…

# ...work in temporary storage
cd $(mktemp -d /var/tmp/$USER-apptainer-XXXXXX)
export APPTAINER_TMPDIR=$PWD
export APPTAINER_CACHEDIR=$PWD

cat > apptainer.def <<EOF
BootStrap: docker
From: quay.io/fedora/fedora:37
%post
dnf install -y @development-tools \
      bat curl exa fd-find fzf findutils \
      gcc-c++ gcc gcc-gfortran git git-delta gnupg2 \
      hostname iproute netcat neovim make patch \
      python3 python3-pip python3-setuptools \
      python3-boto3 psmisc rsync tmux tree wget unzip
%runscript
if ! [ $# -gt 0 ]
then
      /bin/bash --rcfile /etc/profile -l
else
      /bin/bash --rcfile /etc/profile -l -c "$@"
fi
EOF

# ...container image with persitant overlay
apptainer build apptainer.sif apptainer.def
apptainer overlay create --fakeroot --size 40960 overlay.img
apptainer sif add --datatype 4 --partfs 2 --parttype 4 \
                  --partarch 2 --groupid 1 apptainer.sif overlay.img

# ...move the container to permanent storage

# ...start the container writable
apptainer run --writable --fakeroot apptainer.sif

Install to /app/spack…

# ..spack out of a prefix ...other than the git repositor
git clone -c feature.manyFiles=true https://github.com/spack/spack.git /tmp/spack
cd /tmp/spack \
      && git checkout releases/0.19 \
      && source share/spack/setup-env.sh
mkdir /app \
      && spack clone /app/spack \
      && cd /appt/spack \
      && rm -rf /tmp/spack

source /app/spack/share/spack/setup-env.sh

# ...setup bootstrapping store
spack bootstrap root /app/spack-bootstrap
spack bootstrap now

# find the compiles installed by package
spack compiler find --scope system $(which gcc)
spack compiler find --scope system $(which g++)
spack compiler find --scope system $(which gfortran)

--- title: Apptainer Containers & Spack Packages categories: - HPC - Containers date: 2022/12/16 date-modified: 2022/12/22 --- ## Base Image [Apptainer](https://apptainer.org/docs/user/main/) is a widely use container-engine in (HPC) high-performance computing. [Spack](https://spack.readthedocs.io) is a package manager primarily developed to accommodate the installation of applications for high-performance compute-clusters. Using Spack in a container image is therefore a very commen use-case. Following examples builds an Apptainer Spack container and illustrate how to use such a container for staged builds. Typically users would add a specific software package using following container as base layer. Container definition file `apptainer.def`: ```sh # vim:ft=sh BootStrap: docker From: quay.io/fedora/fedora:37 %labels Version Spack 0.19 %environment export SPACK_ROOT=/srv/spack %post export SPACK_ROOT=/srv/spack export SPACK_VERSION=v0.19 # Spack dependencies... dnf install -y @development-tools \ bat curl exa fd-find fzf findutils \ gcc-c++ gcc gcc-gfortran git git-delta gnupg2 \ hostname iproute netcat neovim make patch \ python3 python3-pip python3-setuptools \ python3-boto3 psmisc rsync tmux tree wget unzip # clone the Spack source code repository git clone -c feature.manyFiles=true https://github.com/spack/spack.git $SPACK_ROOT (cd $SPACK_ROOT ; git checkout releases/$SPACK_VERSION) rm -rf $SPACK_ROOT/.git # add Spack to the shell environment source $SPACK_ROOT/share/spack/setup-env.sh # Setup spack bootstrap... spack bootstrap root /srv/spack-bootstrap spack bootstrap now # find the compiles from the host container spack compiler find --scope system $(which gcc) spack compiler find --scope system $(which g++) spack compiler find --scope system $(which gfortran) # configure the Spack installation location spack config add config:install_tree:/srv/spack-store # make sure Spack is loaded when initializing Bash ln -s $SPACK_ROOT/share/spack/setup-env.sh /etc/profile.d/spack.sh %runscript if ! [ $# -gt 0 ] then /bin/bash --rcfile /etc/profile -l else /bin/bash --rcfile /etc/profile -l -c "$@" fi ``` Spack directories in the container image... * `/srv/spack` ...`$SPACK_ROOT` directory * `/srv/spack-bootstrap`...Spack bootstrap software * `/srv/spack-store` ...install tree Build a container image using the `apptainer build` sub-command... ```sh # container build command apptainer build apptainer.sif apptainer.def ``` ## Customization The container image build in the previous section is meant to be used as base-image for derivative containers utilizing Spack to build and install additional software package. Specify a **container base image from the local file-system** in your container [definition file header][dfh]: ```sh Bootstrap: localimage From: ../../spack/apptainer.sif #... ``` Note that [`%post` section][dfp] does not load Spack into the environment by default. Make sure to **`source` the Spack environment setup script**: ```sh #... %post source /srv/spack/share/spack/setup-env.sh #... ``` ### Spack Environments [Spack Environments][sev] are used to build, install and load a selection of Spack packages. This mechanism uses a **configuration file called `spack.yaml`**. This example creates a sub-directory `spack-environment/` to store this file: ```sh # for example... mkdir spack-environment cat > spack-environment/spack.yaml <<EOF spack: specs: - neovim EOF ``` Modify the container definition to... * ...copy the `spack-environment/` directory into the container. * Use `spack env ...` in the `%post` section for software build and installation. ```sh %files spack-environment/ /opt/spack-environment #... %post # ... # ...additional Spack environments source /srv/spack/share/spack/setup-env.sh spack env activate -d /opt/spack-environment/ spack install -j $(nproc) --fail-fast spack gc spack env activate --sh -d /opt/spack-environment >> /etc/profile.d/spack_environment.sh ``` The last line above generates **shell-profile that loads the software** provided by the Spack environment when stating a Bash shell. ### Spack Packages Similar to the previous section it is possible to add a custom Spack [package repository][spr] to a container. Create a sub-directory `spack-packages/` to store the package repository: ```sh mkdir -p spack-packages/packages cat > spack-packages/repo.yaml <<EOF repo: namespace: local EOF # ...add packages to the spack-packages/packages sub-directory ``` Modify the container definition to... * ...copy the `spack-packages/` directory into the container * ...use the `spack repo ...` command in `%post` section to add the package repository to the Spack configuration ```sh %files spack-packages/ /opt/spack-packages #... %post #... source /srv/spack/share/spack/setup-env.sh spack repo add /opt/spack-packages/ ``` Note that this needs to be done before a Spack environment builds and installs packages defined in this repository. ## Usage Test `spack` using a temporary overlay include commands like `spack concretize`: ```sh apptainer run --fakeroot --overlay $(mktemp -d) apptainer.sif ``` However it is not possible to run `spack install`. This would requires to use a Apptainer sandbox as described in the next section or to utilize an definition file with the `apptainer build` command. ### Sandbox Sandboxes are very useful to work interactively. However it is recommend to uses a sandbox **primarily for testing the configuration of Spack** in order to implement an Apptainer definition file. ```sh # ...desposable path cd $(mktemp -d /var/tmp/$USER-apptainer-XXXXXX) export APPTAINER_TMPDIR=$PWD && export APPTAINER_CACHEDIR=$PWD #...create a container within a writable directory apptainer build --fix-perms --sandbox rootfs/ $HPC_CONTAINER_PATH/spack/0.19/apptainer.sif #...make changes within the container apptainer shell --writable --fakeroot --home $PWD rootfs #...build a new container from the sandbox apptainer build apptainer.sif rootfs/ ``` ### Persistent Overlay Following example decouples the file-system layer storing Spack from the base-image layer. This enables a writable overlay within the container image. Build a basic container image... * ...including Spack dependencies installed from the platform * ...create an overlay file-system * ...embed the overlay into the SIF file... ```sh # ...work in temporary storage cd $(mktemp -d /var/tmp/$USER-apptainer-XXXXXX) export APPTAINER_TMPDIR=$PWD export APPTAINER_CACHEDIR=$PWD cat > apptainer.def <<EOF BootStrap: docker From: quay.io/fedora/fedora:37 %post dnf install -y @development-tools \ bat curl exa fd-find fzf findutils \ gcc-c++ gcc gcc-gfortran git git-delta gnupg2 \ hostname iproute netcat neovim make patch \ python3 python3-pip python3-setuptools \ python3-boto3 psmisc rsync tmux tree wget unzip %runscript if ! [ $# -gt 0 ] then /bin/bash --rcfile /etc/profile -l else /bin/bash --rcfile /etc/profile -l -c "$@" fi EOF # ...container image with persitant overlay apptainer build apptainer.sif apptainer.def apptainer overlay create --fakeroot --size 40960 overlay.img apptainer sif add --datatype 4 --partfs 2 --parttype 4 \ --partarch 2 --groupid 1 apptainer.sif overlay.img # ...move the container to permanent storage # ...start the container writable apptainer run --writable --fakeroot apptainer.sif ``` Install to `/app/spack`... ```sh # ..spack out of a prefix ...other than the git repositor git clone -c feature.manyFiles=true https://github.com/spack/spack.git /tmp/spack cd /tmp/spack \ && git checkout releases/0.19 \ && source share/spack/setup-env.sh mkdir /app \ && spack clone /app/spack \ && cd /appt/spack \ && rm -rf /tmp/spack source /app/spack/share/spack/setup-env.sh # ...setup bootstrapping store spack bootstrap root /app/spack-bootstrap spack bootstrap now # find the compiles installed by package spack compiler find --scope system $(which gcc) spack compiler find --scope system $(which g++) spack compiler find --scope system $(which gfortran) ``` [dfh]: https://apptainer.org/docs/user/main/definition_files.html#header [dfp]: https://apptainer.org/docs/user/main/definition_files.html#post [sev]: https://spack-tutorial.readthedocs.io/en/latest/tutorial_environments.html [spr]: https://spack.readthedocs.io/en/latest/repositories.html