Spack HPC Package Manager

HPC

Published

April 20, 2020

Modified

August 14, 2023

spack…package management…for high-performance computing…

Spack handles combinatorial software complexity…

…no root privileges required
…builds on top of any Linux distribution
…targeting a wide variety of hardware platforms
…multiple versions of the same package can coexist
…packages automatically find dependencies
- …embeds RPATH in binary
- …no LD_LIBRARY_PATH or module required
…package recipes written in Python
…dependency DAG (direct acyclic graph) hash appended to each prefix…
- …one configuration of each library per DAG
- …ensures ABI constancy

Documentation

Spack Project, Documentation, Source Code…

…on the command-line…

# list of sub-commands
spack help
# print help for a sub-command
spack help install
spack install --help

Installation

Install the Dependencies…

# Fedora
sudo dnf install -y @development-tools \
      curl findutils gcc-c++ gcc gcc-gfortran \
      git gnupg2 hostname iproute make patch \
      python3 python3-pip python3-setuptools \
      python3-boto3 unzip

…clone the Spack source code…

…alternate prefix
- …relocate Spack out of the Git repository

# ...aka install Spack
export SPACK_ROOT=$HOME/spack
git clone -c feature.manyFiles=true https://github.com/spack/spack.git $SPACK_ROOT

# ..spack out of a prefix
git clone -c feature.manyFiles=true https://github.com/spack/spack.git /tmp/spack
cd /tmp/spack \
      && git checkout releases/0.19 \
      && source share/spack/setup-env.sh \
      && spack clone $SPACK_ROOT \
      && cd - \
      && rm -rf /tmp/spack

spack command will be available in $SPACK_ROOT/bin/spack…

Use setup-script to add spack to the shell environment…

source $SPACK_ROOT/share/spack/setup-env.sh

…will put the spack command in your $PATH
…set up your $MODULEPATH to use Spack’s packages
…loading of modules may introduce a delay when starting a shell
- …disable by setting $SPACK_SKIP_MODULES

Interactive shell support…

# ...load Spack autmatically with your shell environment
cat > ~/.zshrc.d/spack.sh <<EOF
export SPACK_ROOT=$HOME/spack
# export SPACK_SKIP_MODULES=true
source $SPACK_ROOT/share/spack/setup-env.sh
EOF

`clingo` Bootstrap

bootstrap sub-command…

…spack -b find…
- …tools Spack needs for its own functioning
- …lives under the $HOME/.spack
…spack clean -b …remove bootstrap store

# (optional) change path to bootstraping store
spack bootstrap root /opt/spack/bootstrap

# ...bootstrap all the dependencies needed by Spack 
spack bootstrap now
# ...useful when building containers

Spack uses clingo…

…to resolve optimal versions and variants of dependencies
…two ways of bootstrapping clingo
- …pre-built binaries (default)
- …from sources requires…
  - …full install of Python including header files
  - …compiler with support for C++14…static C++ standard libraries

Configuration

Precedents…

Path	Scope	Description
`$SPACK_ROOT/etc/spack/defaults/`	default	…by upstream (do not modify)
`/etc/spack/`	system	…for all users on a system
`$SPACK_ROOT/etc/spack/`	site	…for a Spack installation
`~/.spack`	user	…specific to each user

# view the final “merged” version of any configuration
spack config get config
# ...use --scope option...

Configuration files…

config.yaml…Spack settings
modules.yaml…environment modules…LMod
compilers.yaml…available compilers
mirrors.yaml…source mirror configuration

# ...create a user specific configuration file...
cat > ~/.spack/config.yaml <<EOF
config:
  install_tree:
    root: /opt/spack-store
EOF

add sub-command can modify the configuration

spack config --scope system add config:install_tree:root:/opt/spack-store

Compilers

Compiler configuration…

… build packages with multiple compilers and compiler versions
…compiler sub-command to the configuration…
- …list & infoto inspect the configuration

# available compilers
spack compiler list

# details about a specific compiler
spack compiler info gcc@12.2.1

# ...manual configuration
spack config edit compilers

Completely remove the compiler configuration…

…delete ~/.spack/linux/compilers.yaml
…and run spack clean -b

# find the compiles installed by distirbution packages
spack compiler find --scope system $(which gcc)
spack compiler find --scope system $(which g++)
spack compiler find --scope system $(which gfortran)

Usage

Spack packages installation…

…deployed with the dependencies they built with (similar to Nix/Guix)
…contain platform & environment metadata…
- …compiler build options & optimization flags
- …target platform and microarchitecture
- …configuration for performance optimization

`list` & `find`

Search available packages with following sub-commands…

…list packages names matching a given string
…info …get detailed information on a particular package
…versions …software versions for a given package

>>> spack list gcc
armpl-gcc  gcc  gccmakedep  gccxml
==> 4 packages

>>> spack versions gcc
==> Safe versions (already checksummed):
  master  11.3.0  10.4.0  10.1.0  9.3.0  #... 
  12.2.0  11.2.0  10.3.0  9.5.0   9.2.0  #...
  12.1.0  11.1.0  10.2.0  9.4.0   9.1.0  #...

find shows the specs of installed packages…

…--long/-l or --very-long/-L output
…-d/--deps …dependencies along with found specs
…-p/--paths …show paths to package install directories

# ...software versions with dependency tree
>>> spack find -Ld hwloc
#...
-- linux-debian10-x86_64 / gcc@10.2.0 ---------------------------
rpajza4w5yeshrr7qnbtwiz7cgsc5y4l hwloc@2.8.0
dcbydon6ydygdx25zfodcnxet7xjbvov     libpciaccess@0.16
cwyytdnlhwlxzrsse5j4mar2jo6g72mr         libtool@2.4.7
yhcwefsupb3ej5ifugxw3dhzdxxgo2hj             m4@1.4.19
#....
uemmoahqov5jnjgtbzgjwh4gin47zj4o     libxml2@2.10.1
r6z5bvhxgq7syjhkok6uiulvpxut5fab         libiconv@1.16
#...
nrqlsr5xfwbf2fvne2vfhcxbcuft27lk     pkgconf@1.8.0

# ...absolute path to software
>>> spack find --path hwloc
-- linux-debian10-x86_64 / gcc@10.2.0 ---------------------------
hwloc@2.8.0  /opt/spack/linux-debian10-x86_64/gcc-10.2.0/hwloc-2.8.0-rpajza #...

diff helps to disambiguate to version of a spec

`install`

Install and uninstall packages:

spack install $name                   # install package (prefered versions)
spack install $name@$version [$spec]  # install specific version
spack uninstall $name                 # uninstall package..
spack uninstall --dependents ...      # ..including every packages that depend on
spack clean -a                        # remove temporary build files
spack gc -y                           # remove build time dependencies

Optional version specifier…

…Specs & Dependencies
…concretization fills missing details…
- …if not specified by user
- spack spec $name to inspect
…spec.yaml stores detailed provenance with installed package

spack install mpileaks                             # unconstrained
spack install mpileaks@3.3                         # @ custom version
spack install mpileaks@3.3 %gcc@4.7.3              #  % custom compiler
spack install mpileaks@3.3 %gcc@4.7.3 +threads     # +/- build option
spack install mpileaks@3.3 cppflags="-O3 –g3"      # set compiler flags
spack install mpileaks@3.3 target=skylake          # set target microarchitecture
spack install mpileaks@3.3 ^mpich@3.2 %gcc@4.9.3   # ^ dependency information

Each expression is a spec for a particular configuration
Spec syntax is recursive

`load`

spack load adds a package to the environment…
- …executables are added to $PATH
- …man-pages to $MANPATH
spack unload removes a package from the environment
spack load --list…list already loaded packages
- …same as spack find --loaded

# system-wide package load
spack load --sh $package > /etc/profile.d/$package.sh    # Bash
spack load --sh $package >> /etc/zshenv                  # Zsh
# user-specific package load
spack load --sh $package >> ~/.bashrc                    # Bash
spack load --sh $package >> ~/.zshrc                     # Zsh

`stage` & `cd`

Stage a package and modify before build…

…spack stage…downloaded archive in preparation for install
…spack cd…to pertinent directories
- …-P…top-level packages directory
- …-p…directory enclosing a spec’s package.py file
- …-i install…-b build directories
…spack location…same as spack cd

spack stage readline
spack cd readline ; pwd

Add a package to your $PATH…

export PATH=`spack location --install-dir curl`/bin:$PATH

Binary Cache

Spack Rolling Binary Cache…

…alleviates the need to recompile it from source
…store pre-built versions of common libraries and application
…contain builds for multiple compilers and architectures

spack mirror add binary_mirror https://binaries.spack.io/develop
spack buildcache keys --install --trust

E4S Build Cache for Spack
- https://oaciss.uoregon.edu/e4s/inventory.html

Modules

Spack Tutorial - Module Files

# add support for hierarchical and non-hierarchical module file layouts
spack install lmod
# generate the environment load file
source $(spack location -i lmod)/lmod/lmod/init/bash
# load environment modules
source $SPACK_ROOT/share/spack/setup-env.sh
# list available modules
module avail

Environments

Spack Environments uses to…

…group together a set of specs
- …additional level of configuration scope
…building, rebuilding and deploying in a coherent fashion
…separate the steps of…
- …choosing what to install
- …concretizing
- …installing
…loaded as a whole into the user environment

Required files …

spack.yaml …input file
spack.lock …lock file

Any directory can be treated as an environment

# ...basic example
cat > spack.yaml <<EOF
spack:
  specs:
    - neovim
EOF

# ...activate the environment
spack env activate -d .

# ...concretized the environments
spack concretize

# ...install all specs from the environment
spack install -j $(nproc)

# ...genreate script to load the environment
spack env loads -r

Package Repositories

Packages are installation scripts including…

…where to find and how to retrieve a software
…a list of related software dependencies
…commands and options to build a software

…built-in packages…$SPACK_ROOT/var/spack/repos/builtin

Package repositories allow…

…user to maintain a packages separately from (upstream) Spack
…share packages without relation to the built-in Spack packages
…override built-in Spack packages

Directory Structure

Package repository directory structured…

repo.yaml…configuration metadata
package/…sub-directories for each package
- …each package directory contains a package.py
- …patches or other files needed to build the package

repo.yaml
packages/
    nano/
        package.py
        nano-bugfix.patch
    ...

Every repository has an associated namespace..

builtin for the built-in repository
Spack records the repository namespace of packages…
- …packages with the same name from different namespaces possible
- …choose a namespace name that uniquely identifies a site
- spack find -N display the packages with their namespaces
- spack spec -N list namespaces before build

# repo.yaml
repo:
  namespace: example.org

Spack prevents registering two repositories with the same namespace

Local Repositories

Local repositories configured with repos.yaml…

# contains a single ordered list of paths to repositories
repos:
- ~/spack-packages
- /opt/spack-packages
- $spack/var/spack/repos/builtin

spack repo…manage package source repositories

# typically a package repository is maintained in VCS
git clone $uri /opt/spack-packages/
# add the package repository to the environment
spack repo add /opt/spack-packages/
# list configured repositories
spack repo list

Package Creation

References…

Two key parts of Spack…

…specs…expressions for describing builds of software
- …in a way that a package author can understand
…packages…Python modules that describe how to build software
- …according to a spec provided by a user
- …encapsulate the build logic for…
  - …different versions
  - …dependency combinations
  - …compilers (and compiler options)
  - …platforms (for performance optimization)

Procedure & Phases

Package installation procedure…series of predefined steps…

…fetch source code
…expand source in stage directory
…set stage directory as working directory
…fork a new build environment
…execute installation process…

Installation process…dependent on the build system…

…predefined steps for each build system are called “phases”
…spack info --phases lists phases of a package
…read available build systems and phases

Package Recipes

Two ways of writing a package recipe…

…single class..for metadata (directives, etc.) and build behavior
…explicit builder class…
- …to use more than one build system in a single package recipe
- …read section about multiple build systems

Create a Spack package repository…

…in the user home-directory for development
…read about package repositories above

mkdir -p $HOME/spack-repository/packages
cat > $HOME/spack-repository/repo.yaml <<EOF
repo:
  namespace: example.org
EOF
spack repo add $HOME/spack-repository
spack repo list

spack create …for a new package recipe file…

…first argument…URL of package archive
…option -N…specify a namespace for the package
…opens editor with package recipe package.py
…default to a package recipe using autotools
- …read the list of supported build systems

# example for the GNU Hello program
spack create -N example.org https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz
# ...open the recipe again
spack edit hello
# ...build the package
spack install hello

Adjust the package recipe…

…add description
…adjust homepage
…add list of maintainers
…add depends_on() calls for software dependencies
…configure the build phases…

…variants…enable optional features

Example for the GNU Hello package recipe…

from spack.package import *

class Hello(AutotoolsPackage):
    """The GNU Hello program produces a familiar, friendly greeting."""

    homepage = "https://www.gnu.org/software/hello/"
    url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"

    version("2.12.1", sha256="8d99142afd92576f30b0cd7cb42a8dc6809998bc5d607d88761f512e26c7db20")

    depends_on("iconv")
    depends_on("tar", type="run")
    depends_on("gzip", type="run")

    def configure_args(self):
        args = [
            "--with-libiconv-prefix={0}".format(self.spec["iconv"].prefix)
        ]
        return args

Build manually…

# ...move to the build directory
spack cd hello
# ...ensure the environment is properly set up
spack build-env hello $SHELL

Clean up…

spack uninstall -ay hello
spack repo remove example.com

References

Talks…

How the Spack package manager tames the stat storm, FOSDEM’2023
Keeping the HPC ecosystem working with Spack CI, FOSDEM’23
Automating a rolling binary release for Spack, FOSDEM’23

--- title: Spack HPC Package Manager categories: - HPC date: 2020/04/20 date-modified: 2023/08/14 --- `spack`...package management...for high-performance computing... Spack handles combinatorial software complexity... - ...no root privileges required - ...builds on top of any Linux distribution - ...targeting a wide variety of hardware platforms - ...multiple versions of the same package can coexist - ...packages automatically find dependencies - ...embeds RPATH in binary - ...no `LD_LIBRARY_PATH` or `module` required - ...package recipes written in Python - ...dependency DAG (direct acyclic graph) hash appended to each prefix... - ...one configuration of each library per DAG - ...ensures ABI constancy # Documentation Spack Project, Documentation, Source Code... - <https://spack.io> - <https://spack.readthedocs.io> - <https://spack-tutorial.readthedocs.io> - <https://github.com/spack/spack> ...on the command-line... ```sh # list of sub-commands spack help # print help for a sub-command spack help install spack install --help ``` # Installation Install the [Dependencies](https://spack.readthedocs.io/en/latest/getting_started.html#system-prerequisites)... ```bash # Fedora sudo dnf install -y @development-tools \ curl findutils gcc-c++ gcc gcc-gfortran \ git gnupg2 hostname iproute make patch \ python3 python3-pip python3-setuptools \ python3-boto3 unzip ``` ...clone the Spack source code... * ...[alternate prefix](https://spack.readthedocs.io/en/latest/getting_started.html#optional-alternate-prefix) * ...relocate Spack out of the Git repository ```sh # ...aka install Spack export SPACK_ROOT=$HOME/spack git clone -c feature.manyFiles=true https://github.com/spack/spack.git $SPACK_ROOT # ..spack out of a prefix git clone -c feature.manyFiles=true https://github.com/spack/spack.git /tmp/spack cd /tmp/spack \ && git checkout releases/0.19 \ && source share/spack/setup-env.sh \ && spack clone $SPACK_ROOT \ && cd - \ && rm -rf /tmp/spack ``` `spack` command will be available in `$SPACK_ROOT/bin/spack`... Use setup-script to add `spack` to the shell environment... ```sh source $SPACK_ROOT/share/spack/setup-env.sh ``` - ...will put the `spack` command in your `$PATH` - ...set up your `$MODULEPATH` to use Spack’s packages - ...loading of modules may introduce a delay when starting a shell - ...disable by setting `$SPACK_SKIP_MODULES` [Interactive shell support](https://spack.readthedocs.io/en/latest/packaging_guide.html#interactive-shell-support)... ```sh # ...load Spack autmatically with your shell environment cat > ~/.zshrc.d/spack.sh <<EOF export SPACK_ROOT=$HOME/spack # export SPACK_SKIP_MODULES=true source $SPACK_ROOT/share/spack/setup-env.sh EOF ``` ## `clingo` Bootstrap [`bootstrap`](https://spack.readthedocs.io/en/latest/bootstrapping.html) sub-command... - ...`spack -b find`... - ...tools Spack needs for its own functioning - ...lives under the `$HOME/.spack` - ...`spack clean -b` ...remove bootstrap store ```sh # (optional) change path to bootstraping store spack bootstrap root /opt/spack/bootstrap # ...bootstrap all the dependencies needed by Spack spack bootstrap now # ...useful when building containers ``` Spack uses `clingo`... - ...to resolve optimal versions and variants of dependencies - ...two ways of [bootstrapping `clingo`](https://spack.readthedocs.io/en/latest/getting_started.html#bootstrapping-clingo) - ...pre-built binaries (default) - ...from sources requires... - ...full install of Python including header files - ...compiler with support for C++14...static C++ standard libraries ## Configuration Precedents... Path | Scope | Description -------------------------------------|---------|---------------------- `$SPACK_ROOT/etc/spack/defaults/` | default | ...by upstream (do not modify) `/etc/spack/` | system | ...for all users on a system `$SPACK_ROOT/etc/spack/` | site | ...for a Spack installation `~/.spack` | user | ...specific to each user ```sh # view the final “merged” version of any configuration spack config get config # ...use --scope option... ``` [Configuration files](https://spack.readthedocs.io/en/latest/configuration.html)... - `config.yaml`...Spack settings - `modules.yaml`...environment modules...LMod - `compilers.yaml`...available compilers - `mirrors.yaml`...source mirror configuration ```sh # ...create a user specific configuration file... cat > ~/.spack/config.yaml <<EOF config: install_tree: root: /opt/spack-store EOF ``` `add` sub-command can modify the configuration ```sh spack config --scope system add config:install_tree:root:/opt/spack-store ``` ## Compilers [Compiler configuration](https://spack.readthedocs.io/en/latest/getting_started.html#compiler-configuration)... * ... build packages with multiple compilers and compiler versions * ...`compiler` sub-command to the configuration... * ...`list` & `info`to inspect the configuration ```sh # available compilers spack compiler list # details about a specific compiler spack compiler info gcc@12.2.1 # ...manual configuration spack config edit compilers ``` Completely remove the compiler configuration... * ...delete `~/.spack/linux/compilers.yaml` * ...and run `spack clean -b` ```sh # find the compiles installed by distirbution packages spack compiler find --scope system $(which gcc) spack compiler find --scope system $(which g++) spack compiler find --scope system $(which gfortran) ``` # Usage Spack packages installation... - ...deployed with the dependencies they built with (similar to Nix/Guix) - ...contain platform & environment metadata... - ...compiler build options & optimization flags - ...target platform and microarchitecture - ...configuration for performance optimization ## `list` & `find` Search available packages with following sub-commands... - ...`list` packages names matching a given string - ...`info` ...get detailed information on a particular package - ...`versions` ...software versions for a given package ```sh >>> spack list gcc armpl-gcc gcc gccmakedep gccxml ==> 4 packages >>> spack versions gcc ==> Safe versions (already checksummed): master 11.3.0 10.4.0 10.1.0 9.3.0 #... 12.2.0 11.2.0 10.3.0 9.5.0 9.2.0 #... 12.1.0 11.1.0 10.2.0 9.4.0 9.1.0 #... ``` `find` shows the specs of installed packages... - ...`--long`/`-l` or `--very-long`/`-L` output - ...`-d`/`--deps` ...dependencies along with found specs - ...`-p`/`--paths` ...show paths to package install directories ```sh # ...software versions with dependency tree >>> spack find -Ld hwloc #... -- linux-debian10-x86_64 / gcc@10.2.0 --------------------------- rpajza4w5yeshrr7qnbtwiz7cgsc5y4l hwloc@2.8.0 dcbydon6ydygdx25zfodcnxet7xjbvov libpciaccess@0.16 cwyytdnlhwlxzrsse5j4mar2jo6g72mr libtool@2.4.7 yhcwefsupb3ej5ifugxw3dhzdxxgo2hj m4@1.4.19 #.... uemmoahqov5jnjgtbzgjwh4gin47zj4o libxml2@2.10.1 r6z5bvhxgq7syjhkok6uiulvpxut5fab libiconv@1.16 #... nrqlsr5xfwbf2fvne2vfhcxbcuft27lk pkgconf@1.8.0 # ...absolute path to software >>> spack find --path hwloc -- linux-debian10-x86_64 / gcc@10.2.0 --------------------------- hwloc@2.8.0 /opt/spack/linux-debian10-x86_64/gcc-10.2.0/hwloc-2.8.0-rpajza #... ``` `diff` helps to disambiguate to version of a spec ## `install` Install and uninstall [packages](https://spack.readthedocs.io/en/latest/package_list.html): ```bash spack install $name # install package (prefered versions) spack install $name@$version [$spec] # install specific version spack uninstall $name # uninstall package.. spack uninstall --dependents ... # ..including every packages that depend on spack clean -a # remove temporary build files spack gc -y # remove build time dependencies ``` Optional **version specifier**... - ...[Specs & Dependencies](https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies) - ...concretization fills missing details... - ...if not specified by user - `spack spec $name` to inspect - ...`spec.yaml` stores detailed provenance with installed package ```bash spack install mpileaks # unconstrained spack install mpileaks@3.3 # @ custom version spack install mpileaks@3.3 %gcc@4.7.3 # % custom compiler spack install mpileaks@3.3 %gcc@4.7.3 +threads # +/- build option spack install mpileaks@3.3 cppflags="-O3 –g3" # set compiler flags spack install mpileaks@3.3 target=skylake # set target microarchitecture spack install mpileaks@3.3 ^mpich@3.2 %gcc@4.9.3 # ^ dependency information ``` * Each expression is a spec for a particular configuration * Spec syntax is recursive ## `load` - `spack load` adds a package to the environment... - ...executables are added to `$PATH` - ...man-pages to `$MANPATH` - `spack unload` removes a package from the environment - `spack load --list`...list already loaded packages - ...same as `spack find --loaded` ```sh # system-wide package load spack load --sh $package > /etc/profile.d/$package.sh # Bash spack load --sh $package >> /etc/zshenv # Zsh # user-specific package load spack load --sh $package >> ~/.bashrc # Bash spack load --sh $package >> ~/.zshrc # Zsh ``` ## `stage` & `cd` Stage a package and modify before build... - ...`spack stage`...downloaded archive in preparation for install - ...`spack cd`...to pertinent directories - ...`-P`...top-level packages directory - ...`-p`...directory enclosing a spec's `package.py` file - ...`-i` install...`-b` build directories - ...`spack location`...same as `spack cd` ```sh spack stage readline spack cd readline ; pwd ``` Add a package to your `$PATH`... ```sh export PATH=`spack location --install-dir curl`/bin:$PATH ``` ## Binary Cache Spack Rolling Binary Cache... - ...alleviates the need to recompile it from source - ...store pre-built versions of common libraries and application - ...contain builds for multiple compilers and architectures ```sh spack mirror add binary_mirror https://binaries.spack.io/develop spack buildcache keys --install --trust ``` - E4S Build Cache for Spack - <https://oaciss.uoregon.edu/e4s/inventory.html> # Modules [Spack Tutorial - Module Files](https://spack-tutorial.readthedocs.io/en/latest/tutorial_modules.html) ```bash # add support for hierarchical and non-hierarchical module file layouts spack install lmod # generate the environment load file source $(spack location -i lmod)/lmod/lmod/init/bash # load environment modules source $SPACK_ROOT/share/spack/setup-env.sh # list available modules module avail ``` # Environments [Spack Environments](https://spack-tutorial.readthedocs.io/en/latest/tutorial_environments.html) uses to... * ...group together a set of specs * ...additional level of configuration scope * ...building, rebuilding and deploying in a coherent fashion * ...separate the steps of... * ...choosing what to install * ...concretizing * ...installing * ...loaded as a whole into the user environment Required files ... * `spack.yaml` ...input file * `spack.lock` ...lock file Any directory can be treated as an environment ```sh # ...basic example cat > spack.yaml <<EOF spack: specs: - neovim EOF # ...activate the environment spack env activate -d . # ...concretized the environments spack concretize # ...install all specs from the environment spack install -j $(nproc) # ...genreate script to load the environment spack env loads -r ``` # Package Repositories {#spack:package:repositories} Packages are installation scripts including... - ...where to find and how to retrieve a software - ...a list of related software dependencies - ...commands and options to build a software ...built-in packages...`$SPACK_ROOT/var/spack/repos/builtin` [Package repositories](https://spack.readthedocs.io/en/latest/repositories.html) allow... - ...user to maintain a packages separately from (upstream) Spack - ...share packages without relation to the built-in Spack packages - ...override built-in Spack packages ## Directory Structure Package repository directory structured... - `repo.yaml`...configuration metadata - `package/`...sub-directories for each package - ...each package directory contains a `package.py` - ...patches or other files needed to build the package ```sh repo.yaml packages/ nano/ package.py nano-bugfix.patch ... ``` Every repository has an associated **namespace**.. - `builtin` for the built-in repository - Spack records the repository namespace of packages... - ...packages with the same name from different namespaces possible - ...choose a namespace name that uniquely identifies a site - `spack find -N` display the packages with their namespaces - `spack spec -N` list namespaces before build ```sh # repo.yaml repo: namespace: example.org ``` Spack prevents registering two repositories with the same namespace ## Local Repositories Local repositories configured with `repos.yaml`... ```sh # contains a single ordered list of paths to repositories repos: - ~/spack-packages - /opt/spack-packages - $spack/var/spack/repos/builtin ``` `spack repo`...manage package source repositories ```sh # typically a package repository is maintained in VCS git clone $uri /opt/spack-packages/ # add the package repository to the environment spack repo add /opt/spack-packages/ # list configured repositories spack repo list ``` # Package Creation References... - <https://spack.readthedocs.io/en/latest/packaging_guide.html> - <https://spack-tutorial.readthedocs.io/en/latest/tutorial_packaging.html> Two key parts of Spack... - ...**specs**...expressions for describing builds of software - ...in a way that a package author can understand - ...**packages**...Python modules that describe how to build software - ...according to a spec provided by a user - ...encapsulate the build logic for... - ...different versions - ...dependency combinations - ...compilers (and compiler options) - ...platforms (for performance optimization) ## Procedure & Phases Package installation procedure...**series of predefined steps**... 1. ...fetch source code 2. ...expand source in stage directory 3. ...set stage directory as working directory 4. ...fork a new build environment 5. ...execute installation process... Installation process...**dependent on the build system**... - ...predefined steps for each build system are called “phases” - ...`spack info --phases` lists phases of a package - ...read [available build systems and phases](https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-process) ## Package Recipes Two ways of writing a package recipe... - ...**single class**..for metadata (directives, etc.) and build behavior - ...explicit **builder class**... - ...to use more than one build system in a single package recipe - ...read section about [multiple build systems](https://spack.readthedocs.io/en/latest/packaging_guide.html#multiple-build-systems) Create a Spack package repository... - ...in the user home-directory for development - ...read about [package repositories](#spack:package:repositories) above ```sh mkdir -p $HOME/spack-repository/packages cat > $HOME/spack-repository/repo.yaml <<EOF repo: namespace: example.org EOF spack repo add $HOME/spack-repository spack repo list ``` `spack create` ...for a new package recipe file... - ...first argument...URL of package archive - ...option `-N`...specify a namespace for the package - ...opens editor with package **recipe `package.py`** - ...default to a package recipe using `autotools` - ...read the list of supported [build systems](https://spack.readthedocs.io/en/latest/build_systems.html) ```sh # example for the GNU Hello program spack create -N example.org https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz # ...open the recipe again spack edit hello # ...build the package spack install hello ``` Adjust the package recipe... 1. ...add description 2. ...adjust `homepage` 3. ...add list of maintainers 4. ...add `depends_on()` calls for software [dependencies](https://spack.readthedocs.io/en/latest/packaging_guide.html#dependencies) 5. ...configure the build phases... - ...[variants](https://spack.readthedocs.io/en/latest/packaging_guide.html#variants)...enable optional features Example for the GNU Hello package recipe... ```python from spack.package import * class Hello(AutotoolsPackage): """The GNU Hello program produces a familiar, friendly greeting.""" homepage = "https://www.gnu.org/software/hello/" url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz" version("2.12.1", sha256="8d99142afd92576f30b0cd7cb42a8dc6809998bc5d607d88761f512e26c7db20") depends_on("iconv") depends_on("tar", type="run") depends_on("gzip", type="run") def configure_args(self): args = [ "--with-libiconv-prefix={0}".format(self.spec["iconv"].prefix) ] return args ``` Build manually... ```sh # ...move to the build directory spack cd hello # ...ensure the environment is properly set up spack build-env hello $SHELL ``` Clean up... ```sh spack uninstall -ay hello spack repo remove example.com ``` # References Talks... * [How the Spack package manager tames the stat storm](https://fosdem.org/2023/schedule/event/spack_stat_storm/), FOSDEM'2023 * [Keeping the HPC ecosystem working with Spack CI](https://fosdem.org/2023/schedule/event/spack_ci/), FOSDEM'23 * [Automating a rolling binary release for Spack](https://fosdem.org/2023/schedule/event/rolling_release_spack/), FOSDEM'23