Uberenv

Uberenv automates using Spack to build and deploy software.

Many projects leverage Spack to help build the software dependencies needed to develop and deploy their projects on HPC systems. Uberenv is a python script that helps automate using Spack to build third-party dependencies for development and to deploy Spack packages.

Uberenv was released as part of Conduit (https://github.com/LLNL/conduit/). It is included in-source in several projects. The https://github.com/llnl/uberenv/ repo is used to hold the latest reference version of Uberenv.

uberenv.py

uberenv.py is a single file python script that automates fetching spack, building and installing third party dependencies, and can optionally install packages as well. To automate the full install process, uberenv.py uses a target Spack package along with extra settings such as Spack compiler and external third party package details for common HPC platforms.

uberenv.py is included directly in a project’s source code repo in the folder: scripts/uberenv/ This folder is also used to store extra Spack and Uberenv configuration files unique to the target project. uberenv.py uses a project.json file to specify project details, including the target Spack package name and which Spack repo is used. Conduit’s source repo serves as an example for Uberenv and Spack configuration files, etc:

https://github.com/LLNL/conduit/tree/master/scripts/uberenv

uberenv.py is developed by LLNL in support of the Ascent, Axom, and Conduit projects.

Command Line Options

uberenv.py has a few options that allow you to control how dependencies are built:

Option Description Default
–prefix Destination directory uberenv_libs
–spec Spack spec linux: %gcc osx: %clang
–spack-config-dir Folder with Spack settings files linux: (empty) osx: scripts/uberenv/spack_configs/darwin/
-k Ignore SSL Errors False
–install Fully install target, not just dependencies False
–run_tests Invoke tests during build and against install False

The -k option exists for sites where SSL certificate interception undermines fetching from github and https hosted source tarballs. When enabled, uberenv.py clones spack using:

git -c http.sslVerify=false clone https://github.com/llnl/spack.git

And passes -k to any spack commands that may fetch via https.

Default invocation on Linux:

python scripts/uberenv/uberenv.py --prefix uberenv_libs \
                                  --spec %gcc

Default invocation on OSX:

python scripts/uberenv/uberenv.py --prefix uberenv_libs \
                                  --spec %clang \
                                  --spack-config-dir scripts/uberenv/spack_configs/darwin/

Use the --install option to install the target package (not just its development dependencies):

python scripts/uberenv/uberenv.py --install

If the target Spack package supports Spack’s testing hooks, you can run tests during the build process to validate the build and install, using the --run_tests option:

python scripts/uberenv/uberenv.py --install \
                                  --run_tests

For details on Spack’s spec syntax, see the Spack Specs & dependencies documentation.

Uberenv looks for configuration yaml files under scripts/uberenv/spack_config/{platform} or you can use the –spack-config-dir option to specify a directory with compiler and packages yaml files to use with Spack. See the Spack Compiler Configuration and Spack System Packages documentation for details.

Note

The bootstrapping process ignores ~/.spack/compilers.yaml to avoid conflicts and surprises from a user’s specific Spack settings on HPC platforms.

When run, uberenv.py checkouts a specific version of Spack from github as spack in the destination directory. It then uses Spack to build and install the target packages’ dependencies into spack/opt/spack/. Finally, the target package generates a host-config file {hostname}.cmake, which is copied to destination directory. This file specifies the compiler settings and paths to all of the dependencies.

Project Settings

A few notes on using uberenv.py in a new project:

  • For an example of how to craft a project.json file a target project, see: Conduit’s project.json file
  • uberenv.py hot copies packages to the cloned spack install, this allows you to easily version control any Spack package overrides necessary