Uberenv¶

Uberenv automates using a package manager to build and deploy software. It uses Spack on Unix-based systems (e.g. Linux and macOS) and Vcpkg on Windows systems.

Many projects leverage package managers, like Spack and Vcpkg, to help build the software dependencies needed to develop and deploy their projects on HPC systems. Uberenv is a python script that helps automate the usage of a package manager to build third-party dependencies for development and deployment.

Uberenv was released as part of Conduit (https://github.com/LLNL/conduit/). It is included in-source in several projects. The https://github.com/llnl/uberenv/ repo is used to hold the latest reference version of Uberenv.

Several projects are using Uberenv for Continuous Integration (CI) purposes. The process is documented here.

uberenv.py¶

Uberenv is a single file python script (uberenv.py) that automates fetching Spack or Vcpkg, building and installing third party dependencies, and can optionally install top-level packages as well. To automate the full install process, Uberenv uses a target Spack or Vcpkg package along with extra settings such as compilers and external third party package details for common HPC platforms.

Uberenv is included directly in a project’s source code repo, usually in the folder: scripts/uberenv/. This folder is also used to store extra configuration files unique to the target project. Uberenv uses a project.json file to specify project details, including the target package name and the base branch or commit in the package manager.

Conduit’s source repo serves as an example for Uberenv and Spack configuration files, etc:

https://github.com/LLNL/conduit/tree/master/scripts/uberenv

Uberenv can also be used as a submodule of the user project, where one must provide a configuration file named .uberenv_config.json in a parent directory. This file is similar to project.json in purpose, but should additionally provide the entries spack_configs_path and spack_packages_path. See Project Configuration for more details.

Note

Uberenv requires python 3.3 or above.

Uberenv is developed by LLNL, originally in support of the Ascent, Axom, and Conduit projects. It is now also used by Umpire, CHAI, RAJA and Serac, among others.

Command Line Options¶

Build Configuration¶

Uberenv has a few options that allow you to control how dependencies are built:

Option Description Default

--prefix Destination directory uberenv_libs

--spec Spack spec without preceding package name linux: %gcc osx: %clang

--spack-config-dir Folder with Spack settings files See Spack Configurations

--spack-build-mode Mode used to build third party dependencies dev-build

--spack-debug Turn on spack debug mode for spec/install commands none (False)

-k Ignore SSL Errors False

--install Fully install target, not just dependencies False

--run_tests Invoke tests during build and against install False

--setup-only Only download and setup Spack False

--skip-setup Only install (using pre-setup Spack) False

--project-json File for project specific settings See Project Configuration

--triplet (vcpkg) Target architecture and linkage VCPKG_DEFAULT_TRIPLET environment variable, if present, x86-Windows otherwise

The -k option exists for sites where SSL certificate interception undermines fetching from github and https hosted source tarballs. When enabled, Uberenv clones Spack using:

git -c http.sslVerify=false clone https://github.com/llnl/spack.git

And passes -k to any Spack commands that may fetch via https.

Default invocations:

Linux

python scripts/uberenv/uberenv.py --prefix uberenv_libs --spec %gcc

OSX

python scripts/uberenv/uberenv.py --prefix uberenv_libs --spec %clang

Windows

python scripts/uberenv/uberenv.py --prefix uberenv_libs --triplet x86-windows

See Vcpkg user docs for more information about triplets.

Use the --install option to install the target package (not just its development dependencies):

python scripts/uberenv/uberenv.py --install

If the target Spack package supports Spack’s testing hooks, you can run tests during the build process to validate the build and install, using the --run_tests option:

python scripts/uberenv/uberenv.py --install --run_tests

For details on Spack’s spec syntax, see the Spack Specs & dependencies documentation.

Spack Configurations¶

Uberenv looks for configuration yaml files under scripts/uberenv/spack_configs/{platform} or under {spack_config_paths}/{platform}, where: * {platform} must match the platform determined by uberenv (darwin on OSX). * {spack_configs_path} can be specified in the json config file.

You may instead use the –spack-config-dir option to enforce the use of a specific directory. As long as it provides Uberenv with the yaml files to use with Spack. See the Spack Compiler Configuration and Spack System Packages documentation for details.

Note

The bootstrapping process ignores ~/.spack/compilers.yaml to avoid conflicts and surprises from a user’s specific Spack settings on HPC platforms.

When run, uberenv.py checkouts a specific version of Spack from github as spack in the destination directory. It then uses Spack to build and install the target packages’ dependencies into spack/opt/spack/. Finally, the target package generates a host-config file {hostname}.cmake, which is copied to destination directory. This file specifies the compiler settings and paths to all of the dependencies.

Project Configuration¶

Project level configuration options can also be addressed using a json file and some settings can be overridden on command line. This json file is found in the in the following order:

–project.json=[path/to/project.json] command line option
project.json that lives in the same directory as uberenv.py
.uberenv_config.json found recursively in a parent directory (typically at the root of your project)

Project settings are as follows:

Setting Command line Option Description Default

package_name --package-name Spack package name None

package_version None Spack package version None

package_final_phase --package-final-phase Controls after which phase Spack should stop None

package_source_dir --package-source-dir Controls the source directory Spack should use None

force_commandline_prefix None Force user to specify –prefix on command line false

spack_url None Download url for Spack https://github.com/spack/spack.git

spack_commit None Spack commit to checkout None

spack_activate None Spack packages to activate None

spack_build_mode --spack-build-mode Set mode used to build TPLs with Spack dev-build

spack_configs_path None Directory with Spack configs to be copied spack_configs

spack_packages_path None Directory with Spack packages to be copied packages

spack_concretizer None Spack concretizer to use original, clingo original

spack_setup_clingo None Do not install clingo if set to false true

vcpkg_url None Download url for Vcpkg https://github.com/microsoft/vcpkg

vcpkg_branch None Vcpkg branch to checkout master

vcpkg_commit None Vcpkg commit to checkout None

vcpkg_ports_path --vcpkg-ports-path Folder with vcpkg ports files None

If a spack_commit is present, it supercedes the spack_branch option, and similarly for vcpkg_commit and vcpkg_branch.

When used as a submodule .uberenv_config.json should define both spack_configs_path and spack_packages_path, providing Uberenv with the respective location of spack_configs and packages directories. Note that they cannot sit next to uberenv.py, since by default, the Uberenv repo does not provide them.

Uberenv forcefully copies all directories that exist under spack_packages_path to the cloned Spack in order that they are given. This allows you to easily version control any Spack package overrides necessary.

Note

If you want, you may provide spack_packages_path with a list of directories that will be copied in order on top of each other. For example, spack_packages_path: [“first/path”, “overwriting/path”].

Note

For an example of how to craft a project.json / .uberenv_config.json file a target project, see: Axom’s project.json file.

Optimization¶

Uberenv also features options to optimize the installation

Option Description Default

--mirror Location of a Spack mirror None

--create-mirror Creates a Spack mirror at specified location None

--upstream Location of a Spack upstream None

Note

These options are only currently available for spack.

Spack Concretization¶

Uberenv provides a spack_concretizer setting to select the method by which the “concrete” dependency tree is determined. The original option is the default behavior and is often subject to errors where a valid set of constraints fails to concretize. The clingo option is more robust in this respect but requires the installation of the clingo Python module. This happens automatically when the spack_concretizer option is set to clingo, but requires pip >= 19.3 and Python >= 3.6. If your pip version is out of date, Uberenv will prompt you to upgrade it.