WDMapp on Longhorn at TACC¶

Setting up Spack¶

Spack is a generic package manager for HPC. We rely on it in the following to install WDMapp and its components. Setting up Spack is a one-time process on a given machine – if you already have a working Spack install, you should be able to use it. However, in practice there are plenty of ways that things can wrong, so we provide tested Spack setups for a selection of machines. Following our instructions makes sure that WDMapp is built with compatible compilers and machine-specific system packages (e.g., MPI, CUDA, etc.).

Installing Spack¶

Follow the instructions from the Spack Documentation.

$ git clone -b v0.15.4 https://github.com/spack/spack.git

Note

v0.15.4 is the latest spack stable version on 2020-10-20; newer versions will likely work but have not been tested. Using the default ‘develop’ branch is not recommended, as it does break sometimes and introduces a lot of package version churn if you try to track it.

Enable shell support for Spack.

# For bash/zsh users
$ export SPACK_ROOT=/path/to/spack
$ . $SPACK_ROOT/share/spack/setup-env.sh

# For tcsh or csh users (note you must set SPACK_ROOT)
$ setenv SPACK_ROOT /path/to/spack
$ source $SPACK_ROOT/share/spack/setup-env.csh

Cloning the WDMapp package repo¶

Just clone the repository from github to the same machine that you just set up Spack on.

$ git clone git@github.com:wdmapp/wdmapp-config.git

Longhorn-Specific Setup¶

Employing our provided Spack configuration¶

Warning

The folllowing will overwrite an existing Spack configuration, so be careful if you’ve previously set up Spack. If you have an existing config, consider renaming ~./spack to back it up.

Just copy the provided YAML configuration files to where Spack expects them:

$ mkdir -p ~/.spack/linux
$ cp path/to/wdmapp-config/longhorn/spack/*.yaml ~/.spack/linux

Note

Make sure that you don’t have a spectrum-mpi loaded. By default, Longhorn will load the xl and spectrum-mpi modules for you, and those interfere when Spack tries to perform gcc based builds. You might want to consider adding this to your .bashrc or similar init file:

module unload xl spectrum-mpi

Creating your own setup for Longhorn¶

Alternatively, you can create your own config:

module load gcc/7.3.0
spack compiler find

This should find a number of compilers, including gcc 7.3.0. You may want to repeat this step for gcc 9.1.0 – however, there is currently no preinstalled MPI for this compiler.

The compilers on longhorn require LD_LIBRARY_PATH hackery to function, and spack sanitizes LD_LIBRARY_PATH. The workaround is described here. In this case, edit ~/.spack/linux/compilers.yaml using spack config edit compilers and modify the modules section for gcc 7 like this:

- compiler:
   spec: gcc@7.3.0
   paths:
     cc: /opt/apps/gcc/7.3.0/bin/gcc
     cxx: /opt/apps/gcc/7.3.0/bin/g++
     f77: /opt/apps/gcc/7.3.0/bin/gfortran
     fc: /opt/apps/gcc/7.3.0/bin/gfortran
   flags: {}
   operating_system: rhel7
   target: ppc64le
   modules: [gcc/7.3.0] # <-- ADD THIS
   environment: {}
   extra_rpaths: []

 similar is required for gcc/9.1.0, and possibly for xl.

Add a minimal packages.yaml in ~/.spack/linux/packages.yaml that registers the preinstalled openmpi and cuda modules:

packages:
  openmpi:
    variants: +cuda fabrics=verbs
    buildable: false
    version: []
    target: []
    compiler: []
    providers: {}
    paths:
      openmpi@3.1.2%gcc@7.3.0: /opt/apps/gcc7_3/openmpi/3.1.2
    modules: {}

  cuda:
    modules:
      cuda@10.1: cuda/10.1
    buildable: false
    version: []
    target: []
    compiler: []
    providers: {}
    paths: {}

all:
    providers:
      mpi: [openmpi]
      blas: [netlib-lapack]
      lapack: [netlib-lapack]

The last section above sets defaults for all packages you’ll installing with Spack – you might want to adjust those, or move them to an environment.

Fixing `config.guess` on longhorn¶

Spack has logic that will replace an outdated config.guess in a given package with a newer version. This comes in handy, because apparently the latest autoconf version is from 2012 and the config.guess that comes with it doesn’t know about ppc64le. However, on longhorn, Spack will not find a newer config.guess in /usr/share/automake*/ , hence things still don’t work. To work around this, I hacked Spack like this:

diff --git a/lib/spack/spack/build_systems/autotools.py b/lib/spack/spack/build_systems/autotools.py
index c21b8dad7..f8e8f64fe 100644
--- a/lib/spack/spack/build_systems/autotools.py
+++ b/lib/spack/spack/build_systems/autotools.py
@@ -133,11 +133,12 @@ def _do_patch_config_guess(self):
             if os.path.exists(path):
                 config_guess = path
         # Look for the system's config.guess
-        if config_guess is None and os.path.exists('/usr/share'):
-            automake_dir = [s for s in os.listdir('/usr/share') if
+        path_am = '/opt/apps/autotools/1.2/share'
+        if config_guess is None and os.path.exists(path_am):
+            automake_dir = [s for s in os.listdir(path_am) if
                             "automake" in s]
             if automake_dir:
-                automake_path = os.path.join('/usr/share', automake_dir[0])
+                automake_path = os.path.join(path_am, automake_dir[0])
                 path = os.path.join(automake_path, 'config.guess')
                 if os.path.exists(path):
                     config_guess = path

Building WDMapp¶

You should be able to just follow the generic instructions from Building WDMAPP.