WDMApp on Summit at OLCF¶

Setting up Spack¶

Spack is a generic package manager for HPC. We rely on it in the following to install WDMapp and its components. Setting up Spack is a one-time process on a given machine – if you already have a working Spack install, you should be able to use it. However, in practice there are plenty of ways that things can wrong, so we provide tested Spack setups for a selection of machines. Following our instructions makes sure that WDMapp is built with compatible compilers and machine-specific system packages (e.g., MPI, CUDA, etc.).

Installing Spack¶

Follow the instructions from the Spack Documentation.

$ git clone -b v0.15.4 https://github.com/spack/spack.git

Note

v0.15.4 is the latest spack stable version on 2020-10-20; newer versions will likely work but have not been tested. Using the default ‘develop’ branch is not recommended, as it does break sometimes and introduces a lot of package version churn if you try to track it.

Enable shell support for Spack.

# For bash/zsh users
$ export SPACK_ROOT=/path/to/spack
$ . $SPACK_ROOT/share/spack/setup-env.sh

# For tcsh or csh users (note you must set SPACK_ROOT)
$ setenv SPACK_ROOT /path/to/spack
$ source $SPACK_ROOT/share/spack/setup-env.csh

Cloning the WDMapp package repo¶

Just clone the repository from github to the same machine that you just set up Spack on.

$ git clone git@github.com:wdmapp/wdmapp-config.git

Summit-Specific Setup¶

Rhea and Summit share a common home directory. If you use Spack on both machines, this leads to issues because both instances will share their config files, which by default go into `~/.spack/linux. If you only want to use just one or the other machine, you can ignore the following note.

Note

One way to deal with keeping separate spack setups on Rhea and Summit is to make separate ~/.spack/linux-rhea and ~/spack/linux-summit directories and symlink one or the other to ~/.spack/linux

$ # make sure ~/.spack/linux does exit yet -- if it does, move it  out of the way
$ mkdir -p ~/.spack/linux-rhea
$ mkdir -p ~/.spack/linux-summit
$ ln -snf ~/.spack/linux-rhea ~/.spack/linux # if on rhea
$ ln -snf ~/.spack/linux-summit ~/.spack/linux # if on summit

An alternative is to have two separate spack installs, and instead of keeping the config files in ~/.spack, they can be put into $SPACK_ROOT/etc/spack, so with two different roots they can be kept separate. You can then do this in your .bashrc:

if [ `uname -m` == "ppc64le" ]; then
  export SPACK_ROOT=$HOME/spack-summit
else
  export SPACK_ROOT=$HOME/spack-rhea
fi
source $SPACK_ROOT/share/spack/setup-env.sh

Spack commands that edit configuration files such as spack compiler add can be called spack compiler add --scope site to update files living in $SPACK_ROOT/etc/spack.

Employing our provided Spack configuration¶

Warning

The folllowing will overwrite an existing Spack configuration, so be careful if you’ve previously set up Spack. If you have an existing config, consider renaming ~./spack to back it up.

Just copy the provided YAML configuration files to where Spack expects them:

$ mkdir -p ~/.spack/linux
$ cp path/to/wdmapp-config/summit/spack/*.yaml ~/.spack/linux

You can have a choice of a basic or a more comprehensive setup for Spack on Summit from the wdmapp-config repository.

If you use the provided packages.yaml, it only tells Spack about essential existing pre-installed packages on Summit, ie., CUDA, MPI and the corresponding compilers. Spack will therefore build and install all other dependencies from scratch, which takes time but has the advantage that it’ll generate pretty much the same software stack on any machine you use.

On the other hand, packages-extended.yaml (which needs to be renamed to packages.yaml to be used), tells Spack comprehensively about pre-installed software on Summit, so installation of WDMapp will proceed more quickly and use system-provided libraries where possible.

Warning

Make sure that you don’t have xl or spectrum-mpi loaded. By default, Summit will load the xl and spectrum-mpi modules for you, and those interfere when Spack tries to perform gcc based builds. You might want to consider adding this to your .bashrc or similar init file:

module unload xl spectrum-mpi

Note

On Summit, the cuda module sets environment variables that set a path which nvcc does not otherwise add. Because of this, it is requried to module load cuda/10.1.243 before building GENE, and probably other software that uses CUDA..

Note

Consider also configuring spack to use gpfs scratch space (i.e. $MEMBERWORK) as an alternative when building packages, in addition to the default tmpfs and home filesystem (which can have problems with high workload tasks):

$ mkdir -p /gpfs/alpine/scratch/$USER/<project-id>/spack-stage

and add the following to ~/.spack/config.yaml ($SPACK_ROOT/etc/spack/config.yaml). Spack tries each entry in order for precedence:

config:
  build_stage:
    - $tempdir/$user/spack-stage
    - /gpfs/alpine/scratch/$USER/<project-id>/spack-stage
    - ~/.spack/stage

Adding the WDMapp package repo to Spack¶

This will let Spack search the WDMapp repository for packages that aren’t found in its builtin package repository.

$ spack repo add path/to/wdmapp-config/spack/wdmapp
==> Added repo with namespace 'wdmapp'.

Note

To update the wdmapp package repository to the latest, just run git pull in the directory where you cloned wdmapp-config/.

Building WDMapp¶

You should be able to just follow the generic instructions from Building WDMAPP.

Running the Cyclone Test Case¶

Enable shell support for Spack:

# For bash/zsh users
$ export SPACK_ROOT=/path/to/spack
$ . $SPACK_ROOT/share/spack/setup-env.sh

# For tcsh or csh users (note you must set SPACK_ROOT)
$ setenv SPACK_ROOT /path/to/spack
$ source $SPACK_ROOT/share/spack/setup-env.csh

Load the wdmapp modules:

$ spack load wdmapp arch=linux-rhel7-power9le
$ spack load effis +compose arch=linux-rhel7-power9le

Clone the testcases repo:

$ git clone git@github.com:wdmapp/testcases.git
$ cd testcases/run_1/summit

See the Workflow Composition in EFFIS page for help editing the workflow composition file. As quick pointers, make sure to edit the path to the run directory (/path/to/testDir below) called rundir, the binaries labeled executable_path, and the project, charge, in run_1.yaml.

Since we loaded the wdmapp module via Spack the binaries are in your PATH and their location can be retrieved with the which xgc-es gene cpl command.

Run the effis pre-processor:

$ effis-compose.py run_1.yaml

Submit the job:

$ effis-submit /path/to/testDir