WDMApp on Rhea at OLCF¶

Setting up Spack¶

Spack is a generic package manager for HPC. We rely on it in the following to install WDMapp and its components. Setting up Spack is a one-time process on a given machine – if you already have a working Spack install, you should be able to use it. However, in practice there are plenty of ways that things can wrong, so we provide tested Spack setups for a selection of machines. Following our instructions makes sure that WDMapp is built with compatible compilers and machine-specific system packages (e.g., MPI, CUDA, etc.).

Installing Spack¶

Follow the instructions from the Spack Documentation.

$ git clone -b v0.15.4 https://github.com/spack/spack.git

Note

v0.15.4 is the latest spack stable version on 2020-10-20; newer versions will likely work but have not been tested. Using the default ‘develop’ branch is not recommended, as it does break sometimes and introduces a lot of package version churn if you try to track it.

Enable shell support for Spack.

# For bash/zsh users
$ export SPACK_ROOT=/path/to/spack
$ . $SPACK_ROOT/share/spack/setup-env.sh

# For tcsh or csh users (note you must set SPACK_ROOT)
$ setenv SPACK_ROOT /path/to/spack
$ source $SPACK_ROOT/share/spack/setup-env.csh

Cloning the WDMapp package repo¶

Just clone the repository from github to the same machine that you just set up Spack on.

$ git clone git@github.com:wdmapp/wdmapp-config.git

Rhea-Specific Setup¶

Rhea and Summit share a common home directory. If you use Spack on both machines, this leads to issues because both instances will share their config files, which by default go into `~/.spack/linux. If you only want to use just one or the other machine, you can ignore the following note.

Note

One way to deal with keeping separate spack setups on Rhea and Summit is to make separate ~/.spack/linux-rhea and ~/spack/linux-summit directories and symlink one or the other to ~/.spack/linux

$ # make sure ~/.spack/linux does exit yet -- if it does, move it  out of the way
$ mkdir -p ~/.spack/linux-rhea
$ mkdir -p ~/.spack/linux-summit
$ ln -snf ~/.spack/linux-rhea ~/.spack/linux # if on rhea
$ ln -snf ~/.spack/linux-summit ~/.spack/linux # if on summit

An alternative is to have two separate spack installs, and instead of keeping the config files in ~/.spack, they can be put into $SPACK_ROOT/etc/spack, so with two different roots they can be kept separate. You can then do this in your .bashrc:

if [ `uname -m` == "ppc64le" ]; then
  export SPACK_ROOT=$HOME/spack-summit
else
  export SPACK_ROOT=$HOME/spack-rhea
fi
source $SPACK_ROOT/share/spack/setup-env.sh

Spack commands that edit configuration files such as spack compiler add can be called spack compiler add --scope site to update files living in $SPACK_ROOT/etc/spack.

Employing our provided Spack configuration¶

Warning

The folllowing will overwrite an existing Spack configuration, so be careful if you’ve previously set up Spack. If you have an existing config, consider renaming ~./spack to back it up.

Just copy the provided YAML configuration files to where Spack expects them:

$ mkdir -p ~/.spack/linux
$ cp path/to/wdmapp-config/rhea/spack/*.yaml ~/.spack/linux

On Rhea an olcf repo is also needed to make it possible to use system-installed packages from our Spack. This repo is provided by the wdmapp-config you cloned earlier:

$ spack repo add path/to/wdmapp-config/rhea/spack/olcf
==> Added repo with namespace 'olcf'

Note

Consider also configuring spack to use gpfs scratch space (i.e. $MEMBERWORK) as an alternative when building packages, in addition to the default tmpfs and home filesystem (which can have problems with high workload tasks):

$ mkdir -p /gpfs/alpine/scratch/$USER/<project-id>/spack-stage

and add the following to ~/.spack/config.yaml ($SPACK_ROOT/etc/spack/config.yaml). Spack tries each entry in order for precedence:

config:
  build_stage:
    - $tempdir/$user/spack-stage
    - /gpfs/alpine/scratch/$USER/<project-id>/spack-stage
    - ~/.spack/stage

Adding the WDMapp package repo to Spack¶

This will let Spack search the WDMapp repository for packages that aren’t found in its builtin package repository.

$ spack repo add path/to/wdmapp-config/spack/wdmapp
==> Added repo with namespace 'wdmapp'.

Note

To update the wdmapp package repository to the latest, just run git pull in the directory where you cloned wdmapp-config/.

Note

The E4S project has created a build cache for Rhea. This provides many packages as precompiled binaries, so will reduce the installation time. To use it:

$ wget https://oaciss.uoregon.edu/e4s/e4s.pub
$ spack gpg trust e4s.pub
$ spack mirror add E4S https://cache.e4s.io

Building WDMapp¶

You should be able to just follow the generic instructions from Building WDMAPP.

Using E4S WDMapp docker container¶

Alternatively, the E4S project has created a docker image that mirrors the Rhea environment, which can be used for local development and debugging. To run this image, you need to have docker installed and then do the following:

$ docker pull ecpe4s/ubi7.7_x86_64_base_wdm:1.0
$ docker run —rm -it ecpe4s/ubi7.7_x86_64_base_wdm:1.0

In order for the image to get the access controlled components, you need to provide it with your private SSH key that provides access to the respective private github repos. In the image, do the following in the docker image:

# cat > .ssh/id_rsa # Then copy&paste your private key
# chmod 600 .ssh/id_rsa

This provides an development environment with everything but the private codes preinstalled. All that’s needed to complete building and installing them is:

# spack install wdmapp target=x86_64

Running the Cyclone Test Case¶

Enable shell support for Spack:

# For bash/zsh users
$ export SPACK_ROOT=/path/to/spack
$ . $SPACK_ROOT/share/spack/setup-env.sh

# For tcsh or csh users (note you must set SPACK_ROOT)
$ setenv SPACK_ROOT /path/to/spack
$ source $SPACK_ROOT/share/spack/setup-env.csh

Load the wdmapp modules:

$ spack load wdmapp arch=linux-rhel7-sandybridge
$ spack load effis +compose arch=linux-rhel7-sandybridge

Clone the testcases repo:

$ git clone git@github.com:wdmapp/testcases.git
$ cd testcases/run_1/rhea

See the Workflow Composition in EFFIS page for help editing the workflow composition file. As quick pointers, make sure to edit the path to the run directory (/path/to/testDir below) called rundir, the binaries labeled executable_path, and the project, charge, in run_1.yaml (or run_externalCpl.yaml if wdmapp+passthrough was built in Building WDMAPP).

Since we loaded the wdmapp module via Spack the binaries are in your PATH and their location can be retrieved with the which xgc-es gene cpl command.

Run the effis pre-processor:

$ effis-compose.py run_1.yaml

Submit the job:

$ effis-submit /path/to/testDir

Running the Cyclone Test Case - External Coupler¶

The cyclone test case can be executed with the external coupler (wdmapp+passthrough built in Building WDMAPP by following the instructions for Running the Cyclone Test Case using run_externalCpl.yaml instead of run_1.yaml.