WDMApp on AiMOS at RPI

AiMOS is a 268 node IBM AC922 system with 2x IBM P9 and 6x NVIDIA V100 GPUs with 32GiB of memory each. More info is available on the AiMOS wiki:

https://secure.cci.rpi.edu/wiki/index.php?title=DCS_Supercomputer

Github access requires setting up the http(s) proxy

https://secure.cci.rpi.edu/wiki/index.php?title=Proxy

and you’ll need to create an ssh key-pair for running MPI jobs

$ ssh-keygen # accept defaults

Warning

This will overwrite an existing id_rsa key pair.

Manual Install For Coupler Developers

XGC-Devel

Clone the repo

$ git clone https://github.com/wdmapp/XGC-Devel.git
$ cd XGC-Devel
$ git checkout rpi

Create a environment file with the following contents named envAimosGcc74OpenMPI.sh.

module use /gpfs/u/software/dcs-spack-install/v0133gcc/lmod/linux-rhel7-ppc64le/gcc/7.4.0-1/
module load gcc/7.4.0/1 openmpi/3.1.4-mm5hjuq cmake/3.15.4-mnqjvz6
module load \
  adios/1.13.1-ev2p4am \
  adios2/2.5.0-mklg6ph \
  petsc/3.7.7-int32-hdf5+ftn-real-c-7ewou4w \
  fftw/3.3.8-b2oxdb5 \
  pkg-config/system-cyeqmxc

source the environment file

$ source envAimosGcc74OpenMPI.sh

Create a build directory build-xgcDevel-aimosGcc74OpenMPI, cd into it, and run CMake:

$ mkdir build-xgcDevel-aimosGcc74OpenMPI
$ cd !$
$ # specify the path to the XGC-Devel repo
$ cmake /path/to/xgc-devel/repo \
  -DCMAKE_CXX_COMPILER=g++ \
  -DCMAKE_C_COMPILER=gcc \
  -DCMAKE_Fortran_COMPILER=gfortran \
  -DXGC_USE_ADIOS1=ON \
  -DXGC_USE_ADIOS2=OFF \
  -DXGC_USE_CABANA=OFF \
  -DUSE_SYSTEM_PSPLINE=OFF \
  -DXGC_GENE_COUPLING=ON \
  -DBUILD_TESTING=OFF \
  -DCMAKE_INSTALL_PREFIX=$PWD/install

Run make to compile and link XGC:

$ make -j8

If all goes well the xgc binary will be created; bin/xgc-es-cpp.

GENE

Clone the repo

$ git clone https://github.com/wdmapp/gene.git
$ git checkout rpi

Create a environment file with the following contents named envAimosGcc74OpenMPI.sh.

module use /gpfs/u/software/dcs-spack-install/v0133gcc/lmod/linux-rhel7-ppc64le/gcc/7.4.0-1/
module load gcc/7.4.0/1
module load openmpi/3.1.4-mm5hjuq
module load \
  cmake/3.15.4-mnqjvz6 \
  adios/1.13.1-zrrxpbi \
  adios2/2.5.0-rqsvxj4 \
  fftw/3.3.8-b2oxdb5 \
  netlib-scalapack/2.0.2-7bndnga \
  openblas/0.3.7-x7m3b6w \
  zlib/1.2.11-lpgvqh7 \
  hdf5/1.10.3-ftn-tgragps

export OMPI_CXX=g++
export OMPI_CC=gcc
export OMPI_FC=gfortran

source the environment file

$ source envAimosGcc74OpenMPI.sh

Create a build directory build-gene-aimosGcc74OpenMPI, cd into it, and run CMake:

$ mkdir build-gene-aimosGcc74OpenMPI
$ cd !$
$ # specify the path to the gene repo
$ cmake /path/to/gene/repo \
  -DCMAKE_Fortran_COMPILER=gfortran \
  -DCMAKE_CXX_COMPILER=g++ \
  -DCMAKE_C_COMPILER=gcc \
  -DGENE_USE_FUTILS=on \
  -DGENE_USE_ADIOS2=on \
  -DGENE_DIAG_PLANES=on \
  -DGENE_PERF=none \
  -DGENE_WDMAPP=on

Run make to compile and link GENE:

$ make -j8

If all goes well the gene binary will be created; src/gene.

Run

Clone the testcases repo (https://github.com/wdmapp/testcases) then follow the instructions in run_1/README.md:

https://github.com/wdmapp/testcases/blob/master/run_1/README.md