NVHPC: NVIDIA HPC SDK¶
The NVIDIA HPC SDK version 23.11 is available on Lawrencium. You can load the nvhpc module as:
module load nvhpc/23.11
The nvhpc module consists of the following compilers:
- C:
nvc, - C++:
nvc++ - Fortran:
nvfortran
The CUDA C and CUDA C++ compiler driver nvcc is also present in the module.
CUDA Versions¶
nvhpc/23.11 includes two CUDA toolkit versions: CUDA 11.8 and CUDA 12.3. You can choose a particular version by using the compiler flag -gpu; for example, use -gpu=cuda11.8 to choose CUDA 11.8 when compiling a program using nvhpc compilers.
Target Architecture¶
The -tp=<target> flag can be used to specify a target processor when compiling using nvhpc compilers.
-fast
-fast compiler option is useful to choose an optimal set of vectorization options, but can lead to an auto-selection of the -tp option. This might mean that your compiled code may not work on all Lawrencium partitions as Lawrencium includes a wide range of hardware across several partitions.
MPI¶
nvhpc module includes a version of openmpi. MPI wrapper programs such as mpicc, mpicxx and mpifort are available once nvhpc is loaded.
Running MPI programs
To run MPI jobs compiled with the nvhpc module on Lawrencium, use mpirun instead of srun.
Special considerations
For some GPU nodes with AMD CPU hosts such as A40 nodes, we have found that the following environment variable needs to be set when using nvhpc's mpirun command to launch MPI jobs:
export PMIX_MCA_psec=native
In addition, --bind-to core which is the default for mpirun might not work; in which case, you can try --bind-to none or --bind-to socket. For example:
mpirun -np 2 --bind-to socket ./program