LAMMPS

We provide several versions of LAMMPS through our environment modules system:

module avail lammps

When running a KOKKOS + CUDA version of LAMMPS you need to start exactly one MPI process per GPU you are using. So if you are using 1 GPU you might launch it like this:

lmp_kokkos_cuda_openmpi -sf kk -pk kokkos -k on g 1 < $inputfile

LAMMPS supports doing OMP in addition to using the GPUs but in trial runs it hasn't performed well. If you want to try it out add t Nt where Nt is the number of threads (this example uses 6):

lmp_kokkos_cuda_openmpi -sf kk -pk kokkos -k on t 6 g 1 < $inputfile

Building LAMMPS

Sometimes you are developing plugins for LAMMPS or needs something non-standard. Below is an example walk-through of building LAMMPS with many of the common packages on our systems. Hopefully this provides an idea of how you would modify it for your needs.

#!/bin/bash

N=24 # number of tasks to build in parallel

# navigate to the folder that contains the `src` and `lib` subdirectories
# then:

module purge

# These are needed for most modules
module load gcc/5.5
module load mpi/openmpi-3.0_gcc-5.5

# If you want voronoi support:
module load voro++/0.4.6

cd src 

# Build standard packages except for a few:
make yes-standard
make no-kim
make no-mscg
make no-python
make no-gpu

# These are commonly needed and don't require any extra configuration:
make yes-USER-MISC
make yes-USER-INTEL

# Libraries:
make -j $N lib-meam args="-m mpi -e gfortran"
make -j $N lib-poems args="-m mpi"
make -j $N lib-reax args="-m mpi -e gfortran"

# The main thing; takes a long time:
make -j $N kokkos_omp

# If successful you should have an executable `lmp_kokkos_omp`

If you get the following error you need to use Python version 2.7 (I have not tested Python 3 but may work):

AttributeError: 'module' object has no attribute 'check_output'

You need to alter lib/reax/Makefile.lammps to add -lgfortran to meam_SYSLIB if you get this error:

../../lib/meam/libmeam.a(meam_setup_done.o): In function `compute_pair_meam_':
meam_setup_done.F:(.text+0x4f5e): undefined reference to `_gfortran_runtime_error_at'
meam_setup_done.F:(.text+0x4f6c): undefined reference to `_gfortran_runtime_error'
meam_setup_done.F:(.text+0x4f78): undefined reference to `_gfortran_os_error'
meam_setup_done.F:(.text+0x4f94): undefined reference to `_gfortran_runtime_error_at'
meam_setup_done.F:(.text+0x4fb0): undefined reference to `_gfortran_runtime_error_at'
meam_setup_done.F:(.text+0x4fcc): undefined reference to `_gfortran_runtime_error_at'
meam_setup_done.F:(.text+0x4fe8): undefined reference to `_gfortran_runtime_error_at'
meam_setup_done.F:(.text+0x5004): undefined reference to `_gfortran_runtime_error_at'

This should have been done automatically by:

make lib-meam args="-m mpi -e gfortran"

Building GPU enabled versions

To work with KOKKOS + CUDA on multiple GPUS you need to have an OpenMPI built with CUDA support. To check:

ompi_info | grep cuda

To add GPU support do:

module load cuda/8.0
export KOKKOS_DEVICES=Cuda,OpenMP
export KOKKOS_ARCH=Kepler37
make -j $N kokkos_cuda_openmpi

Note that in some versions theKOKKOS_DEVICES and KOKKOS_ARCH will be over-ridden by target-specific makefiles such as src/MAKE/OPTIONS/Makefile.kokkos_cuda_openmpi so you may need to edit those instead of setting the environment variables.

The KOKKOS_ARCH for our K80s this is Kepler37; for the P100s this is Pascal60.