Quartz (LLNL)
The Quartz Intel CPU cluster is located at LLNL.
Introduction
If you are new to this system, please see the following resources:
Batch system: Slurm
-
/p/lustre1/$(whoami)and/p/lustre2/$(whoami): personal directory on the parallel filesystemNote that the
$HOMEdirectory and the/usr/workspace/$(whoami)space are NFS mounted and not suitable for production quality data generation.
Installation
Use the following commands to download the WarpX source code and switch to the correct branch:
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
We use the following modules and environments on the system ($HOME/quartz_warpx.profile).
# please set your project account
#export proj=<yourProject>
# required dependencies
module load cmake/3.20.2
module load intel/2021.4
module load mvapich2/2.3
# optional: for PSATD support
module load fftw/3.3.8
# optional: for QED lookup table generation support
module load boost/1.73.0
# optional: for openPMD support
# TODO ADIOS2
module load hdf5-parallel/1.10.2
# optional: for PSATD in RZ geometry support
# TODO: blaspp lapackpp
# optional: for Python bindings
module load python/3.8.2
# optional: an alias to request an interactive node for two hours
alias getNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=18 -p pdebug --pty bash"
# fix system defaults: do not escape $ with a \ on tab completion
shopt -s direxpand
# compiler environment hints
export CC=$(which icc)
export CXX=$(which icpc)
export FC=$(which ifort)
# we need a newer libstdc++:
export CFLAGS="-gcc-name=/usr/tce/packages/gcc/gcc-8.3.1/bin/gcc ${CFLAGS}"
export CXXFLAGS="-gxx-name=/usr/tce/packages/gcc/gcc-8.3.1/bin/g++ ${CXXFLAGS}"
We recommend to store the above lines in a file, such as $HOME/quartz_warpx.profile, and load it into your shell after a login:
source $HOME/quartz_warpx.profile
Then, cd into the directory $HOME/src/warpx and use the following commands to compile:
cd $HOME/src/warpx
rm -rf build
cmake -S . -B build
cmake --build build -j 6
The other general compile-time options apply as usual.
That’s it!
A 3D WarpX executable is now in build/bin/ and can be run with a 3D example inputs file.
Most people execute the binary directly or copy it out to a location in /p/lustre1/$(whoami).
Running
Intel Xeon E5-2695 v4 CPUs
The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Quartz at LLNL.
Replace descriptions between chevrons <> by relevant values, for instance <input file> could be plasma_mirror_inputs.
#!/bin/bash -l
# Just increase this number of you need more nodes.
#SBATCH -N 2
#SBATCH -t 24:00:00
#SBATCH -A <allocation ID>
#SBATCH -J WarpX
#SBATCH -q pbatch
#SBATCH --qos=normal
#SBATCH --license=lustre1,lustre2
#SBATCH --export=ALL
#SBATCH -e error.txt
#SBATCH -o output.txt
# one MPI rank per half-socket (see below)
#SBATCH --tasks-per-node=2
# request all logical (virtual) cores per half-socket
#SBATCH --cpus-per-task=18
# each Quartz node has 1 socket of Intel Xeon E5-2695 v4
# each Xeon CPU is divided into 2 bus rings that each have direct L3 access
export WARPX_NMPI_PER_NODE=2
# each MPI rank per half-socket has 9 physical cores
# or 18 logical (virtual) cores
# over-subscribing each physical core with 2x
# hyperthreading led to a slight (3.5%) speedup on Cori's Intel Xeon E5-2698 v3,
# so we do the same here
# the settings below make sure threads are close to the
# controlling MPI rank (process) per half socket and
# distribute equally over close-by physical cores and,
# for N>9, also equally over close-by logical cores
export OMP_PROC_BIND=spread
export OMP_PLACES=threads
export OMP_NUM_THREADS=18
EXE="<path/to/executable>" # e.g. ./warpx
srun --cpu_bind=cores -n $(( ${SLURM_JOB_NUM_NODES} * ${WARPX_NMPI_PER_NODE} )) ${EXE} <input file>
To run a simulation, copy the lines above to a file quartz.sbatch and run
sbatch quartz.sbatch
to submit the job.