Vapor

File Systems

The cluster offers two types of filesystems for storage and computation:

1. NFS Home Directories: - Accessible at /ccs/home/<username>

2. Lustre Filesystem: - A high-performance Lustre system with 1 TB total available storage, mounted at /lustre/OLCFLustre

Storage Areas Overview

The Lustre filesystem is organized into the following storage areas for project-specific work:

Logging In to Vapor

To log in to Vapor, use your OLCF username and passcode:

First ssh to hub and from there ssh to the Vapor login node.

$ ssh <username>@hub.ccs.ornl.gov
$ ssh <username>@login1.vapor.olcf.ornl.gov

Environment Modules (Lmod)

Environment modules are provided through Lmod, a Lua-based module system for dynamically altering shell environments. By managing changes to the shell’s environment variables (such as PATH, LD_LIBRARY_PATH, and PKG_CONFIG_PATH), Lmod allows you to alter the software available in your shell environment without the risk of creating package and version combinations that cannot coexist in a single environment.

Compilers

AMD, GCC, Intel, and LLVM compilers are provided through modules. There is also the system version of GCC available in /usr/bin. The below table lists details.

MPI

Both MPICH and OpenMPI modules are available. But MPICH is recommended and loaded by default. Use mpicc, mpicxx, mpifort compiler wrappers for compiling for C, C++, Fortran with MPI. The compiler wrapper will use the compiler from the currently loaded compiler module.

Login vs Compute Nodes

Recall from the System Overview that Frontier contains two node types: Login and Compute. When you connect to the system, you are placed on a login node. Login nodes are used for tasks such as code editing, compiling, etc. They are shared among all users of the system, so it is not appropriate to run tasks that are long/computationally intensive on login nodes. Users should also limit the number of simultaneous tasks on login nodes (e.g., concurrent tar commands, parallel make

Compute nodes are the appropriate place for long-running, computationally-intensive tasks. When you start a batch job, your batch script (or interactive shell for batch-interactive jobs) runs on one of your allocated compute nodes.

Login vs Compute Nodes

Vapor contains two node types: Login and Compute. When you connect to the system, you are placed on a login node. Login nodes are used for tasks such as code editing, compiling, etc. They are shared among all users of the system, so it is not appropriate to run tasks that are long/computationally intensive on login nodes. Users should also limit the number of simultaneous tasks on login nodes (e.g., concurrent tar commands, parallel make

Compute nodes are the appropriate place for long-running, computationally-intensive tasks. When you start a batch job, your batch script (or interactive shell for batch-interactive jobs) runs on one of your allocated compute nodes.

Slurm Workload Manager

Slurm is the workload manager used to interact with the compute nodes on Vapor. In the following subsections, the most commonly used Slurm commands for submitting, running, and monitoring jobs will be covered, but users are encouraged to visit the official documentation and man pages for more information.

#!/bin/bash
#SBATCH -A <project_id>
#SBATCH -J <job_name>
#SBATCH -o %x-%j.out
#SBATCH -t 00:05:00
#SBATCH -p <partition>
#SBATCH -N 2

srun -n4 --ntasks-per-node=2 ./a.out

$ salloc -A <project_id> -J <job_name> -t 00:05:00 -p <partition> -N 2
salloc: Granted job allocation 313
salloc: Waiting for resource configuration
salloc: Nodes vapor[01-02] are ready for job

$ srun -n 4 --ntasks-per-node=2 ./a.out
<output printed to terminal>

$ srun -n 2 --ntasks-per-node=1 ./a.out
<output printed to terminal>

$ srun -A <project_id> -t 00:05:00 -p <partition> -N 2 -n 4 --ntasks-per-node=2 ./a.out
<output printed to terminal>

Building and running OSU Benchmarks container example (with GCC and MPICH)

• Create a file named simplempich.def

  Bootstrap: docker
  From: opensuse/leap:15.6
  %environment
      # Point to MPICH binaries, libraries man pages
      export MPICH_DIR=/opt/mpich
      export PATH="$MPICH_DIR/bin:$PATH"
      export LD_LIBRARY_PATH="$MPICH_DIR/lib:$LD_LIBRARY_PATH"
      export MANPATH=$MPICH_DIR/share/man:$MANPATH
      # Point to rocm locations
      export ROCM_PATH=/opt/rocm
      export LD_LIBRARY_PATH="/opt/rocm/lib:/opt/rocm/lib64:$LD_LIBRARY_PATH"
      export PATH="/opt/rocm/bin:$PATH"
  
  %post
  echo "Installing required packages..."
  export DEBIAN_FRONTEND=noninteractive
  zypper install -y wget tar make sudo git fakeroot gzip gcc gcc-c++ gcc-fortran
  export MPICH_VERSION=3.4.2
  export MPICH_URL="http://www.mpich.org/static/downloads/$MPICH_VERSION/mpich-$MPICH_VERSION.tar.gz"
  export MPICH_DIR=/opt/mpich
  echo "Installing MPICH..."
  mkdir -p /mpich
  mkdir -p /opt
  # Download
  cd /mpich && wget -O mpich-$MPICH_VERSION.tar.gz $MPICH_URL && tar --no-same-owner -xzf mpich-$MPICH_VERSION.tar.gz
  # Compile and install
  cd /mpich/mpich-$MPICH_VERSION && ./configure --disable-fortran --with-device=ch4:ofi --prefix=$MPICH_DIR && make install
  rm -rf /mpich
  # Set env variables so we can compile our application
  
  export PATH=$MPICH_DIR/bin:$PATH
  export LD_LIBRARY_PATH=$MPICH_DIR/lib:$LD_LIBRARY_PATH
  echo "Compiling the MPI application..."
  cd /
  curl -o osubenchmarks-7.2.tar.gz https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-7.2.tar.gz && tar -xzf osubenchmarks-7.2.tar.gz --no-same-owner
  cd osu-micro-benchmarks-7.2 && ./configure CC=mpicc CXX=mpicc  && make  && rm ../osubenchmarks-7.2.tar.gz

• Build the container with

  apptainer build simplempich.sif simplempich.def

Building and running OSU Benchmarks container example (with Intel and Intel MPI)

If you want to build application in the container with the Intel Classic compiler and Intel MPI, you will need to install the appropriate version of the Intel OneAPI release, and set up several environment variables.

• First create the file intelenvs with the required environment variables. This file will be copied into the container image and will be sourced every time the container is started to set up the environment variables.

  export INTEL_PATH=/opt/intel/oneapi/compiler/2023.2.0
  export INTEL_VERSION=2023.2.0
  export INTEL_COMPILER_TYPE=CLASSIC
  export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.10.0/lib/release:/opt/intel/oneapi/compiler/2023.2.0/linux/lib:/opt/intel/oneapi/compiler/2023.2.0/linux/lib/x64:/opt/intel/oneapi/compiler/2023.2.0/linux/lib/oclfpga/host/linux64/lib:/opt/intel/oneapi/compiler/2023.2.0/linux/compiler/lib/intel64_lin:$LD_LIBRARY_PATH
  export CMAKE_PREFIX_PATH=/opt/intel/oneapi/compiler/2023.2.0/linux/IntelDPCPP:$CMAKE_PREFIX_PATH
  export NLSPATH=/opt/intel/oneapi/compiler/2023.2.0/linux/compiler/lib/intel64_lin/locale/%l_%t/%N:$NLSPATH
  export OCL_ICD_FILENAMES=libintelocl_emu.so:libalteracl.so:/opt/intel/oneapi/compiler/2023.2.0/linux/lib/x64/libintelocl.so
  export ACL_BOARD_VENDOR_PATH=/opt/intel/OpenCLFPGA/oneAPI/Boards
  export FPGA_VARS_DIR=/opt/intel/oneapi/compiler/2023.2.0/linux/lib/oclfpga
  export CMPLR_ROOT=/opt/intel/oneapi/compiler/2023.2.0
  export INTELFPGAOCLSDKROOT=/opt/intel/oneapi/compiler/2023.2.0/linux/lib/oclfpga
  export LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.10.0/lib/release:/opt/intel/oneapi/mpi/2021.10.0/lib/:/opt/intel/oneapi/mpi/2021.10.0/lib/:/opt/intel/oneapi/compiler/2023.2.0/linux/compiler/lib/intel64_lin:/opt/intel/oneapi/compiler/2023.2.0/linux/lib:$LIBRARY_PATH
  export DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2023.2.0/sys_check/sys_check.sh:$DIAGUTIL_PATH
  export MANPATH=/opt/intel/oneapi/compiler/2023.2.0/documentation/en/man/common:$MANPATH
  export PATH=/opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64:/opt/intel/oneapi/compiler/2023.2.0/linux/lib/oclfpga/bin:/opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64:/opt/intel/oneapi/compiler/2023.2.0/linux/bin:$PATH
  export PKG_CONFIG_PATH=/opt/intel/oneapi/compiler/2023.2.0/lib/pkgconfig:$PKG_CONFIG_PATH
  export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.10.0/lib/:/opt/intel/oneapi/mkl/2023.2.0/lib/intel64:$LD_LIBRARY_PATH
  export CPATH=/opt/intel/oneapi/compiler/2023.2.0/linux/include:/opt/intel/oneapi/mkl/2023.2.0/include:$CPATH
  export NLSPATH=/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/locale/%l_%t/%N:$NLSPATH
  export LIBRARY_PATH=/opt/intel/oneapi/mkl/2023.2.0/lib/intel64:$LIBRARY_PATH
  export MKLROOT=/opt/intel/oneapi/mkl/2023.2.0
  export PATH=/opt/intel/oneapi/mpi/2021.10.0/bin:/opt/intel/oneapi/mkl/2023.2.0/bin/intel64:$PATH
  export PKG_CONFIG_PATH=/opt/intel/oneapi/mkl/2023.2.0/lib/pkgconfig:$PKG_CONFIG_PATH
  export INCLUDE_PATH=/opt/intel/oneapi/mpi/2021.10.0/include:$INCLUDE_PATH
  export I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.10.0

• Create the file simpleintelmpi.def

  Bootstrap: docker
  From: opensuse/leap:15.6
  
  %files
  ./intelenvs /intelenvs
  
  %environment
      # Point to MPICH binaries, libraries man pages
      export MPICH_DIR=/opt/mpich
      export PATH="$MPICH_DIR/bin:$PATH"
      export LD_LIBRARY_PATH="$MPICH_DIR/lib:$LD_LIBRARY_PATH"
      export MANPATH=$MPICH_DIR/share/man:$MANPATH
      # Point to rocm locations
      export ROCM_PATH=/opt/rocm
      export LD_LIBRARY_PATH="/opt/rocm/lib:/opt/rocm/lib64:$LD_LIBRARY_PATH"
      export PATH="/opt/rocm/bin:$PATH"
      source /intelenvs
  
  %post
  set -xe
  echo "Installing required packages..."
  export DEBIAN_FRONTEND=noninteractive
  zypper install -y wget tar make sudo git fakeroot gzip gcc gcc-c++ gcc-fortran which vim
  
  
  ## adding intel and internal cray pkg repos
  tee > /etc/zypp/repos.d/oneAPI.repo << EOF
  [oneAPI]
  name=Intel® oneAPI repository
  baseurl=https://yum.repos.intel.com/oneapi
  enabled=1
  gpgcheck=1
  repo_gpgcheck=1
  gpgkey=https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
  EOF
  
  
  zypper --releasever=15.6 --non-interactive --gpg-auto-import-keys  refresh
  ## installing intel 2023.2 since that is the version that has intel-classic 2021.10 (and 2023.2 is the last release that provides intel-classic)
  zypper --non-interactive --gpg-auto-import-keys install -y intel-dpcpp-cpp-compiler-2023.2.0  intel-oneapi-compiler-fortran-2023.2.0 intel-oneapi-mpi-devel-2021.10.0
  
  source /intelenvs
  which mpicc
  echo "Compiling the MPI application..."
  cd /
  curl -o osubenchmarks-7.2.tar.gz https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-7.2.tar.gz && tar -xzf osubenchmarks-7.2.tar.gz --no-same-owner
  cd osu-micro-benchmarks-7.2 && ./configure CC=mpiicc CXX=mpiicpc  && make  && rm ../osubenchmarks-7.2.tar.gz

• Build the container image with

  apptainer build simpleintelmpi.sif simpleintelmpi.def

• To run the container, write a job script that will bind in the host's MPI libraries into the container. For example, create the below file submitbind.sl and submit the job with sbatch submitbind.sl.

  #!/bin/bash
  
  #SBATCH -A stf007uanofn
  #SBATCH -J test
  #SBATCH -N 2
  #SBATCH -o logs/subil_%j.out
  #SBATCH -t 01:00:00
  ###SBATCH --ntasks-per-node=16
  
  module reset
  module load oneapi
  
  
  export APPTAINERENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib64/libibverbs::\$LD_LIBRARY_PATH"
  export APPTAINER_CONTAINLIBS="/usr/lib64/libjansson.so.4,/usr/lib64/libjson-c.so.5,/usr/lib64/libnl-3.so.200,/usr/lib64/libibverbs.so.1,/usr/lib64/libnuma.so.1,/usr/lib64/libnl-cli-3.so.200,/usr/lib64/libnl-genl-3.so.200,/usr/lib64/libnl-nf-3.so.200,/usr/lib64/libnl-route-3.so.200,/usr/lib64/libnl-3.so.200,/usr/lib64/libnl-idiag-3.so.200,/usr/lib64/libnl-xfrm-3.so.200,/usr/lib64/libnl-genl-3.so.200"
  export APPTAINER_BIND=/sw/vapor,/var/spool/slurmd,${PWD},/etc/libibverbs.d,/usr/lib64/libibverbs,/usr/lib64/libnl,${HOME}
  
  set -x
  
  srun --ntasks-per-node=16 apptainer exec --writable-tmpfs simplempich.sif /osu-micro-benchmarks-7.2/c//mpi/collective/blocking/osu_alltoall -m 4096
  srun --ntasks-per-node=16 apptainer exec --writable-tmpfs simpleintelmpi.sif /osu-micro-benchmarks-7.2/c//mpi/collective/blocking/osu_alltoall -m 4096