Steps on Param Pravega

Step-1 : Load SPACK and Initialize environment

 Param Pravega uses SPACK for compilation and linking of modules. For more details on SPACK, refer to the SPACK web page , SPACK user guide , Param Pravega user guideand official SPACK manual.

>> module load spack/0.17
>> . /home-ext/apps/spack/share/spack/setup-env.sh (for bash shell users)
Step-2 : Load any compilers and libraries that you might need for compiling your code

Example:

>> spack load hdf5 /ssgjscn

In order to find a installed library one can use “spack list” or “spack find”. In some cases you will find multiple options for loading in the libraries. For example, if you do

>>spack load hdf5

you will get ssgjscn as one of the options. Please make sure you load all the required compilers, e.g., icc or mpicc or gcc that you might need for the compilation of your code in the same manner.

Step-3 : Providing paths to the libraries that are required for compilation

In some codes you might need to include libraries during compilation such as in CUDA. For such cases the paths to the libraries can either be included in the .bashrc file or can be mentioned in the concerned makefiles used for the compilation. For example, when you want to use cuda@11.1.1 , you will need to specify the paths to /include and /lib. The paths can be figure out in the following manner.

First load the required package (here it is cuda@11.1.1 ) by typing

>> spack load cuda@11.1.1

The paths to the /include and /lib can be found using the command

>> spack find –paths

This lists the paths against the loaded libraries; for example the path to the cuda@11.1.1 library is listed below:

/home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/cuda-11.1.1-uxsh5ipgpwixlbwknjxxbra4j2pv3vad

Paths to the respective include and lib64 libraries are:

  • /home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/cuda-11.1.1-uxsh5ipgpwixlbwknjxxbra4j2pv3vad/include
  • /home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/cuda-11.1.1-uxsh5ipgpwixlbwknjxxbra4j2pv3vad/lib64

There are two possibilities to proceed

  • one is to include the paths in the respective makefiles

as

  • IDIR = /home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/cuda-11.1.1-uxsh5ipgpwixlbwknjxxbra4j2pv3vad/include
  • LDIR = /home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/cuda-11.1.1-uxsh5ipgpwixlbwknjxxbra4j2pv3vad/lib64

and include as

LIBS = -L $(LDIR) -lcufft -lcuda -lcudart -lcublas

Finally include $(LIBS) in your compilation line in the Makefile. Similarly for the IDIR.

  • Second is to add the path in your bashrc file;
  • export PATH=$PATH:/home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/cuda-11.1.1-uxsh5ipgpwixlbwknjxxbra4j2pv3vad/include
  • export LD_LIBRARY_PATH=$LD_LIBARY_PATH: /home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/cuda-11.1.1-uxsh5ipgpwixlbwknjxxbra4j2pv3vad/lib64

and then execute1

>> source ~/.bashrc

Similar procedures can be adopted for the other softwares.

Some software like OpenFoam needs setting the path to the INTEL_LICENSE_FILE that is required for compilation of codes. This needs to be added to the .bashrc file. The path to the INTEL_LICENSE_FILE is : /opt/intel/serverlicenses/COM_L___G5NV-32584FFV.lic

The intel license is configured in 3 nodes (redundant servers) i.e login01, pravega0, mgmt01 and can be accessed with the port 27009.

These additional information can also be added to .bashrc, followed by source command.

Step-4 : Compile your code

Then compile your codes using the usual commands. e.g.,

>> make or cmake or wmake as is suitable for your code

Notes:

  • If you want to use Intel OneAPI compilers:
        o >> spack load intel-oneapi-compilers@2021.4.0
        o >> spack load intel-mpi@2019.10.317 %intel@19.0.5.281
  • If you want to use Intel MPI:
        o >> module load compiler/intel/oneapi/mpi/2021.5.
  • If you want to use hdf5 libraries and h5pcc compiler
        o >> spack load hdf5 (Choose ssgjscn option).
  • If you want to use h5pfc compiler wrapper, do
        o >> spack load hdf5 +fortran %intel
  • For spack environment creation, use the following command
        o >>spack env create -d <path>
  • For compiling GPU code on the PGI compiler:
        o >> spack load nvhpc
        o >>spack load gcc@11.2.0
  • Some programs when loaded with spack may not have their include dirs appended to the CPATH environment variable (specifically zlib)
        o  Find the zlib install dir using
        o >>spack find –paths zlib
        o  Add the include dir to CPATH (bash shell)
        o >>export CPATH=”zlib_install_dir/include:$CPATH”
  • To compile OpenMP with Slurm Option
        o  spack load openmpi+legacylaunchers fabrics=auto schedulers=auto
  • spack automatically sets the CC, CXX, FC, MPICC, MPICXX, MPIF90 etc. environment variables. If need to change, set these environment variables to desired values.
Step-5 : Transfer your data

For small data needs of your program, the data can be accessed from your home directory.

For large data needs, transfer your data to /mnt/scratch/<user_id>

Step-6 : Submit you Job

Create a job script, e.g., job.sh. Param Pravega used SLURM job scheduler. Submit your job using the SLURM submission command.

>> sbatch job.sh

For the use of SLURM job scheduler, refer to Job Submission System.

For sample job submission scripts, refer to Sample Job Submission scripts.

For the queues used in Param Pravega, refer to Param Pravega Queues.

Check the user’s resource usage.
#sreport cluster AccountUtilizationByUser  -t hourper start=01/01/22 end=now
To check the balance from login nodes.
#csreport -a <computational-id>