|Step-1 : Load SPACK and Initialize environment
|Step-2 : Load any compilers and libraries that you might need for compiling your code
|>> spack load hdf5 /ssgjscn
In order to find a installed library one can use “spack list” or “spack find”. In some cases you will find multiple options for loading in the libraries. For example, if you do
you will get ssgjscn as one of the options. Please make sure you load all the required compilers, e.g., icc or mpicc or gcc that you might need for the compilation of your code in the same manner.
|Step-3 : Providing paths to the libraries that are required for compilation
|In some codes you might need to include libraries during compilation such as in CUDA. For such cases the paths to the libraries can either be included in the .bashrc file or can be mentioned in the concerned makefiles used for the compilation. For example, when you want to use firstname.lastname@example.org , you will need to specify the paths to /include and /lib. The paths can be figure out in the following manner.
First load the required package (here it is email@example.com ) by typing
The paths to the /include and /lib can be found using the command
This lists the paths against the loaded libraries; for example the path to the firstname.lastname@example.org library is listed below:
Paths to the respective include and lib64 libraries are:
There are two possibilities to proceed
- one is to include the paths in the respective makefiles
- IDIR = /home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/cuda-11.1.1-uxsh5ipgpwixlbwknjxxbra4j2pv3vad/include
- LDIR = /home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/cuda-11.1.1-uxsh5ipgpwixlbwknjxxbra4j2pv3vad/lib64
and include as
|LIBS = -L $(LDIR) -lcufft -lcuda -lcudart -lcublas
Finally include $(LIBS) in your compilation line in the Makefile. Similarly for the IDIR.
- Second is to add the path in your bashrc file;
- export PATH=$PATH:/home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/cuda-11.1.1-uxsh5ipgpwixlbwknjxxbra4j2pv3vad/include
- export LD_LIBRARY_PATH=$LD_LIBARY_PATH: /home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/cuda-11.1.1-uxsh5ipgpwixlbwknjxxbra4j2pv3vad/lib64
and then execute1
Similar procedures can be adopted for the other softwares.
Some software like OpenFoam needs setting the path to the INTEL_LICENSE_FILE that is required for compilation of codes. This needs to be added to the .bashrc file. The path to the INTEL_LICENSE_FILE is : /opt/intel/serverlicenses/COM_L___G5NV-32584FFV.lic
The intel license is configured in 3 nodes (redundant servers) i.e login01, pravega0, mgmt01 and can be accessed with the port 27009.
These additional information can also be added to .bashrc, followed by source command.
|Step-4 : Compile your code
|Then compile your codes using the usual commands. e.g.,
|>> make or cmake or wmake as is suitable for your code
- If you want to use Intel OneAPI compilers:
o >> spack load email@example.com
o >> spack load firstname.lastname@example.org %email@example.com
- If you want to use Intel MPI:
o >> module load compiler/intel/oneapi/mpi/2021.5.
- If you want to use hdf5 libraries and h5pcc compiler
o >> spack load hdf5 (Choose ssgjscn option).
- If you want to use h5pfc compiler wrapper, do
o >> spack load hdf5 +fortran %intel
- For spack environment creation, use the following command
o >>spack env create -d <path>
- For compiling GPU code on the PGI compiler:
o >> spack load nvhpc
o >>spack load firstname.lastname@example.org
- Some programs when loaded with spack may not have their include dirs appended to the CPATH environment variable (specifically zlib)
o Find the zlib install dir using
o >>spack find –paths zlib
o Add the include dir to CPATH (bash shell)
o >>export CPATH=”zlib_install_dir/include:$CPATH”
- To compile OpenMP with Slurm Option
o spack load openmpi+legacylaunchers fabrics=auto schedulers=auto
- spack automatically sets the CC, CXX, FC, MPICC, MPICXX, MPIF90 etc. environment variables. If need to change, set these environment variables to desired values.
|Step-5 : Transfer your data
|For small data needs of your program, the data can be accessed from your home directory.
For large data needs, transfer your data to /mnt/scratch/<user_id>
|Step-6 : Submit you Job
|Create a job script, e.g., job.sh. Param Pravega used SLURM job scheduler. Submit your job using the SLURM submission command.
For the use of SLURM job scheduler, refer to Job Submission System.
For sample job submission scripts, refer to Sample Job Submission scripts.
For the queues used in Param Pravega, refer to Param Pravega Queues.
Check the user’s resource usage.
#sreport cluster AccountUtilizationByUser -t hourper start=01/01/22 end=now
To check the balance from login nodes.
#csreport -a <computational-id>