|
Introduction Fermi cluster is configured to allow parallel batch jobs built to use GPGPUs. Fermi cluster consists of five GPU nodes fermi1, fermi2, fermi3, fermi4 and fermi5. Node fermi1 to fermi4 have Nvidia Tesla C2070 GPU card, with 448 GPUs on each card and fermi5 has 3 Nvidia Tesla M2090 GPU card, with 512 GPUs on each card. To manage batch jobs the cluster uses Torque software, from opensource. Torque version 3.0.2 is installed on the fermi cluster. Users intending to run GPU based parallel codes are expected to submit jobs to this cluster. GPU specification of the job needs to be mentioned as a node attribute in the job script, as illustrated in the sample job script below. As of now, each job can use a maximum of one GPU card, i.e. a total of 448 GPGPUs, per node(fermi1 – fermi4) or maximum of 3 GPU cards (each card with 512 GPUs) on fermi5. Users expecting to use multiple GPU cards have to specify requisite number of nodes or GPUs accordingly in job scripts. How to use Torque Users can log on to fermi1, use qsub command to submit their jobs. Environmental setup For c shell users : Add the following lines in your .cshrc file
run the command source .cshrc For bash shell users: Add the following lines in your .bashrc file
run the command source .bashrc Queue Configuration There are two queues configured on the fermi cluster : routeq and batch routeq:This is the default queue in which all jobs are placed when submitted. The purpose of this queue is to route the jobs to the queue based on the parameters specified in the job script. batch queue:Once a job is placed in this queue, it is sent for execution depending on the available free resources on the cluster. For this queue the default walltime is 24 hrs. Note: Users cannot directly submit jobs to a particular queue. All the jobs are routed through routeq How to submit jobs To submit jobs on the cluster, users must use qsub command. The job executable is expected to be built using the installed CUDA libraries on this cluster. The general procedure for using the cluster is to first build your CUDA application, create job submission script and submit it using qsub. A sample job submission script for fermi1 – fermi4 as below: #!/bin/bash
#PBS -N testn
#PBS-l nodes=1:ppn=x:gpus=1
cd /path_of_executable
./hello > /localscratch/login-id/hello.out
mv /localscratch/login-id/hello.out /path/hello.out A sample job submission script for fermi5 as below: #!/bin/bash
#PBS-N testn
#PBS-l nodes=1:ppn=x:gpus=y:M2090
cd /path_of_executable
./hello > /localscratch/login-id/hello.out
mv /localscratch/login-id/hello.out /path/hello.out
Here, hello : The executable file
login-id : Your login-id
hello.out: Name of your output file
path : Specify the output file path
The user can change the values of ppn and gpus nodes : number of nodes on which to execute the job. For a simple CUDA application using one GPU card nodes=1; in case an application intends to use multiple GPU cards, nodes valuecan be in between 2 – 4 for this cluster(fermi1 – fermi4).
gpus : number of gpus=1 (for fermi1 – fermi4) and gpus=1 to 3 (for fermi5). ppn : number of processors per node ppn=1 to 4 (for fermi1 – fermi4) and ppn=1 to 16 (for fermi5). Note: CUDA Compilation Path for nvcc: For c shell users : Add the following lines in your .cshrc file
run the command source .cshrc For bash shell users : Add the following lines in your .bashrc file
run the command source .bashrc To Submit the Job Using Torque:
The user can change submit.sh to any other name he would want. Torque/PBS Commands 1. To check the status of the job
Gives the details of the job like the job number, the queue through which it was fired etc. 2. To remove a job from the queue
3. To know about the available queues
If you encounter any problem in using Torque please report to SERC helpdesk at the following email address helpdesk.serc@iisc.ac.in or contact System Administrators in 109 (SERC). |
SUPERCOMPUTER EDUCATION AND RESEARCH CENTRE
WELCOME TO SERC