Gaussian 09- Tyrone Cluster

Running jobs on Tyrone Cluster

Introduction

G09 software can be used only in parallel mode on the tyrone cluster.

Environmental set up for Tyrone-Cluster:

Gaussian 09 is installed in the directory /home/pkg/lic/g09. The Gaussian 09 system includes paths and initialization files to set up user environment for running the program. The paths to be set are

1. The variable g09root must be set to the installation directory of gaussian 09.
csh:   setenv g09root /home/pkg/lic/g09/amd64

bash/ksh:   export g09root=/home/pkg/lic/g09/amd64

sh:   export g09root=/home/pkg/lic/g09/amd64

2. Initialization files to be set up:

csh:   source /home/pkg/lic/g09/amd64/g09/bsd/g09.login

bash/ksh:    /home/pkg/lic/g09/amd64/g09/bsd/g09.profile

sh:   /home/pkg/lic/g09/amd64/g09/bsd/g09.profile

3. Setting scratch directory for G09 runs using GAUSS_SCRDIR environmental variable. This should be set to the base directory where scratch files are to be stored.

csh:

setenv GAUSS_SCRDIR /localscratch/loginid

bash/ksh:

export GAUSS_SCRDIR= /localscratch/loginid

sh:

GAUSS_SCRDIR= /localscratch/loginid ;

export GAUSS_SCRDIR

where, loginid is your computational login id.

Note:
  1. Once your job is complete remove all temporary files created by your job in /tmp or /global/utemp/loginid .
  2. Files in /localscratch/loginid older than 7 days are removed automatically
  3. Using ssh for multi-node jobs:

Gaussian 09 multi-node jobs use Linda libraries.Linda by default uses ‘rsh’ to communicate across multiple nodes. However the tyrone-cluster is only enabled with ‘ssh’ for communication. To use linda with ‘ssh’ Please include the following directive in your job scripts:


csh:

setenv GAUSS_LFLAGS '-opt "Tsnet.Node.lindarsharg: ssh"'

sh/bash:

export GAUSS_LFLAGS ='-opt "Tsnet.Node.lindarsharg: ssh"'

4. Number of processors to be used:

On the tyrone-cluster a user can use maximum of 32 cores on a single node or multiples of 32 cores for multi-node jobs. The resource allocation to a job is done by the “torque” batch scheduler based on the job script submitted by the user and available free resources on the cluster. Whereas, for the software G09, the task creation for multi-core or multi-node job is done by Linda software. Hence, it is necessary that the resources allocated by “torque” be communicated to “Linda”. This can be done by creating a “Default.route” file in the current working directory of the job, and adding the “torque” allocated resource list to this file. This can be done in the job script file by adding the following lines, after the resource specification directive to “torque”:

LINDA =  `cat $PBS_NODEFILE | uniq | tr '\n' "," | sed 's|,$|:4|' | sed 's|,|:4,|g' `
echo "-W- " $LINDA > ./Default.Route

The PBS_NODEFILE is a shell environment variable which contains the names of the nodes that have been allocated to the job by “torque”. The above two lines take input from the PBS_NODEFILE and transform to a format acceptable by “Linda” software and write it to the “Default.route” file. This “Default.route” file has to be present in the current working directory of the submitted job.

Linda software uses two directives “%LindaWorkers” and %NProcShared” for parallel jobs. The “%NProcShared” directive is used to create the number of threads per Linda task and should be typically set to the number of cores supported per CPU(socket) on a node. For the “tyrone-cluster”, “%NProcShared” should be set to eight and this directive must be defined with the G09 input, on the command line or input file. The other directive, namely, “%LindaWorkers” is used by the “Linda” software to define the number of tasks to be created for the job. Typically, this variable must be set to the number of CPUs(sockets) required by the job. For the tyrone-cluster, defining this variable is not necessary since, the “Default.route” file contains this information.

5. Running Gaussian 09 jobs in batch mode.

To run the Gaussian 09 jobs in batch mode set up the environment as above, create a Torque job script file as described below:

1. 32-core G09 sample job script

2. 64-core G09 sample job script

3. 128-core G09 sample job script

4. 256-core G09 sample job script

Software documentation:

Gaussian 09 links / help
http://www.gaussian.com/g_tech/g_ur/g09help.htm

Report problems to:

For any problems with respect to G09 software contact helpdesk.serc@auto.iisc.ac.in or system administrators in room no. 103, SERC.