NWChem at MCSR

Users of NWChem need to be familiar with PBS Pro, our job management software.

NWChem is a computational chemistry package that was developed by the High Performance Computational Chemistry Group from the Environmental Molecular Science Laboratory (EMSL) at Pacific Northwest National Laboratory. The program is designed for parallel computer systems including parallel supercomputers and large distributed clusters. It takes advantage of available parallel computing resources and high networking bandwidth. It can perform many molecular calculations including density functional, Hartree-Fock, Möller-Plesset, coupled-cluster, configuration interaction, molecular dynamics, mixed quantum mechanics, geometry optimizations, vibrational frequencies, static one-electron properties, and periodic system modeling.

NWCHEM is available on MCSR's Altix 3700 (redwood), Origin 2800 (sweetgum), and Pentium IV Linux Beowulf cluster (mimosa). There are two flavors of NWCHEM on sweetgum and mimosa:

1. NWCHEM compiled with MPICH
2. NWCHEM compiled with TCGMSG.

For MPICH-enabled NWCHEM, the user needs to add the path of the nwchem executable file to the PBS batch file. This is an example of such a batch file for cluster:

#!/bin/csh
#PBS -l nodes=4
#PBS -l mem=400mb
#PBS -l cput=18000

setenv PATH /usr/local/apps/mpinwchem/bin/LINUX:/usr/local/apps/mpich/bin:${PATH}

cd `pwd`
mpirun -np 4 nwchem test.nw > test.out

If you get this error message:
exec: nwchem: not found
then specify the absolute path of nwchem to mpirun, rather than relying on PBS to find nwchem based on your path file.
Here is another version of the script for MPICH-enabled NWCHEM on the cluster, this time using bash, absolute pathnames, and only 2 nodes, where your working directory is called "nwchem" and located immediately under your home directory:

#!/bin/bash
#PBS -l nodes=2
#PBS -l mem=400mb
#PBS -l cput=18000

echo PATH=/usr/local/apps/mpinwchem/bin/LINUX:/usr/local/apps/mpich/bin:$PATH

cd $HOME/nwchem
/usr/local/apps/mpich/bin/mpirun -np 2 /usr/local/apps/mpinwchem/bin/LINUX/nwchem test.nw > test.out

The example of the batch file for Origin 2800: #PBS -S /bin/csh
#PBS -l ncpus=2
#PBS -l mem=1024mb
#PBS -l cput=720000
setenv PATH /usr/local/appl/mpich-1.2.1/bin:/usr/local/appl/mpinwchem-4.0.1/bin/SGITFP:${PATH}

cd `pwd`
mpirun -np 2 nwchem test.nw > test.out
For TCGMSG-enabled NWCHEM, the user needs to edit a PBS batch file and a process-group file. This file describes how many processes to start, what program to run, which machines to use, which directories to work in, and under which userid to run the processes. The process-group file is given a name: nwchem.p
An example of a process-group file for Origin 2800 is as follows:

chwang sweetgum 8 /usr/local/appl/nwchem-4.0.1/bin/SGITFP/nwchem /ptmp/chwang/scr

This causes 8 processes to run NWChem on sweetgum as user chwang in directory "/ptmp/chwang/scr"

An example of the batch file for Origin 2800 contains the following:

#PBS -S /bin/csh
#PBS -l ncpus=8
#PBS -l mem=1024mb
#PBS -l cput=200

setenv PATH /usr/local/apps/nwchem-4.0.1/bin/LINUX:${PATH}
cd /ptmp/chwang/scr

parallel nwchem test.nw > test.out.8

Since the Linux cluster has a different host name for each node, the process-group file is a little different from the parallel supercomputer. An example of the file for the cluster looks like this:

chwang node6-4 1 /usr/local/apps/nwchem-4.0.1/bin/LINUX/nwchem /ptmp/chwang/scr
chwang node6-3 1 /usr/local/apps/nwchem-4.0.1/bin/LINUX/nwchem /ptmp/chwang/scr

In this case, node6-4 and node6-3 need to be free nodes when the job is submitted to a two-node queue. To avoid the mistake of submitting jobs to busy nodes, the MCSR user needs to use the script “nwchemsub” located in in the /usr/local/apps/bin directory. In the pbs file, on the line immediately before running a 4-node job, the user should type:

nwchemsub 4 test.nw

This will create couple of files, in the user's current directory, for job submission: test.pbs, nwchem.p. These files will contain the information for submitting job to free nodes. If desired, the user can copy the nwchemsub script to his working directory and customize it for his needs. Here is an example of a bash pbs script to submit a 2-node NWCHEM job on mimosa, with input file "test.nw", using TCGMSG: #!/bin/bash
#PBS -l nodes=2
#PBS -l mem=400mb
#PBS -l cput=18000
export PATH=/usr/local/apps/nwchem-4.01/bin/LINUX:/usr/local/apps/mpich/bin:$PATH
cd $HOME/nwchem
nwchemsub 2 test.nw
parallel nwchem test.nw > nwchem_tcgmsg_bash.out

If you keep typing qstat to watch your job in the queue, almost as soon as the job you submitted moves into Run status, it will Exit and be replaced in the queue by the test.pbs job that was created by thenwchemsub line in your original pbs script:

[jghale@mimosa nwchem]$ qstat

Job id	Name	User	Time Use	S	Queue
13049.mimosa	nwchem_tcgmsg_b	jghale	00:00:00	E	SM-2D
13050.mimosa	test.pbs	jghale	0	Q	SM-2D