by
dplace is a tool for controlling placement of processes onto cpus in SGI systems. SGI systems have two major operating systems as Irix (Unix) and GNU/Linux for SGI. Even though the core of the systems are Unix/Linux based, some tools and usages of them might differ from OS to OS as dplace.Thus, this article will explain the dplace tool without specifying any flags and options.
When a program/code is executed after scheduling, memory placement policies are set up according to command line arguments. By default, memory is allocated to a process on the node that the process is executing on. If a process moves from node to node while it running, a higher percentage of memory references will be to remote nodes. Remote accesses typically have higher access times. Process performance may suffer. Dplace is used to bind a related set of processes to specific cpus or nodes to prevent process migrations. In some cases, this will improve performance since a higher percentage of memory accesses will be to the local node. Dplace maintains a global count of the number of active processes that have been placed (by dplace) on each cpu.
Dplace supports 2 placement modes: load balanced and exact placement. If load balanced placement (default) is selected, dplace will bind a process to the cpu that has the lowest number of processes that were placed by dplace AND is also in the user’s cpu list.
For example, if the current cpumemset consists of physical cpus 2, 5, 8, and 9, and the user types:
mpirun -np 2 dplace -s1 app1
mpirun -np 2 dplace -s1 app2
app1 will run on cpus 2 & 5, and app2 will run on cpus 8 & 9. This assumes that no other processes that were placed by dplace are still running on these cpus. If exact placement is selected "-e", processes are bound to cpus in the exact order that the cpus are specified in the cpu list. Cpu numbers may appear multiple times in the list. A cpu value of "x" indicates that binding should not be done for that the process. If the end of the list is reached, binding starts over at the beginning of the list.
Another most used flag beside -s is -c which refers to "CPU numbers" in dplace tool and it is specified as a list of cpu ranges: Example: "-c1", "-c2-4", "-c1,4-8,3", or as shown below.
mpirun -np $NCPUS dplace -c1-$NCPUS -s1 app3
app3 will run on cpus assigned by cpumemset where $NCPUS are number of cpus to run the application. Cpu numbers are NOT physical cpu numbers. They are logical cpu number that are relative to the cpus that are in the set of allowed cpus as specified by the current cpumemset or "runon" command. A cpu value of "x", in the argument list for -c option, indicates that binding should not be done for that process. "x" should be used only if the -e option is also used. Cpu numbers start at 0. If this option is not specified, all cpus of the current cpumemset are available.
Note that, on redwood, cpusets have not yet been defined by MCSR support staff, so the “current cpumemset” for each users is the set of all CPUs on the system (0 through 63).
We have found that the Amber7 benchmarks work more efficiently when dplace is used. The following line taken from the PBS script at Amber at MCSR shows how Amber7 may be invoked using dplace:
mpirun -np $NCPUS dplace -c1-$NCPUS -s1 nucgen -O -i nuc.in
-o nuc.out -d $AMBERHOME/dat/leap/parm/nucgen.dat -p nuc.pdb
For more information about the dplace tool and its features, you may simply search for it in the man pages of an SGI system or visit SGI 's techpub support system online.
