Why does my job run on only 1 node?


Issue: Why does my job run on only 1 node?

Example:

Resource_List.nodes = 10:ppn=1

 The job then runs on only 1 node and not 10 different nodes. Here is an example of echoing out each thread on the host that my mpi job ran under. 

n8
n8
n8
n8
n8
n8
n8
n8
n8
n8

I would have expected 10 different nodes

n8

n9

n10

n11

n12

n13

n14

n15

n16

n17

n18

 

Solution: 

By default Moab will stack as many threads on a node as possible. You can change this behavior with: 

At submission time:

NMATCHPOLICY
Format One of the valid settings for the parameter JOBNODEMATCHPOLICY
Description Specifies how node resources should be selected and allocated to the job.
Example
> qsub -l nodes=2 -W x=nmatchpolicy:exactnode bw.cmd

Job should use theEXACTNODEJOBNODEMATCHPOLICY.

 

Cluster wide policy:

JOBNODEMATCHPOLICY
Format AUTO, EXACTNODE, or EXACTPROC
Default AUTO
Description

Specifies additional constraints on how compute nodes are to be selected.

  • AUTO overrides the JOBNODEMATCHPOLICY (packs the jobs on any node).
  • EXACTNODE indicates that Moab should select as many nodes as requested even if it could pack multiple tasks onto the same node.
  • EXACTPROC indicates that Moab should select only nodes with exactly the number of processors configured as are requested per node even if nodes with excess processors are available.
Example
JOBNODEMATCHPOLICY EXACTNODE

In a PBS/Native job with resource specification nodes=<x>:ppn=<y>, Moab will allocate exactly <y> task on each of <x> distinct nodes.

 

 

 

Docs: NodeMatchPolicy

 

 

Tags: mpi, NMATCHPOLICY, same node
Last update:
2016-01-26 18:23
Author:
Jason Booth
Revision:
1.2
Average rating:0 (0 Votes)

You cannot comment on this entry

Chuck Norris has counted to infinity. Twice.

Records in this category

Tags