Issue:
How can I setup a job to request a specific set of nodes and then if those nodes are not available use other nodes?
Solution:
One option you can use is NODESETS. The idea here with nodesets is that we search through a ordered list. The first nodeset being the most restrictive leading out to wider and wider groups of resources.
For example if you have 2 groups that have seperate resources but you also what part of those resources to overlap. If you wanted to request both group1 and group2 then you add a feature and nodeset. In this case "overlap" is the name I have chosen.
Example:
NODESETLIST generic,grp1node,grp2node,overlap,clusterwide
NODECFG[ibm10.ac] FEATURES=grp1node,clusterwide,overlap
NODECFG[ibm11.ac] FEATURES=grp2node,clusterwide,overlap
NODECFG[ibm13.ac] FEATURES=generic,clusterwide
NODECFG[ibm14.ac] FEATURES=generic,clusterwide
I tossed in a clusterwide to draw attention to a possibility of adding even more general resource allocation requests.