This can be tricky but after some testing I found a method that seemed to constantly work. Example Below.


SRCFG[acusimres] HOSTLIST=compute-0-1,compute-0-2,compute-0-8,compute-0-9
SRCFG[acusimres] RESOURCES=gres:acusim:2
SRCFG[acusimres] TASKCOUNT=1
SRCFG[acusimres] PERIOD=INFINITY
SRCFG[acusimres] USERLIST=natetest

Here is what the mdiag -r looked like for the reservation

acusimres.1 User SHARED -00:07:06 INFINITY INFINITY 1 1 0
Flags: STANDINGRSV,ISACTIVE
ACL: RSV==acusimres.1= USER==natetest+
CL: RSV==acusimres.1
Task Resources: acusim: 2
SubType: StandingReservation
Attributes (HostExp='compute-0-1,compute-0-2,compute-0-8,compute-0-9')
Rsv-Group: acusimres
Rsv-Parent: acusimres


It seemed to work correctly in that it held two acusim tokens for my user natetest. In the test I had my main user 'nburton' take as many acusim tokens as he could and and then when his additional jobs were sitting idle because there were no more tokens I was able to have the natetest user in the reservation come along and grab the last two tokens. The actual showq looked like this. You can see how it went down by looking at the job ID sequence numbers.


[root@akamai native]# showq

active jobs------------------------
JOBID USERNAME STATE PROCS REMAINING STARTTIME

Moab.10 natetest Running 1 99:23:59:53 Fri Aug 21 11:13:36
Moab.1 nburton Running 1 99:23:59:13 Fri Aug 21 11:12:56
Moab.2 nburton Running 1 99:23:59:20 Fri Aug 21 11:13:03
Moab.3 nburton Running 1 99:23:59:21 Fri Aug 21 11:13:04
Moab.4 nburton Running 1 99:23:59:21 Fri Aug 21 11:13:04
Moab.5 nburton Running 1 99:23:59:21 Fri Aug 21 11:13:04
Moab.6 nburton Running 1 99:23:59:22 Fri Aug 21 11:13:05
Moab.9 natetest Running 1 99:23:59:52 Fri Aug 21 11:13:35

8 active jobs 8 of 121 processors in use by local jobs (6.61%)
2 of 11 nodes active (18.18%)

eligible jobs----------------------
JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME

Moab.7 nburton Idle 1 99:23:59:59 Fri Aug 21 11:13:04
Moab.8 nburton Idle 1 99:23:59:59 Fri Aug 21 11:13:04


Again this is a little tricky so I recommend you run this and test this on a test cluster or on a node or two that will be ok to block. One thing I ran into is that sometimes after changing the SRCFG lines and recycling Moab I noticed the changes might not appear in the reservation as per mdiag -r.

I believe this issue has been fixed in the latest version but for my testing I just blew-away the ck checkpoint files but I would think you should just be able to remove the reservation, recycle and then add it back as well, as we usually don't recommend removing the checkpoints as you will lose some of the stats. I just mention this in case you make a change and it doesn't seem to take it that you take a minute to examine the mdiag -r output to make sure it reflects your change.