Should Moab be reverting the standing reservations node list back after I modify it with mrsvctl?


Issue:

Should Moab be reverting the standing reservations node list back after I modify it with mrsvctl?

 

 

Symptom: 

SRCFG[mod3app2] PERIOD=WEEK DEPTH=3
SRCFG[mod3app2] ACCESS=DEDICATED USERLIST=jbooth
SRCFG[mod3app2] RESOURCES=PROCS:8 TASKCOUNT=16
SRCFG[mod3app2] FLAGS=IGNIDLEJOBS,IGNSTATE,IGNJOBRSV,IGNRSV
SRCFG[mod3app2] HOSTLIST=c4u1[1-9],c4u2[0-6]

Submit a 2 jobs to use 2 nodes:

[jbooth@honu ~]$ echo sleep 300 | msub -l walltime=300 -l nodes=c4u25

Moab.3
[jbooth@honu ~]$ echo sleep 300 | msub -l walltime=300 -l nodes=c4u26

Moab.4
[jbooth@honu ~]$ showres -n
reservations on Thu Jul 16 11:24:02

NodeName             Type      ReservationID   JobState Task       Start    Duration  StartTime           

c4u11                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u12                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u13                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u14                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u15                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u16                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u17                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u18                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u19                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u20                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u21                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u22                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u23                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u24                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u25                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                      Job             Moab.3    Running    1   -00:00:05    00:05:00  Thu Jul 16 11:23:57
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00
c4u26                User         mod3app2.1        N/A    1   -00:04:19  2:12:40:17  Thu Jul 16 11:19:43
                      Job             Moab.4    Running    1   -00:00:03    00:05:00  Thu Jul 16 11:23:59
                     User         mod3app2.2        N/A    1  2:12:35:58  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:58  7:00:00:00  Sun Jul 26 00:00:00

16 nodes reserved

Modify the reservation to only consume nodes being used:

[jbooth@honu ~]$ mrsvctl -m -h  c4u25,c4u26 --flags=force mod3app2.1


INFO:   expression 'c4u25|c4u26' successfully matched 2 node(s)
NOTE:  hostlist changed for rsv mod3app2.1

[jbooth@honu ~]$ mrsvctl -m -h  c4u25,c4u26 --flags=force mod3app2.2


INFO:   expression 'c4u25|c4u26' successfully matched 2 node(s)
NOTE:  hostlist changed for rsv mod3app2.2

[jbooth@honu ~]$ mrsvctl -m -h  c4u25,c4u26 --flags=force mod3app2.3


INFO:   expression 'c4u25|c4u26' successfully matched 2 node(s)
NOTE:  hostlist changed for rsv mod3app2.3
[jbooth@honu ~]$ showres -n
reservations on Thu Jul 16 11:24:50

NodeName             Type      ReservationID   JobState Task       Start    Duration  StartTime           

c4u25                User         mod3app2.1        N/A    8   -00:05:07  2:12:40:17  Thu Jul 16 11:19:43
                      Job             Moab.3    Running    1   -00:00:53    00:05:00  Thu Jul 16 11:23:57
                     User         mod3app2.2        N/A    8  2:12:35:10  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    8  9:12:35:10  7:00:00:00  Sun Jul 26 00:00:00
c4u26                User         mod3app2.1        N/A    8   -00:05:07  2:12:40:17  Thu Jul 16 11:19:43
                      Job             Moab.4    Running    1   -00:00:51    00:05:00  Thu Jul 16 11:23:59
                     User         mod3app2.2        N/A    8  2:12:35:10  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    8  9:12:35:10  7:00:00:00  Sun Jul 26 00:00:00

2 nodes reserved

After the next scheduling iteration the reservation reverts back to the original nodes in the standing resertvation.

[jbooth@honu ~]$ mschedctl -r

scheduling will resume immediately

[jbooth@honu ~]$ showres -n
reservations on Thu Jul 16 11:24:56

NodeName             Type      ReservationID   JobState Task       Start    Duration  StartTime           

c4u11                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u12                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u13                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u14                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u15                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u16                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u17                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u18                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u19                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u20                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u21                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u22                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u23                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u24                User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u25                 Job             Moab.3    Running    1   -00:00:59    00:05:00  Thu Jul 16 11:23:57
                     User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00
c4u26                 Job             Moab.4    Running    1   -00:00:57    00:05:00  Thu Jul 16 11:23:59
                     User         mod3app2.1        N/A    1   -00:00:01  2:12:35:05  Thu Jul 16 11:24:55
                     User         mod3app2.2        N/A    1  2:12:35:04  7:00:00:00  Sun Jul 19 00:00:00
                     User         mod3app2.3        N/A    1  9:12:35:04  7:00:00:00  Sun Jul 26 00:00:00

16 nodes reserved

 

 

Solution:

This is expected behavior. Standing reservations are shepherded by Moab throughout their life cycle to ensure they match what is specified in the configuration. To modify a standing reservation you have to modify the moab.cfg and restart Moab. An alternative is to create the reservation with mrsvctl. In this way Moab will not enforce and shepard the reservation as it normally would with standing reservations.

 

Tags: mrsvctl, reservation
Last update:
2015-07-23 19:12
Author:
Jason Booth
Revision:
1.1
Average rating:0 (0 Votes)

You cannot comment on this entry

Chuck Norris has counted to infinity. Twice.

Records in this category

Tags