Issue:
Should Moab be reverting the standing reservations node list back after I modify it with mrsvctl?
Symptom:
SRCFG[mod3app2] PERIOD=WEEK DEPTH=3 SRCFG[mod3app2] ACCESS=DEDICATED USERLIST=jbooth SRCFG[mod3app2] RESOURCES=PROCS:8 TASKCOUNT=16 SRCFG[mod3app2] FLAGS=IGNIDLEJOBS,IGNSTATE,IGNJOBRSV,IGNRSV SRCFG[mod3app2] HOSTLIST=c4u1[1-9],c4u2[0-6]
Submit a 2 jobs to use 2 nodes:
[jbooth@honu ~]$ echo sleep 300 | msub -l walltime=300 -l nodes=c4u25 Moab.3 [jbooth@honu ~]$ echo sleep 300 | msub -l walltime=300 -l nodes=c4u26 Moab.4
[jbooth@honu ~]$ showres -n reservations on Thu Jul 16 11:24:02 NodeName Type ReservationID JobState Task Start Duration StartTime c4u11 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u12 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u13 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u14 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u15 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u16 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u17 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u18 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u19 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u20 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u21 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u22 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u23 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u24 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u25 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 Job Moab.3 Running 1 -00:00:05 00:05:00 Thu Jul 16 11:23:57 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 c4u26 User mod3app2.1 N/A 1 -00:04:19 2:12:40:17 Thu Jul 16 11:19:43 Job Moab.4 Running 1 -00:00:03 00:05:00 Thu Jul 16 11:23:59 User mod3app2.2 N/A 1 2:12:35:58 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:58 7:00:00:00 Sun Jul 26 00:00:00 16 nodes reserved
Modify the reservation to only consume nodes being used:
[jbooth@honu ~]$ mrsvctl -m -h c4u25,c4u26 --flags=force mod3app2.1 INFO: expression 'c4u25|c4u26' successfully matched 2 node(s) NOTE: hostlist changed for rsv mod3app2.1 [jbooth@honu ~]$ mrsvctl -m -h c4u25,c4u26 --flags=force mod3app2.2 INFO: expression 'c4u25|c4u26' successfully matched 2 node(s) NOTE: hostlist changed for rsv mod3app2.2 [jbooth@honu ~]$ mrsvctl -m -h c4u25,c4u26 --flags=force mod3app2.3 INFO: expression 'c4u25|c4u26' successfully matched 2 node(s) NOTE: hostlist changed for rsv mod3app2.3
[jbooth@honu ~]$ showres -n reservations on Thu Jul 16 11:24:50 NodeName Type ReservationID JobState Task Start Duration StartTime c4u25 User mod3app2.1 N/A 8 -00:05:07 2:12:40:17 Thu Jul 16 11:19:43 Job Moab.3 Running 1 -00:00:53 00:05:00 Thu Jul 16 11:23:57 User mod3app2.2 N/A 8 2:12:35:10 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 8 9:12:35:10 7:00:00:00 Sun Jul 26 00:00:00 c4u26 User mod3app2.1 N/A 8 -00:05:07 2:12:40:17 Thu Jul 16 11:19:43 Job Moab.4 Running 1 -00:00:51 00:05:00 Thu Jul 16 11:23:59 User mod3app2.2 N/A 8 2:12:35:10 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 8 9:12:35:10 7:00:00:00 Sun Jul 26 00:00:00 2 nodes reserved
After the next scheduling iteration the reservation reverts back to the original nodes in the standing resertvation.
[jbooth@honu ~]$ mschedctl -r scheduling will resume immediately [jbooth@honu ~]$ showres -n reservations on Thu Jul 16 11:24:56 NodeName Type ReservationID JobState Task Start Duration StartTime c4u11 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u12 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u13 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u14 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u15 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u16 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u17 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u18 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u19 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u20 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u21 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u22 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u23 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u24 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u25 Job Moab.3 Running 1 -00:00:59 00:05:00 Thu Jul 16 11:23:57 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 c4u26 Job Moab.4 Running 1 -00:00:57 00:05:00 Thu Jul 16 11:23:59 User mod3app2.1 N/A 1 -00:00:01 2:12:35:05 Thu Jul 16 11:24:55 User mod3app2.2 N/A 1 2:12:35:04 7:00:00:00 Sun Jul 19 00:00:00 User mod3app2.3 N/A 1 9:12:35:04 7:00:00:00 Sun Jul 26 00:00:00 16 nodes reserved
Solution:
This is expected behavior. Standing reservations are shepherded by Moab throughout their life cycle to ensure they match what is specified in the configuration. To modify a standing reservation you have to modify the moab.cfg and restart Moab. An alternative is to create the reservation with mrsvctl. In this way Moab will not enforce and shepard the reservation as it normally would with standing reservations.
Tags: mrsvctl, reservation