Issue: How can I sync my Moab and TORQUE batch job IDs?

Affected Versions: All

Symptom: When submitting jobs into Moab you will see a jobid for Moab and for TORQUE that do not match up. For example from showq -v 2965/38465.localserver.pbs

Solution: Set the following in Moab. This will allow Moab to tell the resource manager that it should honor the Job ID Moab sends it.

RMCFG[torque] TYPE=PBS SYNCJOBID=TRUE

RMCFG[torque] FLAGS=PROXYJOBSUBMISSION

 RMCFG[internal] JOBIDFORMAT=INTEGER

Sync Job ID documentaion link.

Additionaly the pbs_server should allow the server, that Moab is running on, to make admin level updates. This can be set via the qmgr setting: "set server managers+=" .

Additional Considerations: Currently it is not possible to sync every jobs job ID. For example if an interactive job is submitted through qsub then TORQUE will give that job the next job ID found in qmgr, "set server next_job_number = 83397", regardless of what Moab thinks the next job ID should be. Likewise when using SLURM, SLURM will do the same. It is suggested that you use a different jobID range for interactive jobs or jobs submitted via the resource manager. For example you can say all non-interactive jobs consume job IDs 1-800,000. Then in the resource manager you configure the job IDs to be from 900,000 to one million. Outlined below is an example configure with a different job ID range for scheduler and resource manager.

moab.cfg

SCHEDCFG[Moab]   MAXJOBID=499999 MINJOBID=1000
RMCFG[slurm]    SYNCJOBID=TRUE EPORT=10777
RMCFG[internal]         JOBIDFORMAT=integer

slurm.conf

FirstJobId=500000
MaxJobId=1000000