Adaptive Computing Inc - Why does TORQUE send the mom hierarchy to down nodes?

Why does TORQUE send the mom hierarchy to down nodes?

Issue: Why does TORQUE send the mom hierarchy to down nodes?

I have 6 node (out of 590) that are marked down and/or offline.

I have 5000+ messages in my server logs complaining that it couldn't connect and/or send the mom hierarchy to these nodes.

Shouldn't the server be intelligent enough to realize that these nodes (especially the ones marked "down") aren't available for such communication?

Solution:

This has been fixed. Please upgrade to5.1.3, 6.0.2.

Tags: mom hierarchy

Last update:: 2016-08-08 19:04
Author:: Jason Booth
Revision:: 1.0

Records in this category

How can I disable a queue / class without losing jobs?
My job exited with a code and I do not know what that code refers to.
How does ALPS integrate with TORQUE and what does a job life-cycle look like?
Moab/TORQUE dies silently with no core and no logging
Moab/TORQUE is reporting a job that is no longer on the node
My MPI jobs are failing to run when I run them through TORQUE however they run externally
How do I download TORQUE with wget?
Why do my jobs appear in batchhold when plenty of resources are available?
Why do starting jobs don't start, but hit walltime and are killed?
How to fix Mass Job cancellation creates DDOS on pbs_server (around 4.2.x)
Why will my job not start when there is no other job on the compute resource but CPU usage on that node is high?
moab: error while loading shared libraries: libtorque.so.2
ALPS error: apsched: required uid is missing
How does Matlab work with Moab and TORQUE?
Why does TORQUE leave many defunct moms behind?
How can I change a class/queue on a job?
How can I have Moab cancel a job if a node fails?
Why does a Job not cancel when a node has failed?
How can I run two different sets of TORQUE client commands on one system?
Torque not honoring $usecp to copy output files
How does Torque calculate and report memory usage?
Cannot release hold on job
The pbs_mom process is hanging on shutdown when I reboot a node.
Jobs missing from queue
Why does my job run on only 1 node?
Interactive jobs fail and then hang
Explanation of Exit Codes
How can I verify that mom to mom communication is happening on the right interface?
Error "MXM ERROR failed to create recv cq: Cannot allocate memory" when launching MPI jobs
"Bad ACL entry in host list" when running torque.setup (or when trying to add operators or managers in qmgr)
Why do I see "could not locate requested gpu resources" when the job requested none?
When using qsub -k what happens to the job output on the pbs_mom?
MPI is not launching the correct tasks on the correct node with 1.10.2
Jobs are exiting almost as soon as they run and restarting over and over again.
Getting Torque source code
How do you create custom Torque RPMs from source?
How to migrate queued jobs to a new Torque server with a different host name
How can I query the compute node processes that belong to a job?
How to get all cgroup cpuset cores with singlejob policy job submissions (without specifying procs)
Open MPI cannot locate tm.h or pbs-config
qsub: submit error (Request not allowed: Server shutting down)
TORQUE appears to be unresponsive with error (Pbs Server is currently too busy to service this request. Please retry this request.)'
TORQUE pbs_server will not start "pbsd_init failed"
What are the accounting "x attributes" in qstat -f?
Are there any known issues with using FQDNs instead of short hostnames for interactive/compute nodes for Torque/MOAB?
NUMA syntax memory default overrides job request
What '#' directives does msub pass to qsub?
TORQUE creates many files for array jobs with a large index.
Why does TORQUE send the mom hierarchy to down nodes?
TORQUE usecp appears to be broken in 6.0.1.
Can I mix Moab and TORQUE versions?
What to check after a crash.
How can I see which gpu is being used by a job?
Why can't I set a negative exit code for my jobs?
Permitted Torque job name characters
How do I adjust the wallclock time on an entire job array?
Torque will not compile when the tk-devel and tcl-devel packages are installed
qstat -f accounting logs sometimes report "resources_used.vmem=0"
"_bindir: Undefined variable." when using csh
Cray: pbs_mom has run out of threads
How to configure scheduling leeway for transient load
Updating serverdb file
Enabling and configuring support for cgroups in Torque
How to see array subjobs in showq
How do I change the log level for pbs_mom?
How to shut down Moab and Torque for a planned outage
Nodes show idle with zero cores
Why is my Torque prologue script unable to find the job file (.JB)?
Configuring a submit node
Bad UID for job execution MSG=ruserok failed validating user1 from ServerNode
job submission with no limit
How do I migrate server settings to a new server, and retain the job numbering?
How to connect to a web based program on a compute node? I.E. Port Forwarding
How do I use a submit filter to enforce submission rules?
X11 Forwarding Failing on qsub submission
How can I get Torque to recognize additional memory added to a node?
I see exit codes from Torque and showq, but what do they mean.
After upgrading to 6.1.2, I can only run single-node jobs. How do I fix this?
Why are my GPU nodes not being scheduled correctly?
Why is trqauthd failing to start during redhat/centos boot?
How do I use the support-diag.py script to upload site information?
Why does "msub" give the error "qsub: illegal -d value"?
How can I temporarily elevate log levels when requested?
Why are workload queries timing out and job completions not being detected?
How can I prevent users who forget to log out of interactive sessions from tying up resources?
How do I increase the stack size for my jobs?