There are three different types of logs that you may be asked to provide with elevated log levels: Moab, Torque server, and the compute node Torque MOM. While They can all have their log levels set permanently in their configuration files, that often is not desirable. It also can require a restart of the daemon, and the change is permanent until the configuration file is changed again. The Adaptive Computing documentation provides those configuration settings.
It is often desirable, however, to only temporarily change the log level until the desired information (often requested by Adaptive Computing support) has been collected. This document addresses the ways that is accomplished.
In the examples below, the text <level> should be replaced by the numeric value of the desired log level (typically 0 through 7, but sometimes as high as 11).
The methods for setting new log levels without restarting the daemons are as follows:
For the Moab server:
Determine current level: mschedctl -l config | grep LOGLEVEL
Set current level: mschedctl -m config LOGLEVEL <level>
Torque server logs:
Determined current level: qmgr -c "p s" | grep log_level
Set current level: qmgr -c "set server log_level=<level>"
Torque MOM logs:
There are a couple of different ways to elevating MOM logs, without restarting the MOM daemon. The first one is easy. Use the following commands to query, or set the loglevel:
Query the log level(s): momctl [-h <host-list>] -q loglevel
Set the log level(s): momctl [-h <host-list>] -q loglevel=<level>
The <host-list> is optional, with the current host being the default. Multiple nodes can be specified in the <host-list> by spearating them with commas. For example:
momctl -h node00,node01,node05,node16 -q loglevel=7
Also, restarting the pbs_mom daemon will revert logging to the configured (or default) level.
Another (more complicated) way is to send the pbs_mom process specific "kill" signals. You can determing the process ID (PID) of the pbs_mom process, as well as it's current log level, by running this command on the compute node: "momctl -d3".
Using the PID value (<pid> in examples below), you can increase the log level by one by sending the process a SIGUSR1 signal, or decrease by one by sending SIGUSR2. For example, if the current log level is found to be 3, and you need to provide level 7 logs, you would run the following four times to acheive that log level:
kill -s SIGUSR1 <pid>
To return to level 3, once you have collected the required information, run this four times:
kill -s SIGUSR2 <pid>