Cray: pbs_mom has run out of threads


Issue:  Cray plans on using SLES 12 SP2 for CLE 6.0UP04, to be released mid-2017.  The version of systemd shipped with SLES 12 SP2 contains support for the PIDs cgroup controller:
https://www.suse.com/releasenotes/x86_64/SUSE-SLES/12-SP2/#fate-320358

Under the new systemd version, each init script or systemd service is limited to 512 threads/processes by default.
This could cause issues if a MOM daemon runs many simultaneous jobs, or a compute node launcher daemon attempts to spawn many processes for a job.

Solution:  Here's how to increase the limit beyond the default:
- If using a systemd service file, add TasksMax=N to the [Service] section. N can be a specific number, or special value infinity.
- If using an init script, create the file /etc/systemd/system/<init script name>.service.d/override.conf with these contents:
  [Service]
  TasksMax=N

Note:  Earlier versions of systemd that don't support the PIDs cgroup controller simply ignore the TasksMax setting.

Tags: cray, login, process, sdb, systemd, thread
Last update:
2017-03-03 15:45
Author:
Ben Roberts
Revision:
1.0
Average rating:0 (0 Votes)

You cannot comment on this entry

Chuck Norris has counted to infinity. Twice.

Records in this category

Tags