Issue: Moab/TORQUE dies silently with no core and no logging

Affected Versions: All

Symptom: Sometimes Moab or TORQUE will die  silently without generating a core file.

Solution: In most cases Adaptive Computing will need a core file to address a crash. In rare cases a core file is not generated. To solve this do the following.

1. Ensure that your ulimit for core fles is set to unlimited "core file size          (blocks, -c) unlimited". Set with "ulimit -c unlimited" in the init script or omn the command line then execuite /opt/moab/sbin/moab.
2. You have enough disk space to write out a core file at about the same size as the Moab process memory footprint.
3. Check the permissions of $MOABHOMEDIR and verify that the user the process Moab runs as can write to that directory.
4. Check the running processes ulimit with "cat /proc/´pgrep moab´/limits"

If the above looks correct then and you still have no core file you can attach to the running process with gdb and continue until a core dump is detected.

1. gdb attach ´pgrep moab´ Note: you may have more then one Moab process. This is normal as Moab forks off from time to time. Watch top or ps to determin the static moab process and substitute PID where needed.
2. Once in (gdb) type "c". This will allow the Moab process to continue. You can start this in tmux, screen or a terminal session while you wait for a core dump. 
3. Once a coredump is triggered (gdb) will return to the (gdb) prompt. Execute the following commands
1. (gdb) bt
2. (gdb) generate-core-file
4. Copy the output of the "bt" command and send that into the support ticket.
5. generate-core-file will dump a core file to your current working directory where you issued gdb. Tar up the core file and your Moab/TORQUE binary.
6. Send in your core and Moab/TORQUE binary to:
1. scp <file>.tar.gz guest@ssh.adaptivecomputing.com:/home/guest/
2. password: hello