How can I get debug information without a core file?


Issue: 

Not all sites generate core files, yet source code information is neede by developers.

Solution: 

Adaptive Computing has a script that will get the relivant binary offsets from the backtrace information in the Moab log files, and look up the source information by reading the Moab executable.

Prerequisites: 

The scripot depends on having the opensource program addr2line installed.  This can be obtained separately, or it's installed as part of the binutils package.  

The Perl script will also need to be downloaded.  It's attached to this ticket, and can also be downloaded from http://files.adaptivecomputing.com/support/moab_addr2line.pl.gz.

Decompress the script and make it executable.  You can either place it someplace in the path, or run it specifying it's path. 

Using the Script:

You will need to run the script as root (either logged in as root, via sudo.  The first step is to identify which log file(s) have the backtrace information.  If your logs rotate frequently, you may want to make a copy of the log file(s).  A good way to find backtraces is by using grep, as will be shown below.  

Here is an example of what a typical run might look like:

root@root $ grep -l "Begin Moab Backtrace" /opt/moab/log/*
moab.log
moab.log.4

root@root $ cp /opt/moab/log
/tmp/root@root $ ./moab_addr2line.pl /tmp/moab.log
----------Begin Moab Backtrace----------
Version: moab server 9.1.1 (revision 2017040313, changeset cde619758eb7890901ade22e3fdd168f36c024d2)
Backtrace timestamp: 2017-09-28T17:03:25.809-0600
Caught Signal 11 (SIGSEGV) Invalid memory reference
kill(2) or raise(3). Sent by pid:28095 uid:0
MSysBacktraceHandler(int, siginfo_t const*, void*) | /opt/src/build/moab/9.1.1/src/moab/MSysSignalHandling.c:330
                              ?? | ??:0
                              ?? | ??:0
    MUSleep(long, unsigned char) | /opt/src/build/moab/9.1.1/src/moab/MTime.c:2103
MUIAcceptRequests(msocket_t const*, long) | /opt/src/build/moab/9.1.1/src/moab/MUIAcceptRequest.c:506
root@root $

Note that the logs can be copied to a different computer to analyze, but the Moab executable will also need to be copied, and will need to be in the location is was on the Moab server. 

Attached files: moab_addr2line.pl.gz

Tags: addr2line, backtrace, core file, source
Last update:
2018-04-27 18:58
Author:
Rob Greenbank
Revision:
1.0
Average rating:0 (0 Votes)

You cannot comment on this entry

Chuck Norris has counted to infinity. Twice.

Records in this category

Tags