Privacy & Legal Notice
Quick Reference | User Manual | Example 1 | Example 2 | Help
Using ZeroFault to Find Memory Errors and Leaks in Large Parallel AIX Applications provides more details than the above "Quick Reference" about the design and use of zf_id, zf_all, zf_idx, and zf_range. It also describes in much more detail the various false positives seen in our codes (slides 13-15) and describes some of the user experiences with ZeroFault (slides 16-17).
The ZeroFault User Manual describes in detail how ZeroFault works, the options for use, how to use the GUI features, and more details about what causes false positives.
The examples below briefly illustrate how to use a few key features of ZeroFault and the ZeroFault scripts.
The source, executables, and ZeroFault output for the following two examples can be found in /usr/local/zf/examples (on Berg or Frost). You should be able to run zf_ui directly on the example .zfo files, but you need to copy the executables to a different directory if you wish to rerun ZeroFault on them.
This example uses a pathologically bad MPI program, mpi_zf_demo, that demonstrates eight common memory errors. This executable was built from mpi_zf_demo.c with the following compile line:
> mpxlc -g mpi_zf_demo.c -o mpi_zf_demo
As described in zf_all's usage information, to run ZeroFault on all tasks, zf_all is just placed between "poe" and your application name. It may be used either interactively or in psub scripts. Here is an example of running mpi_zf_demo interactively in the debug partition with one node and two MPI tasks:
> poe zf_all mpi_zf_demo -nodes 1 -procs 2 -rmpool 0
0:Making directory zf_0 for holding ZeroFault task 0 output
1:Making directory zf_1 for holding ZeroFault task 1 output
0:Running ZeroFault (zf -o zf_0 -d none) on task 0 on blue268.llnl.gov
1:Running ZeroFault (zf -o zf_1 -d none) on task 1 on blue268.llnl.gov
0:Number of tasks= 2 My rank= 0
1:Number of tasks= 2 My rank= 1
0:Null ptr (0) read returned 0
1:Null ptr (0) read returned 0
0:Sum over uninitialized data for rank 0 is 2327
1:Sum over uninitialized data for rank 1 is 2327
The ZeroFault GUI can then be brought up on task 0's ZeroFault output by using zf_ui as follows:
> zf_ui zf_0/mpi_zf_demo.1.zfo
(If you wish a live demo of the zf_ui GUI for this example run, you can run the above command in /usr/local/zf/examples on Berg or Frost. Note: Remember that you need to have your NUM-LOCK off in order to have your mouse clicks recognized by this AIX-based GUI.)
The initial zf_ui screen looks like this:
Eight potential (and in this case real) errors are presented by the ZeroFault GUI. By default, errors in the same function are condensed into groups. Although clicking the double arrow beside "2 BMW" or "5 UMR" would peel off one message from the group for further examination, I prefer to change the "Condense By: Function" option to "Condense By: Error Location", which groups the errors by line as shown here:
Clicking on any single arrow causes the GUI to display more information about that line (including displaying source, if available). In the screenshot above, the BMW on line 30 was expanded, showing the source in the bottom pane and explaining why ZeroFault believes it is a problem (writing to an address, 0x20d4d4ac, one character past the end of the buffer, 0x20d4d44ab). (If the source doesn't appear, clicking on "Source Path" and then "Edit Current Path" allows additional source directories to be specified, one per line. Note: You will have to close and open an arrow for the new source path to be used to update the source display.)
In this example, all eight messages are real errors and the user is encouraged to examine them by running "zf_ui zf_0/mpi_zf_demo.1.zfo" in /usr/local/zf/examples on Berg or Frost. In many of our applications, there will also be false positives reported that our message filters do not catch, so users are encouraged to first focus on messages located directly in their code and then focus on library routines called from their code (such as MPI_Isend, which as demonstrated above, can cause BFCP warnings and other warnings due to invalid user parameters that cause the library routines to cause read or write unallocated or uninitialized memory.)
This example also demonstrates what ZeroFault considers "physical memory leaks." Clicking on "Show Leaks" displays the following screen for this example program:
Although a signficant number of leaks are displayed by the GUI, only one leak is caused by something under the user application's control (line 26 of mpi_zf_demo, shown expanded in the screen shot). There are no filters available for the "Memory Leaks" screen, so the users should just focus only on leaks that occur in their application (because nothing can be done about the other ones). ZeroFault defines physical memory leaks as occurring only when a pointer to allocated memory is lost. In this example, the pointer "buf" is pointed at newly allocated memory on line 26 and then pointed at different allocated memory on line 34. At this point, the pointer to the memory allocated on line 26 is "lost" and therefore this memory can never be freed.
Although the memory allocated on lines 44, 47, and 51 (not shown) are never freed, the pointers to these memory still exist at program termination, so ZeroFault classifies these leaks as "logical memory leaks" and does not show them by default on this screen. For many applications, letting the operating system free all their memory at once on exit, instead of doing a large number of calls to free, is much more efficient. As a result, displaying logical memory leaks often hides useful information from the user and for some of our applications, overwhelms both the GUI and the user. The next example demonstrates a technique to find logical memory leaks where the memory footprint grows slightly with each computational cycle, exhausting memory after running for many cycles.
This example demonstrates how to use zf_idx (interactive ZeroFault on one MPI task id) and memory snapshots to find the logical memory leaks that are occurring in the computational cycle of mpi_snapshot_demo, a small MPI program. For this technique to work, it must be possible to pause the execution before and after the computational cycle that is unexpectedly growing the application's memory footprint. Although, some programs provide interactive prompts that allow stepping through computational cycles, in this case a hard coded "hit enter to continue" point was placed before and after cycle 2, with a barrier to guarantee all tasks are kept in step for this computational cycle. This executable was built from mpi_snapshot_demo.c with the following compile line:
> mpxlc -g mpi_snapshot_demo.c -o mpi_snapshot_demo
As described in zf_idx's usage information, to run ZeroFault interactively on one task (say task 2), place "zf_idx 2" between "poe" and your application name. The zf_idx script may only be used interactively, either in the debug pool or using the "batch xterm trick" to run a large program interactively in the batch pool. Here is an example of running mpi_snapshot_demo interactively in the debug partition with one node and four MPI tasks (and running ZeroFault on task 2).
> poe zf_idx 2 mpi_snapshot_demo -nodes 1 -procs 4 -rmpool 0
2:Making directory zf_2 for holding ZeroFault task 2 output
2:Running ZeroFault w/GUI (zf -o zf_2) on task 2 on blue268.llnl.gov
2:Warning: Cannot convert string "courb12" to type FontStruct
2:Warning: Cannot convert string "courr12" to type FontStruct
2:Warning: Cannot convert string "courr12" to type FontStruct
0:Number of tasks= 4 My rank= 0
1:Number of tasks= 4 My rank= 1
3:Number of tasks= 4 My rank= 3
2:Number of tasks= 4 My rank= 2
0:Start cycle 1:
0:End cycle 1:
0:Paused, take ZeroFault snapshot, then hit enter to run cycle 2:
0:Start cycle 2:
0:End cycle 2:
0:Paused, take ZeroFault snapshot, then hit enter to run cycle 3:
0:Start cycle 3:
0:End cycle 3:
0:Start cycle 4:
0:End cycle 4:
At the beginning of this run, the following ZeroFault GUI was displayed:
Notice that "Take Snapshot" is not grayed out. Once the program reached the first pause point (before cycle two), a memory snapshot was taken by clicking on the "Take Snapshot" button, displaying the following:
As indicated in the above screenshot, this created the snapshot mpi_snapshot_demo.1.zfl.1 in the zf_2 directory. The program execution was advanced to after cycle 2 by hitting "enter" in the execution window and then another snapshot was taken (mpi_snapshot_demo.1.zfl.2 in the zf_2 directory). After clicking on "Compare Snapshots" and selecting these two snapshots in the zf_2 directory, the following GUI was displayed:
The screenshot above was taken after all the arrows were clicked to display all the information about the two blocks allocated but not freed during cycle 2. It shows that the first block of 20000 bytes was allocated on line 72 in main() and the second block of 8 bytes was allocated on line 20, in the function add_node(). Assuming the memory used should be in steady state in cycle 2, then this ZeroFault memory comparison indicates that these two allocations should be examined further to see why they were not freed at the end of cycle 2.
Note: The MPI application mpi_snapshot_demo (or anything invoked with zf_idx) runs exactly as long as the ZeroFault GUI does. If you close the GUI, the application terminates (in this case, task 2 terminates, which kills the application). If you leave the GUI up, you will hold onto all the nodes until the time limit is reached and both the application and GUI will be killed.
The snapshots can be also compared after the fact using zf_ui. Running "zf_ui zf_2/mpi_snapshot_demo.1.zfo" in /usr/local/zf/examples on Berg or Frost brings up the initial GUI. Clicking on "Compare Snapshots" and selecting the two zfl files in the zf_2 directory will bring up the memory comparison above.
Please contact John Gyllenhaal (via e-mail to email@example.com or telephone 925-424-5485) if you would like help using ZeroFault or interpreting ZeroFault's output, or if you are having problems with ZeroFault or the ZeroFault scripts.
ZeroFault, Zerofault, "Zero Fault", ZEROFAULT, "zero fault", zerofault