TotalView 6.1.0-2

Release Features

February 19, 2003

TotalView is a product of Etnus, LLC.

This documents the release of TotalView 6.1.0-2 installed as totalview and totalviewcli on February 19, 2003, on blue/snow/frost/sky/white, also gps/tc2k/sc.

Etnus 6.1.0-2 Release
§ TotalView 6.1 Documentation -- New Features, Getting Started, Reference Guide, User Guide, Release Notes
§ Symbol table name scoping and line number handling
§ Ultra-scale command execution performance enhancements
§ Support for AIX 5.1
§ Non-uniform address space distributed debugging
New Features
§ Diveinall: Display of arrays of structure components
§ Parallel evaluation of expressions
§ Memory utilization statistics
§ Option for Displaying C and C++ Pointer Variables as Arrays
§ Option for Setting/Modifying Data Display Format and Precision
§ Forward-Backward Browser Buttons on Nested Dive
§ Variable Browser
§ Process-thread set (ptset) grouping in GUI
§ Support for calling functions in CLI expressions
§ Window Positions Memorized -- use Window>Memorize, Window>Memorize All
§ Single Click Dive Using Middle Mouse button
§ Previous LLNL/LANL Enhancements to TotalView
§ TotalView 6.1.0-3-LLNL Release Features

TotalView Debugger Tutorial by Blaise Barney
TotalView Quick Reference Page
TotalView CLI Summary Sheet

For problems/questions, send e-mail to Matt Wolfe, Dong Ahn, Karen Warren (LLNL) or Laurie McGavran (LANL).

Display of arrays of structure components

TotalView provides data window Dive In All and Dive In All Anew menu items that restrict the display of a structure array to only one component. Using the right mouse button, click on the desired member and select Dive In All to see all those array members.

Parallel evaluation of expressions, i.e., expression windows evaluate across all or subset of processes

When the Focus selections are Group and Workers, TotalView performs parallel evaluations and displays the values for all the workers in the group.

Memory utilization statistics

Totalview provides a display, Tools>Memory Statistics, of memory statistics including heap size, data size, text size, stack size for all processes. The display also gives mininum and maximum values and the option to sort. This info is also available using the CLI command, dmstat.

Option for Displaying C and C++ Pointer Variables as Arrays

The user can use the preferences menu, File>Preferences, to This option is the default for the LLNL version.

Option for Setting/Modifying Data Display Format and Precision

The user can select and modify the data display format and precision using the File>Preferences Formatting tab.

Forward-Backward Browser Buttons on Nested Dive

The source and data panes have backwards and forwards buttons bound to the undive and redive operations.

Process-thread set (ptset) grouping in Motif user interface

TotalView provides a GUI interface that allows the user to specify a subset of processes and/or threads. The Tools>P/T Set Browser Window displays the current set of processes and threads selected by the user using the P/T Set controls: Width Control, Group Control, P/T Set Selector Control and a P/T Set Tuning Control. The user can also specify a new group using the process window Group>New Group tab.

Caching of Type Casts

In order to increase efficiency, the typecasts of a variable made by the user are now cached (by default) and are used for subsequent dives on the given variable. This action can be turned off by using the Preferences->Options menu.

The cache can be completely purged at any time, or saved to a file, or loaded from that file by selecting the appropriate option on the Tools menu in the process window.
 >Purge Entire Type Cast Cache   --Removes every element associated with 
				   current debug target from the cache.   
 >Save Type-Cast Cache to File   --Generates 2 files: 
				   program name.TVD.breakpoints.cache 
				   and program name.TVD.breakpoints.index. 
				   The index file contains a unique key for
				   each process and offset in the cache file. 
				   Caches are unique to the process.   
 >Load Type-Cast Cache from File --Loads previously saved cache info into 
				   current debugging session.  If an element 
				   already exists in the cache, this operation 
				   overwrites it. 
				   Automatic cache loading is not yet supported.  

The data window Tools menu allows the user to remove that variable from the typecast cache, or, for debugging purposes, allows the user to print out all the cache elements in the current process cache.

Remembering Data Window Layout

Users often restart their code and run to the same Program Counter (PC). Then they examine the same set of variables. Since restart tears down all the data windows, users must resize and rearrange the data windows redundantly. This enhancement eliminates redundant data window resizing and/or rearranging. Selecting (currently default) the "Remember data window positions" as shown in the File>Preference window causes TotalView to retain the window geometry of data windows. With this option on, variables are redisplayed in their previous locations before the restart.

Typically a data window gets enlarged for variables of more complex data type and data windows of the same process are grouped together. Below is a typical debugging session where a user resizes and arranges the data windows according to the data type of a variable and its process (e.g., mpi process). The window geometry on closing is remembered by this enhancement.

Lightweight Corefile Viewer -- IBM SP's

In order to make use of the lightweight corefiles that are produced in subdirectories, one per process, when a parallel run exits abnormally, this enhancement uses the Detailed Lightweight Corefile Viewer (decor). To use, start up totalview with no arguments. In the root window select Tools>Detailed Lightweight Corefile Viewer(decor):

Type the relevant information in the window below:

Clicking on OK invokes decor which presents a folded call tree constructed from the multiple lightweight corefiles. The number of threads in each function is presented as a number enclosed in parentheses. Functions in the tree that are colored blue indicate there were threads currently in that function when the job stopped. Functions colored red indicate that a fault occurred in that function.

Invoking decor also calls up a source window in which action points can be set as the user analyzes the call tree info.

Users can use decor's functionality for the analysis of the lightweight corefile output. Options include filtering and a function stack dump on a thread's segfault as shown below. The user can click on tree nodes to get more information about the threads in that node. By selecting the thread information and pressing the "Show Thread Call Stack" button, the user can get the call stack for the selected thread. The "Show Source" button can be applied to selected threads to display the source file if that information is available. The tree can be filtered to show only stack traces from faults or from specific tasks by selecting menu entries from the "Filter" menu. There is a "Threads Per Function" report which will display the total number of threads in each function.


If the user saves the contents of a variable window using File>Save Pane, then he may compare the value(s) at a later point by using Tools>Compare. The user first gives the name of the file that was used to save the previous data.

CLI commands dsave/dcompare

Two commands, dsave and dcompare, have been added to the CLI to give the same capability as the Save/Compare commands in the window version.

dsave will save the variable data in a temporary file.

dcompare will compare the variable with the data in that file.


	d1.<> dsave a 

	d1.<> as a(3,4) 7

	d1.<> dcompare a 
	< (3,4)   3.464102e+00 
	> (3,4)   7.000000e+00 

Display information across processes in CLI: dlaminate, dfind

We have created a Tcl script that defines functions (commands) to allow displaying information across processes from within the CLI. This script is included in the TotalView startup. The new commands are:

CLI command to print statistics for C++ STL containers with integer or double types:

The following Command Line Interface commands are now available for printing statistics for STL containers with types of integer/double:

d1.<> stl_stat DoubleData
max at index ( 4 )  5.550000000000000e+01
mean. value of DoubleData = 35.5 
min at index ( 0 )  1.550000000000000e+01
sum value of DoubleData = 177.5 
std_dev value of DoubleData = 14.1421356237 

CLI command to print multiple variables: dmprint

The following Command Line Interface command is now available for printing more than one variable or expression:

d1.<> dmprint a x {b[2:5]}
 a = 0x00000016 (22)
 x = 3.1416
 b[2:5] = {
   [2] = 0x00000007 (7)
   [3] = 0x00000009 (9)
   [4] = 0x00000000 (0)
   [5] = 0x00000002 (2)
d1.<> dmprint "3*(a+2)"
  3*(a+2) = 0x00000048 (72)

Periodic ActionPoint Saving

Actionpoints are periodically saved and subsequently available for restarts and new runs. This default action may be turned off using the Preferences>Action Points menu.

Type-Sensitive Data Window

To save screen space, GUI windows for scalar data objects are smaller in size than the usual data window:

No display of poe window.

Tcl script has been added to the global .tvdrc file to eliminate the display of the poe window for parallel runs.

LLNL Disclaimers