TotalView 5.0.0-0-LLNL

Release Features

June 25, 2001

TotalView is a product of Etnus, LLC.

This documents the release of TotalView 5.0.0-0-LLNL installed as tv5 and tv5cli on June 21, 2001 on blue/baby/snow/frost/sky/white.

On compass/tc2k/tc/forest/sc, the latest Etnus binary release, TotalView 5.0.0-1, is installed as tv5 and tv5cli. It was installed June 12, 2001. The Etnus binary release does not include any of the LLNL/LANL feature enhancements.

Below are the features added to create TotalView 5.0.0-0-LLNL.

Etnus 5.0 Release
§ Etnus release notes for 5.0.0-0
§ New Motif User Interface
§ TotalView 5.0 Documentation -- Upgrading Guide, User's Guide, Window Descriptions, CLI Guide
§ User Preferences
§ Start-Up Information
§ Recursive directory search for files
§ Faster TotalView performance of execution and display commands on large parallel codes
§ Support of IBM Fortran/C OpenMP compiled code
§ Asynchronous execution control of threads
§ Allow users to define subgroups for processes and threads in CLI
§ Display and handle C++ long identifiers
§ Integrated graphical Message Queue display
§ Integrated graphical Call Tree display
§ Support attach to subset of processes
§ Support executing non-interactive CLI scripts
LLNL Implemented New Features
§ Scripts to support C++ STL vector for IBM xlC, Compaq cxx, g++ and KCC compilers
Incorporated previous LLNL/LANL 4.1 Features
§ Global Display Format
§ Root Window Condense
§ Memory Utilization Window
§ What's New Information
§ Condense/Uncondense
§ Format - octal, hex, decimal, dump, scientific
§ Save/Compare
§ Save to History Window
§ Vector index
§ Display structure array member
§ Better thread names in thread pane
Previous and Future Releases
§ Previous LLNL/LANL Enhancements to TotalView
§ TotalView V5.0.0-5-LLNL Release Features

TotalView 5.0 Documentation -- Upgrading Guide, User's Guide, Window Descriptions, CLI Guide
TotalView Debugger Tutorial by Blaise Barney
TotalView Quick Reference Page
TotalView CLI Summary Sheet

For problems/questions, send e-mail to Bor Chan, Karen Warren (LLNL) or Laurie McGavran (LANL).













New Motif User Interface

With this release TotalView has changed from it's X window interface to an interface based on Motif. In addition, many new graphical capabilities are available based on TotalView's CornerStone graphics package. You can now view call trees and MPI message queues graphically.

TotalView menus are displayed on pull down menu bars, no longer on the middle mouse button. The mouse buttons are used as follows:

Below is what the new root window, process window, and data variable window look like.















Integrated graphical Message Queue display

TotalView's Message Queue Graph is accessed via the process window command Tools -> Message Queue Graph. In the Message Queue Graph window, select the desired display options Send, Receive or Unexpected, select the communicator group to display, select the processes by dragging a box around the processes to display and lastly select the Update button.

Double clicking on arcs or processes in the graph displays the corresponding message queue window or process window. Graph boxes can be moved by clicking on the box and dragging it to another place on the graph.

This replaces LLNL's prototype wait state graph implemented in TotalView v3X.7.9, February 1998 and enhanced in TotalView v3.8.1-1, March 1999.










Integrated graphical Call Tree display

TotalView's Call Tree display is accessed via the process window command Tools -> Call Tree. In the Call Tree window, select All, Control Group or Process and select Update to display the respective call tree. The arcs display the number of processes with that stack trace.

This replaces LLNL's prototype stack trace graph based on the Ptools Light Weight Core File Browser (LCB) implemented in TotalView v3X.7.9, February 1998.











User Preferences

A new Preferences feature has been added to TotalView. This is obtained via selecting the File -> Preferences... command from the root or process window. You can set your preferences from the Options, Action Points, Launch Strings, Bulk Launch, Dynamic Libraries, Parallel and Fonts tabs and than save the information if you want via the Save button.














Start-Up Information

The ~/.Xdefaults file was used to specify user TotalView start-up information. For example, you would set the directory search path via putting into the ~/.Xdefaults file totalview*searchPath : directory-1. With this release, TotalView has introduced further directory specific start-up files.

Start-up information for TotalView can now be specified in .tvdrc start-up files. The .tvdrc file is a Tcl script and will be sourced by TotalView when the executable is being debugged with TotalView.

When TotalView first starts up, it reads the file containing the Help text, and it executes scripts in the following sequence.

Only after all of these scripts have been executed is any image named on the command line loaded into TotalView.

To set a search path using the .tvdrc start-up file, put into the .tvdrc file:

For more information on the dset command, see dset.












Recursive directory search for files

There exists a Tcl script, rpath, the user can invoke to have TotalView search directories recursively for source files from a given directory. The script sets the TotalView variable EXECUTABLE_PATH to a colon delimited list of files for TotalView to search. To use the script, the user must have a CLI window open, which is done via executing the command Tools -> Command Line. The syntax of the rpath script is:

  • If root is not specified, start at the current directory.
  • filter is a regular expression that removes unwanted entries. If it is not specified, the macro automatically filters out CVS/RCS/SCCS directories.

    Example: if you have a directory structure such as:

       Top-Directory
    	Directory-1
    	    Source-directory-1
    		file
    		file
    	    Source-directory-2
    		file
    	Directory-2
    	    Source-directory-3
    		file
    		file
    	    Source-directory-4
    		file
    	    Directory-3
    	        Source-directory-5
    		    file
    		    file
    	        Source-directory-6
    		    file
    	    Source-directory-7
    		file
    		file
    
    Executing rpath Top-Directory, TotalView will search all directories below Top-Directory for source files.

    In the example below, we set EXECUTABLE_PATH to all directories excluding a specific ones via executing rpath /g/g0/zwak/blue PSECODES|zosel|man.














    Faster TotalView performance of execution and display commands on large parallel codes

    TotalView has enhanced performance debugging execution and display commands on large parallel codes. The overall goal for enhanced performance is tough to measure, but 10's of minutes seems too long, but minutes for large number of processes, like 6000, is tolerable. The continuous goal is to make the debugger be responsive to the user.










    Support of IBM Fortran/C OpenMP compiled code

    TotalView now supports IBM Fortran/C OpenMP compiled code as well as previously supported KCC and guide* OpenMP compiled code on IBM. Debugging code compiled with IBM compilers no longer has incorrect display of parameters and local variables.

  • Compile with -qsmp=noopt, NOT -qsmp=noauto, -qsmp=omp or -qsmp. The -qsmp=noopt option is only available using newxlf90/newxlc.

    All of the -qsmp options allow compilation of OpenMP directives, but noopt tells the compiler to not do certain optimizations and leave debugging information around for the TotalView debugger.










    Better thread names in thread pane

    TotalView first implemented better function specification in threads pane of process window in v4.0.0-1, March 2000.

    Instead of the threads pane (lower left pane in process window) saying thread is in kernel, the top item on the stack, the threads pane now displays a stack routine more meaningful to the user, hopefully closest on the stack to the one the user called and specifying via routine-name.










    Asynchronous execution control of threads

    TotalView now includes thread execution controls that include aysnchronous thread control, thread specific breakpoints, simple thread subsetting and thread barrier points.










    Display and handle C++ long identifiers

    TotalView now allows the full display of long type and function names in programs, as seen when using the C++ Standard Template Library (STL). TotalView allows function and type identifiers to be thousands of characters in length and allows a way for the user to display such identifiers.

    In the example below I have created a long function name. In the stack frame you can see the function name with a ... preceding a portion of the name. Using the left mouse button, clicking on the ... will open a dialog box with the entire function name.










    Support attach to subset of processes

    TotalView now supports attaching to a subset of processes. You can attach to a subset of processes at launch time or after the job is executing.

    This replaces LLNL's prototype attaching to a subset of executing processes implemented in TotalView 4.1.0.0, July 2000.

    Steps to attach to a subset of processes at start-up

    Steps to attach to a subset of already executing processes












    Support executing non-interactive CLI script

    TotalView now supports executing a non-interactive CLI script. TotalView at times asks for a response from the user, e.g., starting up a parallel job TotalView asks "Do you want to stop the job before starting?". If you have the TotalView CLI executing a script, than you need a way to answer the question within the script. The way this is done is thru the TV::respond command.

    	TV::respond {response} {command}
    
    The command can be any legal sequence of CLI commands. If the CLI asks any questions while command is running, the TotalView CLI will not prompt the user for the answer, instead totalview will use the string response to answer the question. If more than one question is asked, and response is used up, TV::respond just starts over at the beginning of response again. If response doesn't end with a newline, TV::respond appends one. The most common values for response are y and n.

    Caveats:











    Scripts to support C++ STL vector for IBM xlC, Compaq cxx, g++ and KCC compilers

    Displaying an STL class requires compiler specific information be known by TotalView to properly display data of these classes. TotalView now knows how to display the STL vector class for xlC, g++ and KCC on the IBM and cxx, g++ and KCC on the alphas. Work is currently in progress for TotalView to display the STL list and map classes.

    Below is an example of viewing an STL vector of ints before this version of TotalView and what it displays as now.










    Global Display Format

    Setting the Display Precision Globally was first made available in TotalView v3.9.0-1, May 1999.

    In the Motif user interface, setting the display precision globally is performed via the root or process window command Tools -> Define Global Variable Display Type.

    TotalView's default global precision display type has been scientific notation since TotalView 3.9.0-0 installed July 1999.

    The default global precision display type can be set or changed in three ways:

    Below shows the data variable before and after the DISPLAY_FORMAT variable is changed from it's default to float,3.
















    Root Window Condense

    Condensing the root window for processes in the same state was first made available in TotalView v4.0.0-1, March 2000. The root window is automatically condensed if there exists more than 16 processes.

    The user can control when TotalView condenses the window by setting the ROOT_WINDOW_BOUNDS variable via the CLI command:

    where n is the number of processes necessary before TotalView will do root window condensing.















    Save to History Window

    The Data History Window, formerly called the Variable History Window, was first made available in TotalView v3.7.9, February 1998.

    In TotalView, variable windows are updated every time the program stops. The previous value(s) of the variable is no longer available to the user unless the user wrote the value(s) down in order to keep track of them. The Data History Window is a mechanism to help the user keep track of changes in variables of interest and to compare how the variables change during a run. Normal variable windows continue to operate as before, updating values. The values in the Data History Window are not updated at any time during debugging. When the user is interested in some value, the variable window command Tools -> Save to History Window will copy the value to the history window. Each value in the history window is identified by the variable name (and context). The history window can be "cleared" simply by closing it.

    If a history window does not exist, one will be created for that process. There is only one active history window per process.

    Saving an array or nested structure will save only the information (values) visible in the variable window to the history window (i.e., if you have not scrolled to the bottom of a large array, only the portion of the array that has been scrolled will be saved to the history window.)

    In the example below we have executed the command Tools -> Save to History Window in three variable windows to show what the contents of the Data History Window looks like.











    What's New Information

    The What's New Window was first made available in TotalView v3.9.0-0, July 1999.

    In the Motif user interface, What's New information is accessed via the window menu Help -> What's New?.

    An alternative is to use the CLI command dwhatsnew.













    Memory Utilization Window

    Displaying Memory Utilization was first made available in TotalView v3.8.0, October 1998.

    In the Motif user interface, the Memory Utilization Window is accessed via the root or process window command Tools -> Memory Utilization.

    An alternative is to use the CLI command dmemsize.













    Condense/Uncondense

    Condensing the variable window for array values with the same value was first made available in TotalView v4.0.0-1, March 2000.

    In the Motif user interface, the commands condense and uncondense are accessed via the variable window command View -> Condense.

    In the example below, the right window shows the data displayed after executing the command View -> Condense -> Condense.













    Format - octal, hex, decimal, dump, scientific

    The Format command was first made available in TotalView v3.8.0, October 1998.

    In the Motif user interface, the Format command is accessed via the variable window command View -> Format.

    The example below shows the data expressed in the default scientific notation format in the first window and after executing the command View -> Format -> Floating Point in the second window.













    Save/Compare

    The Save/Compare Values command was first made available in TotalView v3.8.1-1, March 1999.

    In the Motif user interface, the commands Save and Compare are accessed via the variable window command Tools -> Save/Compare.

    In the example below, we have executed the command Tools -> Save/Compare -> Compare with the data in a previously saved file IT and see a difference in the 6th index..













    Vector index

    Displaying an array with a vector subscript was first made available in TotalView v3.8.0, October 1998.

    In the Motif user this feature is accessible in the two ways:















    Display structure array member

    Displaying all of one member of a structured array was first made available in TotalView v3.9.0-0, July 1999.

    In the Motif user interface, TotalView allows the user to display all of one member in a structured array, a[:].j, via:

    For example, in the latter case, when the structure is displayed, the user can edit the slice field to include the specific structured member they want to see. That is you can modify the slice field [:] to be [:].member and the only members of the structure to be displayed will be member. You can also modify the slice field [:] to be [3:5].member and see only the members of the structure array elements [3], [4] and [5].














    LLNL Disclaimers