TotalView Quick Reference Page

February 23, 2004

Privacy & Legal Notice


TotalView User Documents: http://www.totalviewtech.com/Documentation/index.php
TotalView Tip of the Week: http://www.totalviewtech.com/Documentation/Tips/index.html
TotalView CLI Command Summary: http://www.totalviewtech.com/Documentation/latest/pdf/CLI_Command_Summary.pdf

Select the totalview help menu (in the upper right corner of the source and root windows) for additional information.

Contents
Using the Mouse Buttons
IBM OpenMP Compilation Flag: qsmp=noopt
Other Compilation Flags
Setting Search Paths for Source Display
Running with TotalView
Attaching to a Running Code
Shortcut Keys
Setting Breakpoints
Various Commands, Menu Selections
Available Versions
X windows access authorization to remote machines—See /usr/local/docs/X11_tricks on blue.pacific.llnl.gov
Caveat: -bmaxdata=0X80000000
Using Input Files
Customizing your windows—See /usr/global/tools/tv/vsns/totalview.5.0.0-4/contrib/how_to_customize_windows
Efficient TotalView Restarts
TotalView and Python Scripts—See /usr/global/src/tv/docs/llnl/pdb.html
Evaluating C++ Methods in Expression Window
Process/Thread State Information Codes
Using TotalView with nopoe
Examining Multiple Corefiles on IA32 Linux
Timeout Problems
 

Using the Mouse Buttons

Top


Search Paths

Select File>Search Path.
Enter full paths, one per line.
Click on OK.

Top


Running with TotalView

Debugging a Scalar Program

totalview myprog [-a args_to_myprog]

Debugging a Crashed Scalar Program

totalview myprog core

Debugging a Parallel Program: https://computing.llnl.gov/mpi/libraries.html

For IBM

totalview poe -a myprog [args_to_poe] [args_to_myprog]
Example: totalview poe -a a.out -procs 8 -nodes 2 -rmpool 0

For Alpha Clusters

Using dmpirun

totalview dmpirun -a [args_to_dmpirun] myprog [args_to_myprog]
Example: totalview dmpirun -a -np 4 a.out

Using mpirun

Set the environment variable TOTALVIEW to the version of TotalView that you want.

mpirun -tv [args_to_mpirun] myprog [args_to_myprog]
Example: setenv TOTALVIEW newtv mpirun -tv -np 4 a.out

For Platforms Using SLURM

totalview srun -a -p [partition name] -n [processes] myprog [args_to_myprog]
Example: totalview srun -a -n 4 -p pdebug a.out

See Section 9 of /usr/local/docs/linux.basics for more information on using TotalView with srun.

Type g in the process window where you see the poe/dmpirun code displayed. Answer Yes to the question. Set your breakpoints.

Using TotalView in a Batch Job

Use a psub script similar to the following:

 #########################################################
 #!/bin/csh       # Sets  your shell
 #PSUB -ln 8      # Number of processors you want
 #PSUB -tM 6000s  # Time limit in seconds (default is 30 min)
 cd "directory of executable - a.out"
 totalview  a.out -a arglist    # scalar job or totalview poe -a a.out
 poe_options arglist            #  parallel job
 ######################################################### 

or use the psub script to launch an X-term window and then use TotalView in that window

#########################################################
 #!/bin/csh       # Sets your shell
 #PSUB -eo        # Direct stderr to stdout
 #PSUB -ln 8      # Number of nodes
 #PSUB  -g 16     # Total number of processes
 #PSUB -c frost   # Run job only on frost
 #PSUB -tM 6000s  # Time limit in seconds (default is 30 min)
 #PSUB -x         # Export environment variables
 #setenv DISPLAY euclides:0.0
 cd "directory of executable"
 xterm -title  "xterm window" -ls -sb -sl 10000

 # now you can run totalview in this window:
 #  totalview poe xcode ...
 #########################################################

Note that if you run multiple poes within an Xterm window, you may possibly leave orphaned MPI processes lying around. Use poe /usr/local/bin/killmyorphs to clean them up between poe runs.

Using TotalView in a Batch Job on TC2K

TotalView can be used on the tckk only on the login nodes; therefore, use a psub script similar to the following:

 #########################################################
 #!/bin/csh       # Sets your shell
 #PSUB -tM 6000s  # Time limit in seconds (default is 30 min)
 setenv DISPLAY terminal_id:0.0
 cd "directory of executable"
 xterm
 #########################################################

Submit your psub script:

prun -n no_nodes script_name

The Xterm will run on a login node. When the Xterm window comes up, type in that window:

totalview prun -a ./a.out arglist

Top


Attaching to a Running Code

Top


Shortcut Keys

Lowercase letter applies to single process; uppercase letter applies to all processes. TotalView also has menu selections for execution control when you select Group, Process, or Thread. See TotalView/TV5 Comparison Guide at http://www.totalviewtech.com/Documentation/rel5/html/converter_guide/index.html for a complete table.

Execution Control
g (G) go (continue/run)
h (H) halt
s (S) step to next line, entering functions
n (N) run to next line, stepping over functions
o (O) return out of the current function
q (in root window), exit TotalView
closes other windows
Q closes all windows relative to process window
Display Data/Source
f enter the name of a function or file in the dialog box
v enter the name of a variable including optional slice or indices
V display the global variables window
M display the Fortran modules window

Top


TotalView Breakpoint Methods

Function Method
Set/Disable Breakpoints Select line number or
Select line, Action Point>Set Breakpoint
Use Action Points menu, select bkp to enable/disable it
Dive on bkp to display source
Set/Disable Barrier points Shift-Select line number or
Select line, Action Point>Set Barrier
Conditional Breakpoints Select line
Select Action Point>Properties
Select Evaluate,
Type in conditional code
Select OK
Breakpoint Control Select line
Action Point>Properties
Action Point>Delete All
Action Point>Save All
Action Point>Load All
Watchpoints
(Not available on Blue)
In Variable Window,
Select Tools>Watchpoint,
Select Unconditional or Conditional
Type in conditional code
Select OK

Top


Commands and Equivalent Menu Selections

Command Equivalent Menu Selection
Variable Window Options
Visualize Tools>Visualize
Laminated display
(variable value by process)
View>Laminate>Process
Thread laminated display View>Laminate>Thread
Data watchpoint Tools>Watchpoint
Sort View>Sort>Ascending
View>Sort>Descending
Display variables in a given module Tools>Program Browser
Display Fortran modules and variables Tools>Fortran Modules
Save/Compare Tools>Save/Compare
Process Information
Message state window Tools>Message Queue
Global Wait State graph Tools>Message Queue Graph
Stack Trace graph Tools>Call Tree
Memory utilization Tools>Memory
Miscellaneous
Expression window Crtl+E Tools>Evaluate
Clear input text quadruple click left mouse button
hit delete/back space
Search for string
forward, backward, repeat Ctrl+F
Edit>Find
Help Select Help

Top


Available Versions of TotalView

Various versions of TotalView are available by using the symbolic links: totalview, newtv, oldtv, betatv.

For the available links and their release numbers listed by platforms, see
/usr/global/tools/tv/toolworks/docs-LLNL/html/Links

Release numbers without "-LLNL" appended are the Etnus versions (they have no LLNL enhancements). To see all of the available versions, type:
ls /usr/local/tv/vsns/

The general form for a path to an alternate version is:
/usr/local/tv/vsns/totalview.[release_number]/bin

To use a specific version, prepend the above path to your PATH and type:
totalview

or

use the full path name; .e.g., /usr/local/tv/vsns/totalview.5.0.0-4/bin/totalview

Top


Caveat: -bmaxdata=0X80000000

If you use -bmaxdata=0X80000000 for an MPI code and set MP_SHARED_MEMORY to yes, you may get the following errors when you try to use TotalView:

ERROR: 0031-309  Connect failed during message passing initialization, task 11, 
       reason: Error on shmget or shmat 
0:ERROR: 0031-007 Error initializing communication subsystem: return code -1

You can solve this by using a flag with TotalView:
totalview -compileExpressions=false poe -a code ...

This flag disables TotalView's dynamic patch space allocation for which it may not have room when you need eight segments. (Or you can reduce the number of segments that you are using.)

Top


Using Input Files

In the Process window before saying 'g'o to your executable:

Top


Efficient Restarts

Restarts are more efficient (faster) when they are done from a batch Xterm window. If they are done from the debug pool:

Top


Evaluating Methods in the Expression Window

Until release 7.0, users can:

I. Provide C wrappers for the C++ methods that they want to evaluate in the expression window. Example:

#include 
class num{
public:
   int a;
   int sum(int i);
};
int num::sum(int i)
{
   return a+i;
}
// C wrapper to use in TotalView expression window:
extern "C" void dom(num* x, int i){x->sum(i);};

int main()
{
   num n;
   int s;
   n.a = 17;
   cout << "s: " << n.sum(2) << "\n";
   s = n.sum(3);
   cout << "s: " << s << "\n";
}

Use "dom(&n, 3)" to evaluate n.sum(14) or "method (this)" if inside the class.

or

II. Use Pat Miller's "weak-remangler" tool:
/usr/global/tools/tv/toolworks/contrib/rs6000/bin/weak-remangler
that generates a remap of an object file containing weak remanglings for the g++ and KAI compilers on the IBM platforms. For documentation, see
/usr/global/tools/tv/toolworks/contrib/doc/debugging_c++_methods1.pdf.

Top


State Information Displayed in the Process/Thread Pane

State Code State
R Running
T Stopped
S Sleeping, <20 sec
Z Zombie, child process terminated
M Mixed; some threads running/not
E Error
B At bkp
W At watchpt
K Thread executing inside kernel
I Idle; sleeping for >20 sec

Top


Using TotalView with nopoe (IBM Platforms)

If you want to use totalview and take advantage of the nopoe script that allows you to run an mpi executable with [args] in an environment where the executable will be run on the local host with one task, use /usr/local/scripts/tvnopoe.

Usage: tvnopoe executable [args]

Top

Examining Multiple Corefile on IA32 Linux

If you want to use totalview to examine multiple corefiles from a parallel run on IA32 Linux platforms, use the script, /usr/local/tv/vsns/contrib/linux-x86/bin/tv-ldcores, in the directory containing the core files.

Usage: tv-ldcores executable

Top

Timeout Problems

There are several situations when TotalView pops up an error box:

"Timed out. Waitiing for a process to settle down. Keep waiting? (Yes)(No)".
One should select "Yes" several times to see if things eventually settle down.

A common timeout condition on AIX is as follows:

User sets a breakpoint at which only a subset of MPI tasks stop. The other tasks are waiting on the stopped tasks at MPI calls somewhere further in the code. Any group operations such as "Group>Next" or "Group>Step" may cause TotalView to time out.

There are a couple of workarounds for this problem:

(1)
(a) Stop all the processes at a barrier point nearest to the area of code in which you are interested.
(b) Use Process>Go to move the process of interest forward to the planted breakpoint.
(2)
(a) Change the breakpoint property (see below) such that it stops the entire group. Then when one task hits it, the rest of the tasks are also stopped.
(b) Change the control width to "Process". Then debugging actions such as "Next" and "Step" will affect only the task in which you are interested.

Also keep in mind, that poe should always be kept running. If you notice in the root window that the poe process is not in the run, R, state, dive on it and when its window comes up, use Process>Go. The you can return to the window of your process of interest.

To review:

Breakpoint properties:

A breakpoint can stop:

By default, a breakpoint stops a process. You can use Tools>Action Points>Properties to change this; for example, to stop a thread.

Control Width:

Using Group>Next advances all of the members of the given group. On the main menu bar, switch to "Share" group to advance all the members that share the same executable. Or use Process>Next to advance a single process or Thread>Next to advance a thread.

Top


UCRL-WEB-20081