Author: Blaise Barney, Lawrence Livermore National Laboratory UCRL-MI-133316

Table of Contents

  1. Abstract
  2. Part 1: The Very Basics
    1. Overview
    2. Starting TotalView
      1. Environment Setup
      2. Compiling Your Program
      3. Starting TotalView
    3. TotalView's Basic Look and Feel
      1. Primary Windows
      2. Dialog Boxes
      3. Mouse Usage
      4. Menus
      5. Accelerator Keys
      6. Scrolling, Resizing and Memorizing
      7. Process and Thread State Codes
      8. Session Manager
    4. TotalView's Basic Functions
      1. Viewing Source Code
      2. Setting a Breakpoint
      3. Controlling Execution
      4. Diving
      5. Viewing and Modifying Data
      6. Text Editing and Searching
      7. Saving Window Contents
      8. Getting Help
      9. Exiting TotalView
    5. Exercise 1

  3. Part 2: Common Functions
    1. Loading Executables
    2. Attaching / Detaching Processes
    3. Viewing a Core File
    4. Expression Evaluation and Code Fragments
    5. More on Action Points
    6. Setting Executable Command Arguments
    7. Setting Source Code Search Paths
    8. Setting stdin, stdout, and stderr
    9. Signal Handling
    10. Displaying Your Program's Call Tree
    11. Debugging Memory Problems
    12. Setting Preferences
    13. Visualizing Array Data
    14. Command Line Interpreter (CLI)
    15. Exercise 2

  4. Part 3: Debugging Parallel Programs
    1. Process/Thread Groups
    2. Debugging Threaded Codes
      1. Overview
      2. Finding Thread Information
      3. Selecting a Thread
      4. Execution Control for Threaded Programs
      5. Viewing and Modifying Thread Data
    3. Debugging OpenMP Codes
      1. Overview
      2. Debugging OpenMP Programs
    4. Debugging MPI Codes
      1. Overview
      2. Starting an MPI Debug Session
      3. Selecting an MPI Process
      4. Controlling MPI Process Execution
      5. Viewing and Modifying Multi-process Data
      6. Displaying Message Queue State
    5. Debugging Hybrid Codes
      1. Overview
      2. Debugging Hybrid Programs
    6. Batch System Debugging
    7. Topics Not Covered
    8. References and More Information
    9. Exercise 3


TotalView is a sophisticated and powerful tool used for debugging and analyzing both serial and parallel programs. TotalView provides source level debugging for serial, parallel, multi-process, multi-threaded, accelerator/GPU and hybrid applications written in C/C++ and Fortran. Most HPC platforms and systems are supported. Both a graphical user interface and command line interface are provided. Advanced, dynamic memory debugging tools and the ability to perform "replay" debugging are two additional features. TotalView has been selected as the DOE ASC Program's debugger of choice for its HPC platforms.

This tutorial has three parts, each of which includes a lab exercise. Part 1 begins with an overview of TotalView and then provides detailed instructions on how to set up and use its basic functions. Part 2 continues by introducing a number of new functions and also providing a more in-depth look at some of the basic functions. Part 3 covers parallel debugging, including threads, MPI, OpenMP and hybrid programs. Part 3 concludes with a discussion on debugging in batch mode.

Level/Prerequisites: This tutorial is intended for those who are new to TotalView. A basic understanding of parallel programming in C or Fortran is required. The material covered in the following tutorials would also be beneficial for those who are unfamiliar with parallel programming in MPI, OpenMP and/or POSIX threads:
EC3505: MPI
EC3506: POSIX Threads
EC3507: OpenMP

TotalView Part 1:
The Very Basics


What is TotalView?

  • TotalView is a sophisticated software debugger product from Rogue Wave Software, Inc.
    ...and before that, TotalView Technologies, LLC (2007-2009)
    ...and before that, Etnus LLC. (1998-2007)
    ...and before that, Dolphin Interconnect Solutions, Inc. (1996-1998)
    ...and before that, BBN Systems and Technologies, a division of BBN Corporation (1993-1996)   

Key Features of TotalView:

Supported Platforms and Languages

TotalView at LLNL:

Starting TotalView

Environment Setup

Compiling Your Program


Beyond -g:

Starting TotalView

Several Ways:

Some Examples:

TotalView's Basic Look and Feel

Primary Windows

Root Window:

Process Window:

Variable Window:

TotalView's Basic Look and Feel

Dialog Boxes

TotalView's Basic Look and Feel

Mouse Usage

TotalView's Basic Look and Feel


Two Types of Menus:

TotalView's Basic Look and Feel

Accelerator Keys

Short Cut:


TotalView's Basic Look and Feel

Scrolling, Resizing and Memorizing

Conventional Scrolling Behavior:

Resizing Windows and Panes:

Memorizing Windows:

  • The "Window" menu (if present) will allow you to save the position and size of that window, or all windows.

  • A convenience feature for those who like to have their TotalView sessions customized.

  • Resized panes inside a window are not memorized.

TotalView's Basic Look and Feel

Process and Thread State Codes

TotalView's Basic Look and Feel

Session Manager

TotalView's Basic Functions

Viewing Source Code

Source, Assembler or Both:

Displaying Function / File Source Code:

TotalView's Basic Functions

Setting a Breakpoint

What Is a Breakpoint?

Several Ways to Set / Unset a Breakpoint:

  • Method 1: The easiest way to set a breakpoint is to simply click on a source code line number with the left mouse button. A red STOP icon will then appear on the source line number, as shown at right.

  • Method 2: Right mouse click anywhere on the desired source line until the pop-up menu appears (right). Then select Set Breakpoint.

  • Method 3: First, click on a source line to select it (make sure it's highlighted). Then use:

    Process Window  >  Action Point Menu  >  Set Breakpoint 

  • Method 4: For any arbitrary line number, use the path below. A dialog box will then open to prompt you for the line number.

    Process Window  >  Action Point Menu  >  At Location 

  • To unset the breakpoint, simply click on the red STOP icon or select "delete" from the pop-up menu or Action Point menu.

Viewing Breakpoints:

Breakpoint Options:

TotalView's Basic Functions

Controlling Execution

Execution Control Commands:

Group, Process, Thread Command Scopes:

TotalView's Basic Functions



Nested Dives and Undiving:

TotalView's Basic Functions

Viewing and Modifying Data

Viewing Data:


Modifying Variable Data:


Changing Variable Display Format:

Changing Variable Data Types:

TotalView's Basic Functions

Text Editing and Searching

Text Editing:
  • TotalView provides a basic field editor for use within certain debugger fields and windows. Text which can be edited will be highlighted and display a field editor cursor.

  • Cutting and pasting can be accomplished by using the middle mouse button or by selecting Cut, Copy, or Paste from any window's Edit pull-down menu.

Text Searching:

  • Most TotalView windows will permit you to search for text strings. Simply select Find from any window's Edit pull-down menu.

  • A dialog box will appear for you to enter the string to search for, plus any search options, as shown below.

  • Select Find Again from the same Edit menu to repeat a search.

TotalView's Basic Functions

Saving Window Contents

  • Most TotalView windows enable you to save their contents as ASCII text. You can also pipe the contents to UNIX shell commands.

  • For windows with multiple panes, you have to save each pane individually.

  • Make sure your mouse pointer is in the window or pane of interest. Then select Save Pane from any window's File pull-down menu. A dialog box will then appear for your input, as shown below:

       Original Stack Frame pane   Saved contents of pane

  • Using the "Send To Pipe" option - allows you to direct the pane/window contents to a UNIX shell command. The command is entered in the File Name box. For example:

    Unless specified otherwise, output from the command appears as stdout in the window where you started TotalView.

TotalView's Basic Functions

Getting Help

  • TotalView provides an extensive, web browser based online Help system.

  • All primary TotalView windows have a Help pull-down menu that includes access to the vendor's complete set of product documentation.

  • Context sensitive help is available by left-clicking on an object and then selecting "Help" from the Help pull-down menu, or hitting the F1 key.

  • Additionally, many dialog boxes have a context-sensitive Help button.

  • The Help pull-down menu and Help Documentation are shown below.

TotalView's Basic Functions

Exiting TotalView

  • You can exit the debugger in several ways:
    • From any window select File Menu  >  Exit
    • Typing CTRL-q or CTRL-Q in any window
    • Closing the Root Window via your window manager

  • After selecting any of these ways to exit TotalView, you will be prompted to confirm your choice to exit:

This concludes TotalView Part 1

Where would you like to go now?