Using Thunder

presented by

Blaise Barney
Livermore Computing

NOTE: Thunder was retired in January, 2009. This tutorial is no longer being maintained, but will be saved for archival purposes.

Table of Contents

  1. Abstract
  2. Background of Linux Clusters at LLNL
  3. Hardware Overview
    1. Thunder Configuration
    2. Itanium 2 Processor
    3. E8870 Chipset
    4. Quadrics Interconnect
    5. SAN Architecture
  4. Accounts and Access
  5. Software and Development Environment
  6. Intel Compilers
  7. Quadrics MPI
  8. Running on Thunder
    1. Batch Versus Interactive
    2. Starting and Terminating Jobs
    3. Displaying Queue and Job Status Information
    4. Optimizing CPU Usage
    5. Memory Constraints
    6. Performance Considerations
  9. Debugging
  10. Tools
  11. References and More Information
  12. Exercise


This tutorial is intended to be an introduction to using LC's IA64 Thunder Linux cluster. It begins by providing a brief historical background of Linux clusters at LC, noting their success and adoption as a production, high performance computing platform. The primary hardware components of Thunder are then presented, including a summary of Thunder's overall configuration, Intel's IA64 Itanium 2 processor, the E8870 Chipset and the Quadrics interconnect switch.

After covering the hardware related topics, a brief discussion on how to obtain an account and access Thunder follows. Software topics are then discussed, including the LC development environment, compiling with the Intel compilers, Quadrics MPI and how to run both batch and interactive parallel jobs. Special attention is paid to IA64 issues in each of these areas as relevant. Available debuggers and performance related tools/topics are briefly discussed, however detailed usage is beyond the scope of this presentation. The tutorial concludes with a brief listing of known issues and problems and where to go for more information. A lab exercise using the IA64 Thunder Linux cluster follows the presentation.

Level/Prerequisites: Intended for those who are new to developing parallel programs in LC's Intel IA64 cluster environment. A basic understanding of parallel programming in C or Fortran is assumed. The material covered by EC3501 - Introduction to Livermore Computing Resources would also be useful.

Background of Linux Clusters at LLNL

The Linux Project:

Alpha Linux Clusters:

PCR Clusters:

MCR Cluster...and More:

Which Led To...

  • In September, 2003 the RFP for LC's first IA-64 cluster was released. Proposal from California Digital Corporation, a small local company, was accepted.

  • 1024 node system comprised of 4-CPU Itanium 2 "Madison Tiger4" nodes

  • Thunder debuted as #2 on the supercomputer list in June, 2004.
Thunder Photo

Hardware Overview

Thunder Configuration


Additional Details:

Hardware Overview

Itanium 2 Processor


Itanium 2 Block Diagram:


Hardware Overview

E8870 Chipset

Block Diagram:


Hardware Overview

Quadrics Interconnect

Primary components:




Hardware Overview

SAN Architecture

LC OCF SAN Configuration

Accounts and Access



Software and Development Environment

Note: Like the IA32 Linux clusters, Thunder's software and development environment is very similar to that described in the Introduction to LC Resources tutorial. Only highlights and items specific to Thunder are discussed below.

CHAOS Operating System:

Batch System:

File Systems:


MKL - Intel Math Kernel Library

Debuggers and Performance Analysis Tools:

Man Pages:

Intel Compilers

Optimizing Compilers:

Compiler Invocation Commands:


Common / Useful Options:

GNU Compatibility:


Quadrics MPI

Quadrics MPI:

MPI Build Scripts:

Static Linking:

Libelan Environment Variables:


Running on Thunder

Batch Versus Interactive

A Few General Notes First:

Interactive Jobs:

Batch Jobs:

Quick Summary of Common Batch Commands:

Running on Thunder

Starting and Terminating Jobs

Invoking the Executable:

Terminating Jobs:

Running on Thunder

Displaying Queue and Job Status Information

Running on Thunder

Optimizing CPU Usage

Running on Thunder

Memory Constraints

What Constraints?

Process Stack vs. Heap Memory:

Pthreads Stack Limits:

OpenMP Stack Limits:

In Conclusion:

Running on Thunder

Performance Considerations

Quadrics Environment Variables:

Compiler Hints:

Local MPI Test Results on Thunder:

Web Documentation:


Available Debuggers:

TotalView: Small TotalView screen shot

DDT: Small ddd screen shot


DDD: Small ddd screen shot

IDB: Small idb screen shot

Debugging in Batch: batchxterm:

A Few Additional Useful Debugging Hints:


We Need a Book!

References and More Information

This completes the tutorial.

Evaluation Form       Please complete the online evaluation form - unless you are doing the exercise, in which case please complete it at the end of the exercise.

Where would you like to go now?