Privacy and Legal Notice

LLNL Machine Characteristics

The road to parallel application performance enlightenment is fraught with dangerous curves and slippery slopes. It is helpful to understand the machine characteristics when working through application performance, and this page identifies typical elements. Further, it provides guidance numbers for these elements when possible.

The key hardware components to application performance (detailed in the following tables) are:

Important note about the guidance aspect of this information: Except where otherwise noted, the guidance numbers presented in the tables below were obtained on similar machines; they are not measurements obtained on the actual LLNL machine. For example, the Spec numbers reported for UP are from the Spec Web site for an IBM pSeries 570 model node (eight-way 1900 Mhz Power5 nodes) rather than actually loading the Spec benchmark suite onto UP and performing measurement runs. As such, these numbers are not expected to be exact; the number reported will probably have been obtained in a slightly different environment (e.g., different compiler, different motherboard rev, etc.).

Overall Machine
Network System Node
Type
Chipset Interconnect Batch
Nodes
Batch
CPUs
Batch Memory
(GB)
Batch Peak
(GFLOPS)
OCF UP IBM pSeries 570 Power5 Federation 99 792
(99*8)
3168
(99*32)
6019
(99*60.8)
Zeus Appro 1143H AMD Opteron 8216 DDR IB 288 2304
(288*8)
2304
(288*8)
11,059
(288*38.4)
SCF BlueGene/L IBM BlueGene P440 65,536 131,072
(65,536*2)
32,768
(64K*.512)
366,346
(64K*5.59)
Purple IBM pSeries 570 Power5 Federation 1336 10,688
(1336*8)
42,752
(1336*32)
81,350
(1336*60.8)
Rhea Appro 1143H AMD Opteron 8216 DDR IB 540 4320
(540*8)
4320
(540*8)
20,736
(540*38.4)

 

Processor Performance
Network System Processor
Type
Clock
(GHz)
Est. Spec2000 Int
(per CPU)
Est. Spec2000 FP
(per CPU)
CPUs
(per node)
Est. Spec2000 Int Rate
(per node)
Est. Spec2000 FP Rate
(per node)
OCF UP IBM Power5 570 1.9 1398 2576 8 141 241
Zeus AMD Opteron 2.4 1366 1872 8 127 142
SCF BlueGene/L IBM PowerPC 667 MHz 2
Purple IBM Power5 570 1.9 1398 2576 8 141 241
Rhea AMD Opteron 2.4 1366 1872 8 127 142

Note: Results for new Spec2006 benchmark suite are not directly comparable to Spec2000 results.

 

Memory Performance
Network System Memory
Type
Peak Transfer Rate
(GB/s)
Caches Stream No.
(per node)
Memory per Node (GB)/
No. CPUs per Node
OCF UP DDR226 2.133 L2=1920 KB unified (on chip)/chip
L3=36 MB unified (off chip)/DCM
32,372 MB/s 32/8
Zeus DDR2 667 5.336 L1=64KBI + 64KBD on chip
L2=1024KB
19,963 MB/s 16/8
SCF BlueGene/L   5.50 offchip L1=32k/32k on core
L3=4 MB on chip
  512 MB/2
Purple DDR266 2.133 L2=1920 KB unified (on chip)/chip
L3=36MB unified (offchip)/DCM
32,372 MB/s  
Rhea DDR2 667 5.336 L1=64KBI + 64KBD on chip
L2=1024KB
19,963 MB/s 16/8

 

Interconnect Performance
Network System Interconnect
Link Type
Interconnect Link
Speed per Direction
(MB/s)
No. Links Host Link Type Host Bus-Link Speed
(MB/s)
MPI Bandwidth
(MB/s)
MPI Latency
(µsec)
OCF UP IBM Federation 2000
(each link, each direction)
2 GX bus 18.1 GB/s
(theoretical eight 1.9 Ghz   Cores)
1tpn
5.68 GB/s bi-directional
3.31 GB/s uni-directional

8tpn
5.85 GB/s bi-directional
3.77 GB/s uni-directional
1tpn
5.01 µs

8tpn
6.00 µs
Zeus x4 DDR InfiniBand 1600
(each link, each direction)
(20 Gbit/s full-duplex 10-bit signaling rate)
2 PCIe x8 2 * (4000 MB/s duplex) 979 (sdr) 4
SCF BlueGene/L Custom 64x32x32 torus

Custom global tree
Torus=6*(175 MB/s)

Tree=700 MB/s
6   NA 175 MB/s each direction for six nearest neighbors 3
(2300 cycles)
Purple IBM Federation 2000
(each link, each direction)
2 GX+ bus 18.1 GB/s
(theoretical eight 1.9 Ghz   Cores)
1tpn
5.68 GB/s bi-directional
3.31 GB/s uni-directional

8tpn
5.85 GB/s bi-directional
3.77 GB/s uni-directional
1tpn
5.01 µs

8tpn
6.00 µs
Rhea x4 DDR InfiniBand 1600
(each link, each direction)
(20 Gbit/s full-duplex 10-bit signaling rate)
2 PCIe x8 2 * (4000 MB/s duplex) 979 (sdr) 4

* PCI: 64 bit @ 66 MHz
**PCI-X: 64 bit @ 133 MHz


High Performance Computing at LLNL    Lawrence Livermore National Laboratory

Last modified October 21, 2009
UCRL-WEB-218462