Running Dynamically Linked Programs on BG/Q

In addition to the discussion below, see Chapter 7 of the Blue Gene/Q Application Development Redbook.

A program that is created with a compiler that targets the Blue Gene/Q system identifies the path to the dynamic linker as /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux/lib/lib64/ld64.so.1. This directory is available on the front end nodes and the I/O node. The readelf tool displays this information.

Running a dynamically linked program on the Blue Gene/Q system is similar to the way a statically linked program is run. When dynamically linked programs are run on the Blue Gene/Q system, the paths to those libraries must be known to the Blue Gene/Q dynamic linker, or the program fails to run. By default, the dynamic linker searches the directories that are expected to contain shared libraries for use with Blue Gene/Q system. These locations include the directories for the Blue Gene/Q toolchain shared libraries and the Python shared library. The linker also searches for MPI or PAMI shared libraries for the Blue Gene/Q driver. The driver is in the /usr/lib64/bgq/ directory on the I/O node.

The Blue Gene/Q dynamic linker follows the same search conventions as the native GNU dynamic linker. At program load time, the dynamic linker attempts to load all the dependent libraries in the program that are identified as NEEDED in the dynamic section of the program (ELF) file, as displayed by the readelf tool. The search path order for a dynamically linked program on the Blue Gene/Q compute node contains the following locations:

If your program fails at runtime with an error such as
libxlsmp.so.1: cannot open shared object file: No such file or directory
this may be due to that library not being available in the I/O node image. Please contact the LC Hotline to open an issue on that. Until the I/O node image can be fixed, the workaround is to create your own subdirectory to hold the needed shared libraries, copy them there from their default location (look for them in DTRPATH or DTRUNPATH), then set LD_LIBRARY_PATH to include your subdirectory.

The NEEDED, DT_RPATH, and DT_RUNPATH information for a program can be viewed using the readelf command:

% /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-readelf -l -d a.out

Elf file type is EXEC (Executable file)
Entry point 0x1100288
There are 7 program headers, starting at offset 64
Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000001000040 0x0000000001000040
                 0x0000000000000188 0x0000000000000188  R E    8
  INTERP         0x00000000000001c8 0x00000000010001c8 0x00000000010001c8
                 0x0000000000000044 0x0000000000000044  R      1
      [Requesting program interpreter: /bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/lib/ld64.so.1]
  LOAD           0x0000000000000000 0x0000000001000000 0x0000000001000000
                 0x000000000000098c 0x000000000000098c  R E    10000
  LOAD           0x0000000000010000 0x0000000001100000 0x0000000001100000
                 0x00000000000003f8 0x0000000000000480  RW     10000
  DYNAMIC        0x00000000000100b8 0x00000000011000b8 0x00000000011000b8
                 0x00000000000001c0 0x00000000000001c0  RW     8
  NOTE           0x000000000000020c 0x000000000100020c 0x000000000100020c
                 0x0000000000000040 0x0000000000000040  R      4
  GNU_EH_FRAME   0x0000000000000968 0x0000000001000968 0x0000000001000968
                 0x0000000000000024 0x0000000000000024  R      4
 Section to Segment mapping:
  Segment Sections...
   00     
   01     .interp 
   02     .interp .note.ABI-tag .note.BGQ-tag .hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.plt .init .text .fini .rodata .eh_frame_hdr 
   03     .eh_frame .ctors .dtors .jcr .dynamic .data .opd .branch_lt .got .plt .bss 
   04     .dynamic 
   05     .note.ABI-tag .note.BGQ-tag 
   06     .eh_frame_hdr 
Dynamic section at offset 0x100b8 contains 23 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x000000000000000f (RPATH)              Library rpath: [/opt/ibmcmp/lib64/bg/bglib64]
 0x000000000000000c (INIT)               0x11002d0
 0x000000000000000d (FINI)               0x11002e8
 0x0000000000000004 (HASH)               0x1000250
 0x0000000000000005 (STRTAB)             0x1000310
 0x0000000000000006 (SYMTAB)             0x1000280
 0x000000000000000a (STRSZ)              198 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x11003f8
 0x0000000000000002 (PLTRELSZ)           96 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x1000418
 0x0000000070000000 (PPC64_GLINK)        0x10008a0
 0x000000006ffffffe (VERNEED)            0x10003e8
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x10003d6
 0x0000000000000000 (NULL)               0x0