ICC Home Privacy and Legal Notice LC User Documents Banner

UCRL-WEB-200040

CHAOS: Linux from Livermore


Environment Variables For CHAOS

Just as AIX supports many environment variables unique to IBM's "Parallel Operating Environment" (POE), intended to help manage parallel jobs only on IBM machines, so too does CHAOS provide unique environment variables to manage parallel jobs on the LC Linux clusters where it is the operating system. Currently the CHAOS-only environment variables related to the Elan library (a Quadrics library of low-level message-passing functions) are in use on LC production clusters (such as Thunder, Lilac, and ALC).

Default values for CHAOS environment variables are noted if they exist, and for most jobs the default value is the optimal value. Additional Elan library environment variables, mostly related to controlling Quadrics switch support for MPI optimization or debugging, are explained at this dedicated web site (open-network only):


http://www.llnl.gov/computing/mpi/elan.html


The CHAOS Simple Linux Utility for Resource Management (SLURM) uses its own additional environment variables (all with names that begin with "SLURM_") to store resource-allocation values for the jobs that SLURM manages. See the "Environment Variables" section of the SLURM Reference Manual for an explanatory list. For a big-picture look at how the CHAOS environment variables here and the SLURM environment variables both fit into the larger, complex set with which LC manages its systems and jobs, consult LC's Environment Variables user guide.

LIBELAN_GALLOC_EBASE
(default value: 0xb0000000) resizes the Elan global memory heap for MPI collective operations. EBASE is a pointer to a base virtual address in Elan memory to be used for the global heap. (Set this variable and the next two if you use MPI collectives, such as REDUCE, GATHER, SCATTER, or their ALL versions, with more than about 100 processes.)
LIBELAN_GALLOC_MBASE
(default value: 0xb0000000) resizes the Elan global memory heap for MPI collective operations. MBASE is a pointer to the main memory base in Elan memory to be used for the global heap.
LIBELAN_GALLOC_SIZE
(default value: 16777216) resizes the Elan global memory heap for MPI collective operations. SIZE is the size in bytes of the Elan global heap.
LIBELAN_WAITTYPE
(suggested value: POLL) specifies how a blocking MPI process will share computing resources (comparable to MP_WAIT_MODE under POE on IBM machines). Possible values are:
POLL
(default) has the receiving thread actively poll for incoming messages. Use this choice for all MPI jobs on clusters that have a Quadrics interconnect.
SLEEP
has the receiving thread sleep and thus remove itself from the active dispatching queue.
YIELD
has the receiving thread stay in the queue but yield the processor if it has no work to do.
MALLOC_TRIM_THRESHOLD
(suggested value: -1) see the next item for joint use.
MALLOC_MMAP_MAX
(suggested value: 0) when combined, MALLOC_TRIM_THRESHOLD and MALLOC_MMAP_MAX force MALLOC to use SBRK() rather than MMAP() to allocate memory. This improves performance, but it may reduce the total amount of memory available to your user processes (to no more than 1 Gbyte/process).
MPI_USE_LIBELAN
toggles the Elan library optimizations. Possible values and their roles are:
1
(default) enables the Elan library optimizations.
0
disables the Elan library optimizations. Use this only for debugging, if you suspect problems with the Elan libraries themselves.
OMP_NUM_THREADS
controls the number of threads spawned by Intel's Math Kernel Library (MKL) routines when you invoke this threaded library on any of LC's Intel Linux machines (such as ILX, MCR, or Thunder). By default, MKL sets the number of threads equal to the number of processors.



Navigation Links: [ Document List ] [ HPC Home ] [ Next ]