Large Linux data centers require flexible system management. At Livermore Computing (LC), we are committed to supporting our Linux ecosystem at the high end of commodity computing and on our most advanced machines. Administrators of Linux clusters will find an array of robust tools developed at LLNL for platform management, authentication, and I/O analysis. LLNL's open source tools are available on our GitHub-based software portal. Select open-source tools are described below.

ConMan

ConMan provides centralized console logging and access for large clusters, making it easier for administrators to diagnose and recover systems. For example, administrators can troubleshoot operating-system failures when normal network access may be down. A number of console access mechanisms are supported, including local serial devices, remote terminal servers, and IPMI Serial-over-LAN.

FreeIPMI

The health of system hardware—including temperature, voltage, fans, power supply, bus errors, and system physical security—requires constant monitoring of recovery capabilities, issue logging, and inventory information. Intelligent Platform Management Interface (IPMI) software facilitates this platform management by operating independently and remotely. FreeIPMI provides in-band and out-of-band IPMI software based on the IPMI v1.5/2.0 specification, which defines a set of interfaces for platform management and is implemented by a number of vendors.

MUNGE

MUNGE is an authentication service for creating and validating credentials. It is designed to be highly scalable for use in a high performance computing cluster environment. It allows a process to authenticate the UID and GID of another local or remote process within a group of hosts having common users and groups. These hosts form a security realm that is defined by a shared cryptographic key. Clients within this security realm can create and validate credentials without the use of root privileges, reserved ports, or platform-specific methods.

pdsh

pdsh (short for Parallel Distributed SHell) enables system administrators to execute commands across many cluster nodes quickly and at scale. This is important for tasks such as deploying configuration changes or collecting diagnostic information. Parallel execution reduces the time and effort required to manage large clusters. Different modules allow pdsh to support different remote execution services and ways to select nodes for remote execution. The pdcp (Parallel Distributed CoPy) and rpdcp (Reverse Parallel Distributed CoPy) are also part of this project.

powerman

powerman enables administrators to control the electrical power of systems remotely and in parallel. This is important when nodes are unresponsive, require recovery, or must be rebooted during maintenance. It is important for operators to be able to power-cycle large numbers of failed nodes across a cluster without visiting the data center. Powerman supports a large number of devices and protocols, so that a consistent interface can be used in heterogenous environments. Supported devices and protocols include: Baytech, Cyclade, HP Ilo, IPMI, SNMP, and Redfish.