Python

Overview | Installation | Useful Techniques | Documentation

Python Overview | Debugging Python | Python Site Packages

Overview

In conjunction with our Python user community, Livermore Computing (LC) maintains Python and a set of site-specific packages (modules) on all production CHAOS systems. The information herein, which includes the supported versions of Python and site-packages, the description of each site-package, and Python development techniques, will be useful in using Python under LC environments. Where possible, the links to external sites are also provided. General information about the Python programming language is available at the Python Web site. A Confluence Space allows users to share tips and tricks and to post requests for site-package additions.

Installation

On Linux clusters, LC maintains the default Python as /usr/local/bin/python, /usr/local/lib/python<version>, and /usr/local/include/python<version>. Unless users modify their standard search paths, /usr/local/bin/python gets invoked when the python command is typed at a UNIX prompt because /usr/local/bin takes precedence over any other paths in the standard search path that LC configures. It is useful to note, particularly when installing your own site-packages, that the /usr/local/bin/python path is a symlink to /usr/apps/python/bin/python. More details about installing modules can be found in Installing Your Own Site Packages. Furthermore, since only the python command is symlinked in /usr/local/bin, other commands, such as 2to3 and f2py, will invoke the OS-installed /usr/bin versions, and additional commands such as cython will not be in the default search path. To work around this, users are advised to use the python dotkit or manually add /usr/apps/python/bin to their PATH environment variable.

In addition to the default version, LC maintains an older version of Python—typically the previous default—to ensure a smooth transition between upgrades. This old version is available in /usr/local/bin/python<old_version>. For example, when version 2.7.3 replaced version 2.6 as the default version, the latter became accessible via the python2.6 command. LC supports the old version until the transition is believed to be complete. Newer versions may be similarly available. Users can use Dotkit to manage Python versions.

For each Python version, LC supports a set of modules, also known as site-packages, generally beneficial to our user community. Our package selection process begins when a package is requested by a user. LC first studies if it can generally benefit the Python users on the LC machines before committing to install and maintain it. If it turns out to be too specific to the individual requester, we recommend that this user make a side-installation and add the installation path to their Python module search path via the PYTHONPATH environment variable or that the user use virtualenv to manage their own Python environment. Site package requests can also be made and tracked on the site-package request wiki. The complete list of the site-packages that LC maintains is available through the following links:

Starting with versions 2.7.10 and 3.5.1, we will be installing Python using the Spack packaging management tool. The absolute path for the installation will be different under this scheme, but symlinks will be generated to mirror all the legacy paths.

Note: Our operating system installation includes a build of Python in /usr/bin/python, which is typically a slightly older version of Python. You may use this version if it works for you; however, be aware that updates, patches, and site-package additions to this Python build are usually scheduled along with the OS upgrades. We therefore encourage you to use /usr/local/bin/python because we can more flexibly modify that installation to your needs.

Using virtualenv to Manage Your Own Python Environment

We have installed virtualenv to enable users to manage their own Python build while leveraging the LC-supported Python installation. Virtualenv copies portions of a base Python installation into a user-specified directory while still pointing to the base Python installation. The virtualenv copy thus has access to the base Python installation's site-packages (and will track updates) while also allowing the user to install additional or updated site packages.

To create a virtualenv environment, run:

/usr/apps/python/bin/virtualenv --system-site-packages <yourprefix>

Note that with virtualenv 1.7 or later (i.e., with our Python 2.7.3 installation), you need to supply the --system-site-packages argument to pick up the default site-packages. With earlier versions (i.e., with our Python 2.6 installation), virtualenv will pick up the site-packages automatically, so users do not need this argument:

/usr/apps/python2.6/bin/virtualenv <yourprefix>

This command creates a virtualenv environment in the specified <yourprefix> directory. You can then access this Python via <yourprefix>/bin/python. There are many options and customizations possible with virtualenv. More information about vitrualenv is available on the virtualenv Web page.

Installing Your Own Site Packages

If you are using virtualenv, you can install site packages directly into your virtualenv Python environment. One method is to use pip, which is included in the /bin directory of your virtualenv environment. There are several ways to use pip, as mentioned in the pip documentation. One method is to download the source tarball for the desired package and then run:

<yourprefix>/bin/pip install <packagename-version>.tar.gz

You can also manually install packages using distutils by running the following command from the package source directory with your virtualenv python:

<yourprefix>/bin/python setup.py install

With virtualenv and the above examples, the site package can be imported directly, without having to set PYTHONPATH, when you run <yourprefix>/bin/python.

If you do not use virtualenv, you can still install your own site packages, but they cannot be built-in to the actual Python installation. Instead, you will have to run distutils with the --prefix option, for example:

/usr/apps/python/bin/python setup.py install --prefix=<siteprefix>

Note that the path used to install the package should be /usr/apps not /usr/local. The /usr/local/bin/python path is a symlink to /usr/apps/python/bin/python, but due to the symlink structure and how distutils works, the build may not work properly if you build the package with the /usr/local/bin/python path. In order to load the site package, you will need to add the installation directory, which is typically <siteprefix>/lib/python<version>/lib/site-packages, to your PYTHONPATH environment variable.

Additional information about installing Python modules is available on the Installing Python Modules page.

PyMPI

LC also maintains and supports pyMPI, an MPI extension to the Python programming language. PyMPI is supported on all LC OCF and SCF CHAOS systems. A version is available at /usr/local/bin/pyMPI that always depends on the default Python. For more information about pyMPI, please read Section 1 of "pyMPI—An Introduction to Parallel Python Using MPI" [PDF]. The default pyMPI is currently pyMPI 2.5b6.

Python 3

Builds of Python 3.1.3, 3.2, 3.4.2, and 3.5.1 are available with a reduced set of site packages. The site packages provided are the subset of packages that LC supports with the 2.X installations that have been ported to Python 3. As additional packages are ported, they will be installed in the Python 3 builds. Note that the executable for Python 3 is python3 as opposed to python. Using a Python 3.X dotkit will only affect the version of the python3 command and not the python command.

Useful Techniques

Below are a few basic, useful techniques for running Python. Users are also encouraged to share pointers on the Python tips and tricks wiki.

Using Python with Dotkit

You can use Dotkit to control which version of Python to use. For more information, see the Dotkit Web pages.

To see which versions of Python are available:

use -l | grep python

python-2.6 - Python

python-2.7.1 - Python

python-2.7.3 - Python

python-2.7.7 - Python

python-2.7.10 - Python

python-3.1.3 - Python

python-3.2 - Python

python-3.4.2 - Python

python-3.5.1 - Python

python - Load the full LC-built Python environment

To use the default version of Python:

python

Python 2.7.3 (default, Feb 20 2013, 16:28:57)
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

To use a specific version of Python:

use python-2.7.1

Prepending: python-2.7.1 (ok)

python

Python 2.7.1 (r271:86832, Nov 22 2011, 17:06:42)
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Invoking Python in a Script

Place the following line at the beginning of your Python script:

#! /bin/env python

Change the permissions of your script to add execute privileges:

chmod +x myscript.py

Run your script like an executable:

./myscript.py

Note: While you can also use #! /usr/local/bin/python, we do not recommend doing so because it will cause problems if you are using Dotkit to manage which Python version you are running. Also be aware that /bin/env python will pick up any alias to Python that you may have set. However, if your script depends on a specific Python version, you should use the full path.

Documentation

A lot of documentation for Python is available on the Web and in published hardcopy. The Python Web site contains links that are extremely useful. Documents and information specific to LLNL or not easily found at the Python Web site are:

If you know of other documentation that should be listed here, please contact the LC Hotline.

Top